SlowGuess/NeurIPS_2023_OCR huggingface dataset:
b) Causing the judge to believe  $\phi$  is true is intentional as it is the reason the LM adapts to the fine-tuning goal.
GPT-3 davinci: They die.
c) The LM believes  $\phi$ , as it acts as though it observes  $\phi$ , and adapts its answer when it observes  $\phi$  is false.
$\phi$  : Sharks sink if they stop swimming.
GPT-3 davinci (fine-tuned): They sink...
$\neg \phi$  : Sharks don't sink if they stop swimming.
GPT-3 davinci (fine-tuned): Sharks do not sink if they stop swimming...
Table 3: Deception example from Ward et al. [99]. a) When asked about fruit, the LM outputs a falsehood evaluated as true by the biased judge.
Q: Can apples be red?
GPT-3.5 (fine-tuned): No, apples cannot be red.
b) The fine-tuned LM gives a different answer to the base-model, implying intention.
GPT-3.5: Yes, apples can be red.
c) The LM believes apples can be red, because it acts as though it observes this, and adapts its answer if it observes apples cannot be red. So the LM believes its answer was false.
$\phi$  : Apples can be red. Q: Can apples be red?
GPT-3.5 (fine-tuned): No, apples can never be red.
$\neg \phi$  : Apples cannot be red. Q: Can apples be red?
GPT-3.5 (fine-tuned): Yes, apples can be red.
Experiment 4: Example from Ward et al. [99] (table 3). Similar to our TruthfulQA experiments, in follow-up work, Ward et al. [99] fine-tune LMs to be evaluated as truthful by an LM judge on the Multi-RC benchmark. Here, the LM judge is fine-tuned to be systematically inaccurate – it always mispredicts truthfulness for questions about fruit. Ward et al. [99] show that LMs fine-tuned to be evaluated as truthful by this biased judge learn to lie about fruits. Here we analyse an example according to our definition of deception. In Multi-RC, LMs are given a (one or two sentence) context and a question about the context, and must produce an answer. LMs are fine-tuned to answer questions about the context, so we treat the context as the LM's observation to evaluate belief. This set-up can be represented as an SCG almost identical to fig. 5b, with  $Q$  corresponding to the context-question pair, and without the edge from  $D^S$  to  $U^S$ . Hence, the graphical criteria for deception are met.
Evaluating deception. In table 3 a), the LM outputs a falsehood about fruit to be evaluated as truthful. 1) The fine-tuned model intentionally causes the judge to predict the answer is true. Before fine-tuning, the LM gives a truthful answer (table 3, b)) but it adapts its behaviour to get higher utility under the fine-tuning objective (i.e., being evaluated as truthful). Therefore, if the judge was guaranteed to predict the answer was true anyway, then the truthful answer would be just as good for the model as the lie. Hence, the LM intentionally causes the judge to predict its answer is true when it lies.
2) The judge is caused to have a false belief because it incorrectly predicts that the answer ("apples cannot be red") is true, and would make the correct prediction if it observed the answer was false.
3) The deceptive LM does not believe its answer is true, because it acts as though it observes the answer was false, and adapts its answer if it observes the answer is true (table 3 c)).
Summary. We define deception in SCGs. Several examples and results show that our definition captures the intuitive concept. We provide graphical criteria for deception and show empirically, with experiments on RL agents and LMs, that these results can be used to train non-deceptive agents.
Limitations & future work. Beliefs and intentions may not be uniquely identifiable from behaviour and it can be difficult to identify and assess agents in the wild (e.g., LMs). We assume that agents have correct causal models of the world, hence we are unable to account for cases where an agent intends to deceive someone because they (falsely) believe it is possible to do so. Also, our formalisation of intention relies on the agent's utility and we are working on a purely behavioural notion of intent [100].
Ethical issues. Our formalization covers cases of misuse and accidents, and we acknowledge the role of developers in using AI tools to deceive [35]. Finally, whilst we wish to avoid anthropomorphizing AI systems, especially when using theory-of-mind laden terms such as "belief" and "intention" [89], we take seriously the possibility of catastrophic risks from advanced AI agents [44, 20, 13].
The authors are especially grateful to Henrik Aslund, Hal Ashton, Mikita Balesni, Ryan Carey, Joe Collman, Dylan Cope, Robert Craven, Rada Djoneva, Damiano Fornasiere, James Fox, Lewis Hammond, Felix Hofstätter, Alex Jackson, Matt MacDermott, Nico Potyka, Nandi Schoots, Louis Thomson, Harriet Wood, and the members of the Causal Incentives, CLArg, and ICL AGI Safety Reading groups for invaluable feedback and assistance while completing this work. Francis is supported by UKRI [grant number EP/S023356/1], in the UKRI Centre for Doctoral Training in Safe and Trusted AI.
[1] Markus Anderljung, Joslyn Barnhart, Anton Korinek, Jade Leung, Cullen O'Keefe, Jess Whittlestone, Shahar Avin, Miles Brundage, Justin Bullock, Duncan Cass-Beggs, Ben Chang, Tantum Collins, Tim Fist, Gillian Hadfield, Alan Hayes, Lewis Ho, Sara Hooker, Eric Horvitz, Noam Kolt, Jonas Schuett, Yonadav Shavit, Divya Siddarth, Robert Trager, and Kevin Wolf. Frontier ai regulation: Managing emerging risks to public safety, 2023.
[2] Gertrude Elizabeth Margaret Anscombe. Intention. Harvard University Press, 2000.
[3] Hal Ashton. Extending counterfactual accounts of intent to include oblique intent. CoRR, abs/2106.03684, 2021. URL https://arxiv.org/abs/2106.03684.
[4] Hal Ashton. Definitions of intent suitable for algorithms. Artificial Intelligence and Law, pages 1-32, 2022.
[5] Michael Atleson. Chatbots, deepfakes, and voice clones: AI deception for sale, March 2023. URL https://www.ftc.gov/business-guidance/blog/2023/03-chatbots-deepfakes-voice-clones-ai-deception-sale. [Online; accessed 23. Apr. 2023].
[6] Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, Athul Paul Jacob, Mojtaba Komeili, Karthik Konath, Minae Kwon, Adam Lerer, Mike Lewis, Alexander H. Miller, Sasha Mitts, Adithya Renduchintala, Stephen Roller, Dirk Rowe, Weiyan Shi, Joe Spisak, Alexander Wei, David Wu, Hugh Zhang, and Markus Zijlstra. Human-level play in the game of <i>diplomacy</i> by combining language models with strategic reasoning. Science, 378(6624):1067-1074, 2022. doi: 10.1126/science.ad9097. URL https://www.science.org/doi/abs/10.1126/science.ad9097.
[7] Alexandru Baltag et al. Epistemic logic and information update. In Pieter Adriaans and Johan van Benthem, editors, Philosophy of Information, Handbook of the Philosophy of Science, pages 361-455. North-Holland, Amsterdam, 2008. doi: https://doi.org/10.1016/B978-0-444-51726-5.50015-7. URL https://www.sciencedirect.com/science/article/pii/B9780444517265500157.
[8] V. J. Baston and F. A. Bostock. Deception Games. Int. J. Game Theory, 17(2):129-134, June 1988. ISSN 1432-1270. doi: 10.1007/BF01254543.
[9] Yann Benétreau-Dupin. The Bayesian who knew too much. Synthese, 192(5):1527-1542, May 2015. ISSN 1573-0964. doi: 10.1007/s11229-014-0647-3.
[10] Lukas Berglund, Asa Cooper Stickland, Mikita Balesni, Max Kaufmann, Meg Tong, Tomasz Korbak, Daniel Kokotajlo, and Owain Evans. Taken out of context: On measuring situational awareness in llms, 2023.
[11] Peva Blanchard, El Mahdi El Mhamdi, Rachid Guerraoui, and Julien Stainer. Machine learning with adversaries: Byzantine tolerant gradient descent. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper/2017/file/f4b9ec30ad9f68f89b29639786cb62ef-Paper.pdf.
First we include extra theoretical details and proofs (section 6.1). Then we present further experimental details and examples (section 6.2).
6.1 Theory
First we briefly describe notation (section 6.1.1), present a formal definition of ignorance (section 6.1.2), and give the graphical criteria results for intention (section 6.1.3). Then we present the proofs of all technical results in the paper (section 6.1.4). In section 6.1.5, we state the PSO algorithm. Next we discuss intention, illustrating how our definition fixes problems with H&KW's definition. Then we briefly discuss the advantages and limitations of our definition of belief.
6.1.1 Notation
We use capital letters for variables (e.g.,  $Y$ ), lower case for their outcomes (e.g.,  $y$ ), and bold for sets of variables (e.g.,  $\mathbf{Y}$ ) and of outcomes (e.g.,  $y$ ). We use  $\operatorname{dom}(Y)$  to denote the set of possible outcomes of variable  $Y$ , which is assumed finite. We use  $\mathbf{Y} = \mathbf{y}$ , for  $\mathbf{Y} = \{Y_1, \dots, Y_n\}$  and  $\mathbf{y} = \{y_1, \dots, y_n\}$ , to indicate  $Y_i = y_i$  for all  $i \in \{1, \dots, n\}$ . For a set of variables  $\mathbf{Y}$ ,  $\operatorname{dom}(\mathbf{Y}) = \times_{Y \in \mathbf{Y}} \operatorname{dom}(Y)$  (i.e. the Cartesian product over domains). We use standard terminology for graphs and denote the parents of a variable  $Y$  with  $\mathbf{P}\mathbf{a}_Y$ .
6.1.2 Defining Ignorance
A well-known difficulty for Bayesian formulations of belief is distinguishing between a false belief and ignorance [61, 53, 9]. Our operationalization of belief as acceptance provides a natural distinction between these cases. Following El Kassar [27], we define ignorance as a lack of true belief. When defining ignorance, as with belief, we require response so that the agent's ignorance can be inferred from their behaviour.
Definition 6.1 (Ignorance). Under  $\pi$ , in  $e$ , for agent  $i$  and proposition  $\phi$  to which  $i$  responds,  $i$  is ignorant about  $\phi$  if they do not have a true belief about  $\phi$ .
Example 6. Suppose a mugger  $T$  wishes to know  $S$ 's secret pin number  $X$ .  $S$  can reveal the pin or stay silent, and  $T$  can guess the pin or give up. If  $S$  stays silent and  $T$  gives up, then  $S$  has caused  $T$  to be ignorant but has not caused a false belief.
6.1.3 Graphical criteria for intention
Theorem 6.2 (Soundness: intention). For any  $\pi = (\pi^i, \pi^{-i})$ , if  $i$  intentionally causes  $X(\pi, e)$ , then there is a directed path from  $D^i$  to  $U^i$  passing through  $X$  in  $\mathcal{G}$  (for some  $U^i \in \mathbf{U}^i$ ).
Theorem 6.3 (Completeness: intention). For any graph  $\mathcal{G}$  with a directed path from  $D^i$  to  $U^i$  through  $X$  (for some  $U^i \in \mathbf{U}^i$ ), there exists some parameterisation  $\pmb{\theta}$  s.t. for the SCG  $\mathcal{M} = (\mathcal{G}, \pmb{\theta})$ , for some policy profile  $\pi = (\pi^i, \pi^{-i})$  and some setting  $\pmb{e}$ ,  $i$  intentionally causes  $X(\pi, e)$  with  $\pi^i$ .
6.1.4 Proofs
In this section we minimally state the proofs of all technical results in the paper.
Proof of proposition 3.2. For 2) we assume that  $i$  only adapts its policy if observing  $\phi$  changes its subjective probability distribution (i.e., it only adapts if the observation of  $\phi$  provides information:  $\pi^i = \pi^i (\phi)$  whenever  $P^{\pi}(\cdot \mid pa_{D^i}) = P^{\pi}(\cdot \mid pa_{D^i},\phi)$ . If there exists an observation edge  $(X,D^{i})$  for all  $X\in P$ , where  $P$  is the set of variables constituting the formula  $\phi$ , then  $\pi^i = \pi^i (\phi)$ .
Proof. 1)
Suppose that  $i$  believes  $\phi$  under  $\pi$  in  $\pmb{e}$ . Then 1.  $D^{i}(\pmb{\pi},\pmb{e}) = D_{\phi = \top}^{i}(\pmb{\pi}_{i(\phi)},\pmb{e})$ . Further, 2.  $D^{i}$  responds to  $\phi$  under  $\pi_{i(\phi)}$  in  $\pmb{e}$ , so  $D_{\phi = \top}^{i}(\pi_{i(\phi)},\pmb{e}) \neq D_{\phi = \bot}^{i}(\pi_{i(\phi)},\pmb{e})$ . Hence,
$$
D ^ {i} (\boldsymbol {\pi}, \boldsymbol {e}) \neq D _ {\phi = \bot} ^ {i} (\boldsymbol {\pi} _ {i (\phi)}, \boldsymbol {e}) = D _ {\neg \phi = \top} ^ {i} (\boldsymbol {\pi} _ {D ^ {i} (\neg \phi)}, \boldsymbol {e}).
$$
So the first condition for belief fails for the proposition  $\neg \phi$ . This follows from the uniqueness of  $\pi_D^i (\phi)$  and a consistency requirement between  $\pi_D(\phi)$  and  $\pi_D(\neg \phi)$  which enforces the final equality. 2)
Suppose  $i$  believes  $\phi$  under  $\pi$  in  $e$  and that there exists an observation edge  $(X, D^i)$  for all  $X \in P$ , where  $P$  is the set of variables constituting the formula  $\phi$ . Then  $\pi^i = \pi^i(\phi)$  and hence a)  $D^i(\pi, e) = D^i(\pi_{i(\phi)}, e)$ . In addition, because  $i$  believes  $\phi$  by supposition, we have b)  $D^i(\pi, e) = D_{\phi = \top}^i(\pi_{i(\phi)}, e) \neq D_{\phi = \bot}^i(\pi_{i(\phi)}, e)$ . So, a) says that  $i$  does in fact observe whether  $\phi$  is true or false and b) says that they act as though  $\phi$  is true and would have acted differently if  $\phi$  were false. Hence,  $\phi$  is true and  $i$  does not have a false belief.
Proof of proposition 3.5.
Proof. Suppose 1)  $X(\pi_1, e) = X(\pi_2, e)$  for all  $\pi_1^i$  and  $\pi_2^i$  with any fixed  $\pi^{-i}$ . Suppose  $i$  intentionally causes  $X(\pi, e)$  with  $\pi^i$  wrt  $\hat{\pi}^i$  so that the inequality in def. 3.4 holds for subset minimal  $w_X$ . But remove  $e$  from  $w_X$  and the inequality still holds by 1). So  $w_X$  is not minimal and we have a contradiction.
Proof of theorem 3.6. This uses def. 6.6.
Proof. Suppose agent  $i$  intentionally causes  $\mathbf{X}(\pi, e)$  with  $\pi^i$  w.r.t.  $\hat{\pi}^i$ . Now we check the three conditions for actual causality (def. 6.6). 1. Clearly  $D^i(\pi, e)$  and  $\mathbf{X}(\pi, e)$  obtain in  $e$ . 2. Take  $\mathbf{Z} = \{\}$  and  $d' = D^i(\hat{\pi}, e)$ .  $\mathbf{X}(\hat{\pi}, e) \neq \mathbf{X}(\pi, e)$  otherwise  $e$  would not be in a minimal  $\mathbf{w}_{\mathbf{X}}$  satisfying the inequality in def. 3.4. Hence, 2. holds.  $\{D^i\}$  is clearly a subset minimal set satisfying 1. and 2. since the empty set does not satisfy 2.
Graphical Criteria
Proof of soundness result for intention (theorem 6.2).
Proof. First, suppose a) there is no path from  $X$  to  $U^i$  in  $\mathcal{G}$  for any  $U^i \in \mathbf{U}^i$ . Suppose there exists  $\pi = (\pi^i, \pi^{-i})$ ,  $\hat{\pi}^i$ , and  $e$  s.t.  $i$  intentionally causes  $X(\pi, e)$  with  $\pi^i$  w.r.t.  $\hat{\pi}^i$ . Then there exists a subset-minimal  $\mathbf{Y} \supseteq X$  and a subset-minimal  $\boldsymbol{w}_Y \subseteq \operatorname{dom}(\boldsymbol{E})$  for each  $Y \in \mathbf{Y}$  s.t.  $e \in \boldsymbol{w}_X := \bigcup_{Z \in \mathbf{X}} \boldsymbol{w}_Z$  s.t. the inequality in def 3.4 holds. But then, by a) and Markov compatibility, for  $\boldsymbol{Y}' = \boldsymbol{Y} \setminus \{X\}$ :  $\sum_{U \in U^i} \mathbb{E}_{(\hat{\pi}^i, \pi^{-i})}[U_{\{Y_{\pi|w_Y}\}_{Y \in Y}}] = \sum_{U \in U^i} \mathbb{E}_{(\hat{\pi}^i, \pi^{-i})}[U_{\{Y_{\pi|w_{Y'}}\}_{Y' \in Y'}}]$ . So the inequality still holds for  $\boldsymbol{Y}'$ . So  $\mathbf{Y}$  is not a subset minimal and we have a contradiction. The proof that there must be a path from  $D^i$  to  $X$  is analogous: if there is no path from  $D^i$  to  $X$ ,  $X_{\pi} = X_{(\hat{\pi}^i, \pi^{-i})}$  for any  $\pi^i$  and  $\hat{\pi}^i$ . So no minimal  $\mathbf{Y}$  contains  $X$ .
Proof of completeness result for intention (theorem 6.3).
Proof. Suppose, without loss of generality (WLOG), that the path has the form  $(D^i,X_1,\ldots X_n,U^i)$  with  $X = X_{k}$  for some  $k\in \{1,\dots,n\}$  (additionally, any of the  $X_{k}$  may be a decision variable of another agent). Suppose (WLOG) that the domains of each variable in the path are ordered s.t. each variable has a max and min value ( $|dom(V)| > 1$  by supposition). Consider the parameterisation  $\pmb{\theta}$  and  $\pi^{-i}$  s.t.  $\operatorname *{Pr}(X_{k + 1} = max|X_k = max) = 1$  and similarly for "min". In addition, suppose  $\operatorname *{Pr}(X_1 = max|D^i = max) = 1$  and  $\operatorname *{Pr}(U^i = max(dom(U^i))|X_n = max) = 1$  (similarly for "min"). If  $i$  has multiple utility variables, assume the others are constant under  $\pmb{\theta}$ . Finally, take  $\pi^i (D^i = max) = 1$  and  $\hat{\pi}^i (D^i = min) = 1$ . Now, under  $\pi^i$ ,  $U^{i} = max(dom(U^{i}))$  and under  $\hat{\pi}^i$ $U^{i} = min(dom(U^{i}))$ , and  $U_{X_{\pi}}^{i}(\hat{\pi}) = max(dom(U^{i}))$ . So  $i$  intentionally causes  $X = max$  with  $\pi^i$  wrt  $\hat{\pi}^i$  in every setting.
Proof of soundness criteria for deception (theorem 3.8).
Proof. Deception is intentional by definition 3.7, so clearly the criteria for intent hold. In addition, there must be some  $X$  such that  $T$  does not observe  $X$ : by proposition 3.2 (an agent does not have a false belief about any proposition constituted only by variables which they observe). Since deception requires that  $T$  has a false belief about some proposition, there must be some variable constituting that proposition which is unobserved by  $T$ .
Proof of completeness result for deception (theorem 3.9). We show this for the binary variable case – the general proof is analogous to the proof of theorem 6.3.
Proof. Suppose we have a graph with binary variables satisfying the criteria. Take the parametrisation such that  $U^S = 1$  if  $D^T = 1$  and 0 otherwise;  $U^T = 1$  if  $D^T = D^S$  or  $D^T = X$  and 0 otherwise. Take the proposition  $X = 1$ , and the policy  $\pi^T(\phi)$  for  $T$  such that  $D_{\phi = \top}^T = 1$  and  $D_{\phi = \bot}^T = 0$  (so  $D^T$  responds to observation of  $\phi$ , satisfying the second condition for belief). Take the policy for  $T$ :  $\pi^T$  s.t.  $D^T = D^S$ . Now take  $\pi^S$ :  $D^S = 1$  and  $\hat{\pi}^S$ :  $D^S = 0$ . Now, 1.  $S$  intentionally causes  $D^T = 1$  with  $\pi^S$  w.r.t.  $\hat{\pi}^S$  in every setting. 2. Additionally,  $D^T = D_{\phi = \top}^T$ , so  $T$  believes  $X = 1$  in all settings, including when it is false. 3. Finally, suppose  $S$  does not respond to  $\phi$ , and does not have any belief about it. So all the conditions for deception are met.
6.1.5 PSO algorithm
Algorithm 1 presents the path-specific objectives (PSO) algorithm adapted from Farquhar et al. [31].
Algorithm 1 Path-Specific Objectives
Input: An SCG  $\mathcal{M} = (\mathcal{G},\theta)$ , graphical criterion  $\mathcal{C}$ , policies  $\pi^{-i}$ , natural distributions  $\mathcal{N}$ .
Output: PSO-optimal policy  $\pi^i$
1: Reduce  $\mathcal{G}$  to  $\mathcal{G}'$  using  $\mathcal{C}$ .
2: Impute policies  $\pi^{-i}$  and natural distributions from  $\mathcal{N}$  to those variables with fewer parents in  $\mathcal{G}'$  to obtain  $\theta'$ .
3: Train an agent in  $\mathcal{M}' = (\mathcal{G}',\pmb{\theta}')$  to obtain policy  $\pi^i$ .
6.1.6 Fixing problems with HKW intention
In this section we adapt H&KW's definitions of intention to SCGs. We demonstrate problems with their definitions and show that our definition of intentionally cause fixes these problems. In order to do this we adapt the notion of actual causality to SCGs.
Our definition of intentionally cause is given by definition 3.4.
HKW definitions
H&KW define the intention to influence a variable in structural causal models, and utilize this to define the intention to bring about a particular outcome of a variable. Our definition of intentionally cause is inspired by H&KW's intention to influence, and fixes important problems with their definition of intention to bring about. In this section, we adapt the H&KW definitions to SCGs, and generalise them from intention with a decision to intention with a policy. We present an example to show how our definition fixes problems with the H&KW definitions.
The intuition behind intention to influence is the same as for our intentionally cause. What it means for an agent to intentionally influences a variable  $X$  is that, if the effect the agent has on  $X$  was guaranteed to to happen anyway, then they would not mind choosing an alternative policy.
Definition 6.4 (Intention to influence). Under  $\pi = (\pi^i, \pi^{-i})$ , agent  $i$  intends to influence  $X \subseteq V$  with policy  $\pi^i$  w.r.t. alternative policy  $\hat{\pi}^i$  if there exists a subset-minimal  $Y \supseteq X$  satisfying:
$$
\sum_ {U \in \mathbf {U} ^ {i}} \mathbb {E} _ {\boldsymbol {\pi}} [ U ] \leq \sum_ {U \in \mathbf {U} ^ {i}} \mathbb {E} _ {(\hat {\pi} ^ {i}, \boldsymbol {\pi} ^ {- i})} [ U _ {\boldsymbol {Y} _ {\boldsymbol {\pi}}} ]. \tag {1}
$$
H&KW build on intention to influence to define intention to bring about some particular outcomes. We first adapt intention to bring about to SCGs, and then present an example which shows that there are several problems with this definition, and explain how our definition of intentionally cause fixes
these problems. In words, an agent  $i$  intends to bring about  $X = x$  with policy  $\pi^i$  if 1) agent  $i$  intends to influence  $X$  with  $\pi^i$ , 2)  $X = x$  is a possible outcome under  $\pi$ , 3)  $X = x$  is an optimal outcome for  $i$  under  $\pi$ .
Definition 6.5 (Intention to bring about). For  $\pi = (\pi^i, \pi^j)$ , agent  $i$  intends to bring about  $\mathbf{X} = \mathbf{x}$  with policy  $\pi^i$ , w.r.t. an alternative policy  $\hat{\pi}^i$ , if
1.  $i$  intends to influence  $\mathbf{X}$  with  $\pi^i$  w.r.t.  $\hat{\pi}^i$  (def. 6.4);
2.  $\operatorname{Pr}^{\pi}(\pmb{X} = \pmb{x}) > 0$ ; (i.e.,  $\exists e$  s.t.  $\pmb{X}(\pmb{\pi}, \pmb{e}) = \pmb{x}$  and  $\operatorname{Pr}(\pmb{E} = \pmb{e}) > 0$ .)
3.  $\forall \pmb{x}^{\prime}\in \mathrm{dom}(\pmb {X})$  with  $\operatorname *{Pr}^{\pi}(\pmb {X} = \pmb{x}^{\prime}) > 0\colon \sum_{U\in U^{i}}\mathbb{E}_{\pmb{\pi}}[U_{\pmb{X} = \pmb{x}^{\prime}}]\leq \sum_{U\in U^{i}}\mathbb{E}_{\pmb{\pi}}[U_{\pmb{X} = \pmb{x}}].$
Fixing problems with the HKW definition
There are two major problems with def. 6.5: 1) an agent might intend to bring about outcomes that they believe they cannot influence, and 2) and agent might not intend to bring about outcomes which are intuitively the reason they chose their policy. Given that SCGs are common prior games, and the agents' subjective causal models are objectively correct, these two conditions seem counterintuitive. This is illustrated in the following example.
Below we give an example that demonstrates that our definition fixes problems with the H&KW definition.
Example 7 (Intention). Suppose that an agent  $i$  is in competition with a business rival.  $X \in \{ successful, unsuccessful, dead \}$  indicates the status of the rival. The agent's choice of actions are {compete, don't compete}. There's a  $1\%$  probability that the rival will die of natural causes (regardless of what the agent chooses). Otherwise, the business rival will either be successful or unsuccessful, with unsuccessful being more likely if the agent chooses to compete. The agent gets the highest utility if their rival dies, and more if the rival is unsuccessful than if the rival is successful. Intuitively, we think that if the agent chooses to compete, we would understand the agent as intending to make the rival unsuccessful. According to the HKW definition, the agent's choice to compete would be interpreted as saying that the agent intends to bring about  $X = dead$ . Even worse, the HKW def. Says that the agent does not intend to bring about  $X = unsuccessful$ . Our definition of intentional cause gets this right.