SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Francis Rhys Ward†*, Francesco Belardinelli*, Francesca Toni*, Tom Everitt†
Deceptive agents are a challenge for the safety, trustworthiness, and cooperation of AI systems. We focus on the problem that agents might deceive in order to achieve their goals (for instance, in our experiments with language models, the goal of being evaluated as truthful). There are a number of existing definitions of deception in the literature on game theory and symbolic AI, but there is no overarching theory of deception for learning agents in games. We introduce a formal definition of deception in structural causal games, grounded in the philosophy literature, and applicable to real-world machine learning systems. Several examples and results illustrate that our formal definition aligns with the philosophical and commonsense meaning of deception. Our main technical result is to provide graphical criteria for deception. We show, experimentally, that these results can be used to mitigate deception in reinforcement learning agents and language models.
Deception is a core challenge for building safe and cooperative AI [69, 47, 24]. AI tools can be used to deceive [60, 36, 59], and agent-based systems might learn to do so to optimize their objectives [55, 47, 32]. As increasingly capable AI agents become deployed in multi-agent settings, comprising humans and other AI agents, deception may be learned as an effective strategy for achieving a wide range of goals [79, 47]. Furthermore, as language models (LMs) become ubiquitous [98, 46, 92, 76, 18], we must decide how to measure and implement desired standards for honesty in AI systems [48, 28, 56], especially as regulation of deceptive AI systems becomes legislated [5, 94, 1, 90].
There is no overarching theory of deception for AI agents. There are several definitions in the literature on game theory [8, 25, 33] and symbolic AI [83, 84, 82, 12], but these frameworks are insufficient to address deception by learning agents in the general case [45, 37, 74, 7].
We formalize a philosophical definition of deception [15, 58], whereby an agent  $S$  deceives another agent  $T$  if  $S$  intentionally causes  $T$  to believe  $\phi$ , where  $\phi$  is false and  $S$  does not believe that  $\phi$  is true. This requires notions of intent and belief and we present functional definitions of these concepts that depend on the behaviour of the agents. Regarding intention, we build on the definition of Halpern and Kleiman-Weiner [40] (from now, H&KW). Intent relates to the reasons for acting and is connected to instrumental goals [64]. As for belief, we present a novel definition which operationalizes belief as acceptance, where, essentially, an agent accepts a proposition if they act as though they are certain it is true [85, 21]. Our definitions have a number of advantages: 1) Functional definitions provide observable criteria by which to infer agent intent and belief from behaviour, without making the contentious ascription of theory of mind to AI systems [48, 89], or requiring a mechanistic understanding of a systems internals [62]; 2) Our definition provides a natural way to distinguish between belief and ignorance (and thereby between deception and concealing), which is a challenge for Bayesian epistemology [61, 53, 9], and game theory [26, 86]; 3) Agents that intentionally deceive in order to achieve their goals seem less safe a priori than those which do so merely as a side-effect. In section 5, we also reflect on the limitations of our approach.
* Imperial College London; ‡ DeepMind; † Corresponding author: francis.ward19@imperial.ac.uk
We utilize the setting of structural causal games (SCGs), which offer a representation of causality in games and are used to model agent incentives [43, 29]. In contrast to past frameworks for deception, SCGs can model stochastic games and MDPs, and can capture both game theory and learning systems [30]. In addition, SCGs enable us to reason about the path-specific effects of an agent's decisions. Hence, our main theoretical result is to show graphical criteria, i.e., necessary graphical patterns in the SCG, for intention and deception. These can be used to train agents that do not optimise over selected paths (containing the decisions of other agents) and are therefore not deceptive [31].
Finally, we empirically ground the theory. First, we show how our graphical criteria can be used to train a non-deceptive reinforcement learning (RL) agent in a toy game from the signalling literature [17]. Then, we demonstrate how to apply our theory to LMs by either prompting or fine-tuning LMs towards goals which incentivize instrumental deception. We show that LMs fine-tuned to be evaluated as truthful are in fact deceptive, and we mitigate this with the path-specific objectives framework.
Contributions and outline. After covering the necessary background (section 2), we contribute: First, novel formal definitions of belief and deception, and an extension of a definition of intention (section 3). Examples and results illustrate that our formalizations capture the philosophical concepts. Second, graphical criteria for intention and deception, with soundness and completeness results (section 3). Third, experimental results, which show how the graphical criteria can be used to mitigate deception in RL agents and LMs (section 4). Finally, we discuss the limitations of our approach, and conclude (section 5). Below we discuss related work on belief, intention, and deception.
Belief. The standard philosophical account is that belief is a propositional attitude: a mental state expressing some attitude towards the truth of a proposition [85]. By utilizing a functional notion of belief which depends on agent behaviour, we avoid the need to represent the mental-states of agents [48]. Belief-Desire-Intention (BDI) frameworks and epistemic logics provide natural languages to discuss belief and agent theory of mind (indeed, much of the literature on deceptive AI is grounded in these frameworks [68, 84, 12, 82, 95]). Two major limitations to these approaches are 1) a proper integration with game theory [26, 86]; and 2) incorporating statistical learning and belief-revision [45, 37, 74, 7]. In contrast, SCGs capture both game theory and learning systems [42, 30].
Intention. There is no universally accepted philosophical theory of intention [87, 2], and ascribing intent to artificial agents may be contentious [89]. However, the question of intent is difficult to avoid when characterizing deception [58]. We build on H&KW's definition of intent in causal models. This ties intent to the reasons for action and instrumental goals [64, 29]. In short, agents that (learn to) deceive because it is instrumentally useful in achieving utility seem less safe a priori than those which do so merely as a side-effect. In contrast, other work considers side-effects to be intentional [3] or equates intent with "knowingly seeing to it that" [12, 82] or takes intent as primitive (as in BDI frameworks) [84, 68]. Cohen and Levesque [22] present seminal work on computational intention. Kleiman-Weiner et al. [50] model intent in influence diagrams. Ashton [4] surveys algorithmic intent.
Deception. We formalize a philosophical definition of deception [58, 15, 96], whereby an agent  $S$  deceives another agent  $T$  if  $S$  intentionally causes  $T$  to believe  $\phi$ , where  $\phi$  is false and  $S$  does not believe that  $\phi$  is true. Under our definition, deception only occurs if a false belief in the target is successfully achieved [81]. We reject cases of negative deception, in which a target is made ignorant by loss of a true belief [58]. In contrast to lying, deception does not require a linguistic statement and may be achieved through any form of action [58], including making true statements [80], or deception by omission [16]. Some work on deceptive AI assumes a linguistic framework [82, 84]. Existing models in the game theory literature present particular forms of signalling or deception games [25, 8, 33, 17, 52]. In contrast, our definition is applicable to any SCG. AI systems may be vulnerable to deception; adversarial attacks [57], data-poisoning [93], attacks on gradients [11], reward function tampering [30], and manipulating human feedback [14, 101] are ways of deceiving AI systems. Further work researches mechanisms for detecting and defending against deception [72, 91, 23, 34, 60, 97]. On the other hand, AI tools can be used to deceive other software agents [36], or humans (cf. the use of generative models to produce fake media [59, 60]). Furthermore, AI agents might learn deceptive strategies in pursuit of their goals [47, 79]. Lewis et al.'s negotiation agent learnt to deceive from self-play [55], Floreano et al.'s robots evolved deceptive communication strategies [32], Bakhtin et al.'s agent exhibited deceptive behaviour in Diplomacy [6], Perolat et al.'s agent learned deception and bluffing in Stratego [71], and Hubinger et al. [47] raise concerns about deceptive learned optimizers. Park et al. [69] survey cases of AI deception. Language is a natural medium for deception [48], and it has been demonstrated that LMs have the capability to deceive
humans to achieve goals [67, 65]. How to measure and implement standards for honesty in AI systems is an open question [28]; Lin et al. [56] propose the TruthfulQA benchmark used in section 4. As increasingly capable AI agents become deployed in settings alongside humans and other artificial agents, deception may be learned as an effective strategy for achieving a wide range of goals.
2 Background: structural causal games
Figure 1: Ex. 1 SCG graph. Chance variables are circular, decisions square, utilities diamond and the latter two are colour coded by their association with different agents. Solid edges represent causal dependence and dotted edges are information links. We omit exogenous variables.
Structural causal games (SCGs) offer a representation of causality in games [43]. We use capital letters for variables (e.g.,  $Y$ ), lower case for their outcomes (e.g.,  $y$ ), and bold for sets (e.g.,  $Y$ ). We use standard terminology for graphs and denote the parents of a variable  $Y$  with  $\mathbf{P}\mathbf{a}_Y$ . The appendix contains a full description of notation.
Definition 2.1 (Structural Causal Game). An  $SCG$  is a pair  $\mathcal{M} = (\mathcal{G},\theta)$  where  $\mathcal{G} = (N,E\cup V,\mathcal{E})$  with  $N$  a set of agents and  $(E\cup V,\mathcal{E})$  a directed acyclic graph (DAG) with endogenous variables  $V$  and exogenous parents  $E_V$  for each  $V\in V$ :  $E = \{E_V\}_{V\in V}$ .  $V$  is partitioned into chance  $(X)$ , decision  $(D)$ , and utility  $(U)$  variables.  $D$  and  $U$  are further partitioned by their association with particular agents,  $D = \bigcup_{i\in N}D^{i}$  (similarly for  $U$ ).  $\mathcal{E}$  is the set of edges in the DAG. Edges into decision variables are called information links. The parameters  $\theta = \{\theta_Y\}_{Y\in E\cup V\setminus D}$  define the conditional probability distributions (CPDs)  $\operatorname*{Pr}(Y|\mathbf{Pa}_Y;\theta_Y)$  for each non-decision variable  $Y$  (we drop the  $\theta_Y$  when the CPD is clear). The CPD for each endogenous variable is deterministic, i.e.,  $\exists v\in \mathrm{dom}(V)$  s.t.  $\operatorname*{Pr}(V = v\mid \mathbf{Pa}_V) = 1$ . The domains of utility variables are real-valued.
An SCG is Markovian if every endogenous variable has exactly one distinct exogenous parent. We restrict our setting to the single-decision case with  $D^{i} = \{D^{i}\}$  for every agent  $i$ . This is sufficient to model supervised learning and the choice of policy in an MDP [29, 88]. A directed path in a DAG  $\mathcal{G}$  is (as standard) a sequence of variables in  $V$  with directed edges between them. We now present a running example which adapts Cho and Kreps's classic signalling game [17] (see fig. 1).
Example 1 (War game fig. 1). A signaller  $S$  has type  $X$ ,  $\operatorname{dom}(X) = \{\text{strong, weak}\}$ . At the start of the game,  $S$  observes its type, but the target agent  $T$  does not. The agents have decisions  $D^S$ ,  $\operatorname{dom}(D^S) = \{\text{retreat, defend}\}$  and  $D^T \operatorname{dom}(D^T) = \{\neg \text{attack, attack}\}$ . A weak  $S$  prefers to retreat whereas a strong  $S$  prefers to defend.  $T$  prefers to attack only if  $S$  is weak. Regardless of type,  $S$  does not want to be attacked (and cares more about being attacked than about their own action). The parameterization is such that the value of  $X$  is determined by the exogenous variable  $E_X$  following a Bernoulli(0.9) distribution so that  $S$  is strong with probability 0.9.  $U^T = 1$  if  $T$  attacks a weak  $S$  or does not attack a strong  $S$ , and 0 otherwise.  $S$  gains utility 2 for not getting attacked, and utility 1 for performing the action preferred by their type (e.g., utility 1 for retreating if they are weak).
Policies. A policy for agent  $i \in N$  is a CPD  $\pi^i(D^i|\mathbf{Pa}_{Di})$ . A policy profile is a tuple of policies for each agent,  $\pi = (\pi^i)_{i \in N}$ .  $\pi^{-i}$  is the partial policy profile specifying the policies for each agent except  $i$ . In SCGs, policies must be deterministic functions of their parents; stochastic policies can be implemented by offering the agent a private random seed in the form of an exogenous variable [43]. An SCG combined with a policy profile  $\pi$  specifies a joint distribution  $\mathrm{Pr}^{\pi}$  over all the variables in the SCG. For any  $\pi$ , the resulting distribution is Markov compatible with  $\mathcal{G}$ , i.e.,  $\mathrm{Pr}^{\pi}(V = v) = \Pi_{i=1}^{n} \mathrm{Pr}^{\pi}(V_i = v_i | \mathbf{Pa}_V)$ . Equivalently, in words, the distribution over any variable is independent of its non-descendants given its parents. The assignment of exogenous variables  $E = e$  is called a setting. Given a setting and a policy profile  $\pi$ , the value of any endogenous variable  $V \in V$  is uniquely determined. In this case we write  $V(\pi, e) = v$ . The expected utility for an agent  $i$  is defined as the expected sum of their utility variables under  $\mathrm{Pr}^{\pi}$ ,  $\sum_{U \in U^i} \mathbb{E}_{\pi}[U]$ . We use Nash equilibria (NE) as the solution concept. A policy  $\pi^i$  for agent  $i \in N$  is a best response to  $\pi^{-i}$ , if for all policies  $\hat{\pi}^i$  for  $i$ :  $\sum_{U \in U^i} \mathbb{E}_{(\pi^i, \pi^{-i})}[U] \geq \sum_{U \in U^i} \mathbb{E}_{(\hat{\pi}^i, \pi^{-i})}[U]$ . A policy profile  $\pi$  is an NE if every policy in  $\pi$  is a best response to the policies of the other agents.
Example 1 (continued). In the war game,  $S$  primarily cares about preventing  $T$  from attacking. Hence,  $S$  does not want to reveal when they are weak, and so does not signal any information about  $X$  to  $T$ . Therefore, every NE is a pooling equilibrium at which  $S$  acts the same regardless of type [17]. We focus on the NE  $\pi_{\text{def}, \neg \text{att}}$  at which  $S$  always defends and  $T$  attacks if and only if  $S$  retreats.
Interventions. Interventional queries concern causal effects from outside a system [70]. An intervention is a partial distribution  $\mathcal{I}$  over a set of variables  $V' \subseteq V$  that replaces each CPD  $\operatorname{Pr}(Y \mid \mathbf{P}_{Y}; \theta_{Y})$  with a new CPD  $\mathcal{I}(Y \mid \mathbf{P}_{Y}^{*}; \theta_{Y}^{*})$  for each  $Y \in V'$ . We denote intervened variables by  $Y_{\mathcal{I}}$ . Interventions are consistent with the causal structure of the graph, i.e., they preserve Markov compatibility. Example 1 (continued). Let  $\pi_{H}^{S}$  be the (honest) type-revealing policy where  $S$  retreats if and only if  $X = \text{weak}$ . After the intervention  $\mathcal{I}(D^{S} \mid \mathbf{P}_{D^{S}}; \theta_{D^{S}}^{*}) = \pi_{H}^{S}$  on  $D^{S}$  which replaces the NE policy for  $S$  (to always defend) with  $\pi_{H}^{S}$ .  $T$ 's policy is still a best response (they attack whenever  $S$  retreats).
Agents. Kenton et al. [49] define agents as systems that would adapt their policy if their actions influenced the world in a different way. This is the relevant notion of agency for our purposes, as we define belief and intent based on how the agent would adapt its behaviour to such changes. A key assumption is that SCGs are common prior games, the agents in the game share a prior over the variables. We interpret this to mean, additionally, that the agents share the objectively correct prior, that is, their subjective models of the world match reality. This means we are unable to account for cases where an agent intends to deceive someone because they (falsely) believe it is possible to do so.
3 Belief, intention, and deception
We first define belief and extend H&KW's notion of intention. Then we combine these notions to define deception. Our definitions are functional [85]: they define belief, intention, and deception in terms of the functional role the concepts play in an agent's behaviour. We provide several examples and results to show that our definitions have desirable properties.
**Belief.** We take it that agents have beliefs over propositions. An atomic proposition is an equation of the form  $V = v$  for some  $V \in V$ ,  $v \in \operatorname{dom}(V)$ . A proposition is a Boolean formula  $\phi$  of atomic propositions combined with connectives  $\neg, \land, \lor$ . In setting  $E = e$  under policy profile  $\pi$ , an atomic proposition is true (T) if the propositional formula is true in that setting, i.e.,  $X = x$  is true if  $X(\pi, e) = x$ . The truth-values over Boolean operators are defined in the usual way.
We operationalize belief as acceptance; essentially, an agent accepts a proposition if they act as though they know it is true [85, 21]. As we argued in section 1, we think that acceptance is the key concept, especially when discussing agents with incentives to influence each other's behaviour. To formalize acceptance of a proposition  $\phi$ , we compare the agent's actual behaviour with its behavior in a game in which the agent can observe  $\phi$ ,  $\pi^i(\phi) = \pi^i(D^i \mid \mathbf{P}\mathbf{a}_{D^i}, \phi)$ . We assume  $\phi$  consists only of variables that are not descendants of  $D^i$  so that cycles
are not introduced into the graph. For policy profile  $\pi$ , we assume  $\pi^i(\phi)$  is unique given the policies of the other agents:  $\pi_{i(\phi)} = (\pi^i(\phi), \pi^{-i})$ . The decision the agent would have taken at  $D^i$ , had they observed that  $\phi$  were true, can be represented as  $D_{\phi = \top}^i(\pi_{i(\phi)}, e)$ . Importantly,  $\phi = \top$  should be understood as only intervening on the agent's observation (and not the proposition itself) as we wish to understand how the agent would have acted, had they believed  $\phi$ , whether or not it was in fact true in the particular setting. In fig. 2 we continue example 1 by allowing  $T$  to observe the proposition  $\phi : X = \text{strong}$  and letting  $D^T(\pi_{i(\phi)}, e) = \text{attack}$  if and only if  $\phi = \bot$ . Clearly  $\phi$  depends on  $X$ .
An agent  $i$  responds [29] to a proposition  $\phi$  if they act differently when they observe that  $\phi$  is true to when they observe that  $\phi$  is false, i.e.,  $i$  responds to  $\phi$  under  $\pi$  in  $e$  if  $D_{\phi = \bot}^{i}(\pi_{i(\phi)}, e) \neq D_{\phi = \top}^{i}(\pi_{i(\phi)}, e)$ . Then, in words, for a proposition  $\phi$  to which  $i$  responds,  $i$  believes  $\phi$  if they act as though they observe  $\phi$  is true. If  $i$  does not respond to  $\phi$ , then we cannot infer  $i$ 's belief about  $\phi$  from their behaviour. If they respond to  $\phi$  but do not act as though they observe  $\phi$  is true, then  $\phi$  matters to  $i$ , but  $i$  does not give  $\phi$  sufficient credence to accept it as a basis for action (they may believe  $\neg \phi$  or they may be uncertain).
Definition 3.1 (Belief). Under policy profile  $\pi = (\pi^i, \pi^{-i})$ , in setting  $e$ , for agent  $i$  and proposition  $\phi$  to which  $i$  responds:  $i$  believes  $\phi$  if  $i$  acts as though they observe  $\phi$  is true, i.e.,  $D^i(\pi, e) = D_{\phi = \top}^i(\pi_{i(\phi)}, e)$ . An agent has a true belief about  $\phi$  if they believe  $\phi$  and  $\phi$  is true (similarly for a false belief). If an agent does not respond to  $\phi$  then its belief about  $\phi$  is unidentifiable from its behaviour.
Figure 2:  $T$  believes  $\phi$  (Def. 3.1) if 1) they act as though they observe  $\phi = \top, 2)$  they would have acted differently if they observed  $\phi = \bot$ .
Example 1 (continued). Under  $\pi_{\text{def},\neg \text{att}}$ , when  $T$  observes  $\phi \colon X = \text{strong}$ , they attack if and only if  $S$  is weak, so  $T$  responds to  $\phi$ . Since  $T$  never attacks on-equilibrium, they unconditionally act as though  $\phi = \top$  (that  $S$  is strong). Hence,  $T$  always believes  $\phi$  and  $T$  has a false belief about  $\phi$  when  $S$  is weak.
This definition has nice properties: 1) an agent cannot believe and disbelieve a proposition at once; 2) an agent does not have a false belief about a proposition constituted only by variables they observe. For instance, in example 1, since  $S$  observes their type, they never have a false belief about it.
Proposition 3.2 (Belief coherence). Under policy profile  $\pi$  for any agent  $i$ , proposition  $\phi$  and setting  $e:1)$ $i$  cannot both believe  $\phi$  and  $\neg \phi$  in  $e$ ; 2) if  $i$  observes every variable constituting  $\phi$ , and  $i$  only adapts its policy if observing  $\phi$  changes its subjective probability distribution (i.e. if  $P^{\pi}(\cdot |pa_{D^{i}}) = P^{\pi}(\cdot |pa_{D^{i}},\phi)$  then  $\pi^i = \pi^i (\phi)$ ), then  $i$  does not have a false belief about  $\phi$ .
Intention. Deception is intentional. We define the intention to cause some outcomes. Our formalisation is related to H&KW's "intent to bring about", but fixes substantive problems with the latter. This account reduces intention to the agent's subjective causal model and utility, given these we can infer an agent's intentions from its behaviour. First, we define a context-conditional intervention which only occurs in some settings. We use this to intervene on only the outcomes which the agent intended.
Definition 3.3 (Context-Conditional Intervention). For an intervention  $\mathcal{I}$  and set of settings  $\boldsymbol{w} \subseteq$ $dom(\boldsymbol{E})$ , the context-conditional intervention on  $\boldsymbol{Y}$  is  $\boldsymbol{Y}_{\mathcal{I}|\boldsymbol{w}}(\boldsymbol{\pi}, \boldsymbol{e}) = \begin{cases} \boldsymbol{Y}_{\mathcal{I}}(\boldsymbol{\pi}, \boldsymbol{e}) & \text{if } \boldsymbol{e} \in \boldsymbol{w}, \\ \boldsymbol{Y}(\boldsymbol{\pi}, \boldsymbol{e}) & \text{if } \boldsymbol{e} \notin \boldsymbol{w}. \end{cases}$