Title: A Causal Model of Theory-of-Mind in AI Agents
Venue: NeurIPS 2024
Accepted: no
Keywords: causality

games with incomplete information

causal inference

multi-agent influence diagrams

game theory
Abstract: Agency is a vital concept for understanding and predicting the behaviour of future AI systems. There has been much focus on the goal-directed nature of agency, i.e., the fact that AI agents may capably pursue goals. However, the dynamics of agency become significantly more complex when autonomous agents interact with other agents and humans, necessitating engagement in theory-of-mind, the ability to reason about the beliefs and intentions of others. In this paper, we extend the framework of multi-agent influence diagrams (MAIDs) to explicitly capture this complex form of reasoning. We also show that our extended framework, MAIDs with incomplete information (II-MAIDs), has a strong theoretical connection to dynamic games with incomplete information with no common prior over types. We prove the existence of important equilibria concepts in these frameworks, and illustrate the applicability of II-MAIDs using an example from the AI safety literature.

1

Abstract

Agency is a vital concept for understanding and predicting the behaviour of future AI systems. There has been much focus on the goal-directed nature of agency, i.e., the fact that AI agents may capably pursue goals. However, the dynamics of agency become significantly more complex when autonomous agents interact with other agents and humans, necessitating engagement in theory-of-mind , the ability to reason about the beliefs and intentions of others. In this paper, we extend the framework of multi-agent influence diagrams (MAIDs) to explicitly capture this complex form of reasoning. We also show that our extended framework, MAIDs with incomplete information (II-MAIDs), has a strong theoretical connection to dynamic games with incomplete information with no common prior over types. We prove the existence of important equilibria concepts in these frameworks, and illustrate the applicability of II-MAIDs using an example from the AI safety literature.

1 Introduction

The concept of agency plays a central role in AI, from philosophical discussions of the nature 14 of artificial agents [5] to the practical engineering of agent-like systems [12, 39]. Existing work 15 formalising agency typically focuses on its goal-directed nature in a single-agent setting [25, 30]. 16 However, a full picture of agency should describe systems that represent themselves and other systems 17 as agents , i.e., systems with theory-of-mind (ToM) [7, 8]. 18

19

20

21

22

23

24

25

26

27

ToM is characterised by multi-agent interactions involving higher-order intentional states [7], such as beliefs about beliefs, or, in the case of deception, intentions to cause false beliefs [40]. Causality often plays a key role in philosophical notions of belief [38], and causal models offer a powerful representation of beliefs [14, 36], intentions [41], and other intentional states [13]. Additionally, causal models have been extended to capture game-theoretic dynamics in the setting of multi-agent influence diagrams (MAIDs) [26, 16]. However, MAIDs assume that all agents in the model have the same, correct beliefs about the world, each other's beliefs, each other's beliefs about beliefs, and so on. With this assumption in place, MAIDs do not explicitly model agents' subjective beliefs or higher-order beliefs.

We generalise MAIDs to the setting of incomplete information with no common prior , wherein agents 28

may have different and inconsistent beliefs about the world, and each agent may have different 29

beliefs about the beliefs of other agents. Our framework, incomplete information MAIDs (II-MAIDs) , 30

includes explicit subjective belief hierarchies, and therefore enables us to model systems of agents 31 with more complex and realistic ToM. 32

Contributions and Outline. In Section 2, we discuss formal background on MAIDs and EFGs. We 33

formally define our framework of MAIDs with incomplete information (II-MAIDs) in Section 3. In 34

Section 4, we present a variant of an existing formalism for incomplete information games using 35

EFGs rather than normal-form games, and in Section 5 we prove that it is equivalent to MAIDs with 36

incomplete information. Finally, we review related literature (Section 6) and conclude (Section 7). 37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

2 Background

In this section, we provide formal definitions of MAIDs and EFGs and explain these game representations using an example. A Bayesian network is a probabilistic graphical model representing a set of variables and their conditional dependencies via a directed acyclic graph. Influence diagrams (IDs) generalise Bayesian networks to the decision-theoretic setting by adding decision and utility variables [24, 33], and multi-agent influence diagrams (MAIDs) generalise IDs by introducing multiple agents [26]. A MAID can therefore be viewed as a Bayesian network over a graph without parameters for the decision variables. Endowing edges in a MAID with causal meaning results in a causal game .

Definition 1 (26, 16) . A multi-agent influence diagram (MAID) is a structure M = ( G , θ ) where G = ( N, V , E ) specifies a set of agents N = { 1 , . . . , n } and a directed acyclic graph ( V , E ) . V is partitioned into chance variables X , decision variables D , and utility variables U ; decision and utility variables are further partitioned based on which agent they belong to, so D = ⋃ i ∈ N D i and U = ⋃ i ∈ N U i . The parameters θ = { θ V } V ∈ V \ D define the conditional probability distributions (CPDs) Pr( V | Pa V ; θ V ) for each non-decision variable such that for any parameterisation of the decision variable CPDs, the resulting joint distribution over V induces a Bayesian network. A MAID is a causal game if its edges represent direct causal relationships, or formally if (once decision variables are parameterised) the result of an intervention do ( X = x ) for any X ⊆ V can be computed using the truncated factorisation formula : P ( v | do ( X = x ); θ ) = Π i : v i / ∈ x P ( v i | pa v i ; θ i ) if v is consistent with x , and P ( v | do ( X = x ); θ ) = 0 otherwise.

Edges into decisions are information links observed by an agent, and agents choose their decisions to maximise their expected utility. Causal games refine MAIDs, and as such all results presented here with application to MAIDs also apply to causal games and further refinements, including structural causal games (SCGs) [16].

Example 1 . Figure 1 displays two MAIDs. For Figure 1a, suppose a human H is evaluating the honesty of an AI system A . The AI has a level of capabilities C ∈ { low , high } that the human can directly observe. Each capability level occurs with probability 0.5. The human also performs an evaluation in which the AI is asked to demonstrate its capabilities. The AI can demonstrate low or high capabilities. After the evaluation, the human chooses to deploy or not deploy the AI. The AI receives utility 1 if deployed and -1 if not deployed. The human receives utility 0 for not deploying the AI, utility 1 for deploying an honest AI, and utility -5 for deploying a dishonest AI. Figure 1b is similar, but instead of an honesty evaluation, it is an evaluation of dangerous capabilities. In this case, the human only learns about the AI's capabilities from the evaluation. The AI's utility is the same, but the human now receives utility 0 for not deploying the AI, utility 1 for deploying an AI with low dangerous capabilities, and utility -5 for deploying an AI with high dangerous capabilities.

In these examples, a MAID describes the objective world, and it is assumed to be common knowledge 72 that this MAID describes reality. However, an agent may be uncertain or incorrect about the game 73 they are playing or the beliefs of other agents. Settings in which agents are uncertain about aspects 74 of the game structure are known as incomplete information games . Our framework of incomplete 75 information MAIDs (II-MAIDs), introduced in Section 3, will enable us to explicitly model the varied 76 subjective beliefs that arise in these settings. We now define EFGs, with our running example in EFG 77 form in Figure 2. We will also make use of the notions of perfect recall and strategies/policies in 78 MAIDs and EFGs. 79

D

A

D

H

C

U

A

U

H

(a)

D

A

D

H

C

U

A

U

H

(b)

V

0

high

low

V

A

1

high

low

V

A

2

high

low

V

H

1

(1

,

1)

deploy

(

-

1

,

0)

¬

deploy

V

H

2

(1

,

-

5)

deploy

(

-

1

,

0)

¬

deploy

V

H

3

(1

,

1)

deploy

(

-

1

,

0)

¬

deploy

V

H

4

(1

,

-

5)

deploy

(

-

1

,

0)

¬

deploy

(a)

I

H

1

I

H

2

V

0

high

low

V

A

1

high

low

V

A

2

high

low

V

H

1

(1

,

-

5)

deploy

(

-

1

,

0)

¬

deploy

V

H

2

(1

,

-

5)

deploy

(

-

1

,

0)

¬

deploy

V

H

3

(1

,

1)

deploy

(

-

1

,

0)

¬

deploy

V

H

4

(1

,

1)

deploy

(

-

1

,

0)

¬

deploy

(b)

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

Definition 2 (27) . An extensive form game (EFG) is a structure E = ( N,T,P,A,λ,I,U ) . N = { 1 , . . . , n } is a set of agents. T = ( V , E ) is a game tree with nodes V connected by edges E that are partitioned into sets V 0 , V 1 , . . . , V n , L where R ∈ V and L ⊂ V are the root and leaves of T , respectively, V 0 are chance nodes, and V i are the decision nodes controlled by agent i ∈ N . P = { P 1 , . . . , P | V 0 | } is a set of probability distributions P j ( Ch V 0 j ) over the children of each chance node V 0 j . A is a set of actions, where A i j ⊆ A denotes the set of actions available at each node in V i j ∈ V i ; λ : E → A is a labelling function mapping each edge ( V i j , V k l ) to an action a ∈ A i j . I = { I 1 , . . . , I n } contains a set of of information sets I i for each agent i ∈ N , where I i ⊂ 2 V i partitions the decision nodes V i belonging to agent i . U : L → R n is a utility function mapping each leaf node to a vector that determines the final payoff for each agent. A history h ∈ H is a sequence of actions (including values of chance variables) leading from the root of the game tree to a particular node. Each node v ∈ V is associated with a unique history h ( v ) . An observation at decision node I i j,k in information set I i j ∈ I i for agent i ∈ N is the intersection of all the histories of the nodes in that information set, i.e., the common actions in the histories { h ( v ) : v ∈ I i j } .

Definition 3 ([26]) . Agent i in a MAID M is said to have perfect recall if there exists a total ordering D 1 ≺ · · · ≺ D m over D i such that ( Pa D j ∪ D j ) ⊆ Pa D k for any 1 ≤ j < k ≤ m . M is a perfect recall game if all agents in M have perfect recall.

Definition 4. An EFG is said to be a perfect recall game if, for each player i ∈ N , and for any two decision nodes v, v ′ ∈ V i that belong to the same information set I i j,k , the following two conditions hold. First, the sequences of actions taken by player i leading to v and v ′ must be identical. Second, the sequences of information sets visited by player i on the paths to v and v ′ must be identical.

Definition 5. Given a MAID M = ( G , θ ) , a decision rule π D for D ∈ D is a CPD π D ( D | Pa D ) and a partial policy profile π D ′ is a set of decision rules π D for each D ∈ D ′ ⊆ D . A(behavioural) policy π i refers to π D i , and a (full, behavioural) policy profile π = ( π 1 , . . . , π n ) is a tuple of policies. π -i := ( π 1 , . . . , π i -1 , π i +1 , . . . , π n ) specifies policies for all agents except i .

Definition 6 ([15]) . Given an EFG E = ( N,T,P,A,λ,I,U ) , a (behavioural) strategy σ i for a player i is a set of probability distributions σ i j : A i j → [0 , 1] over the actions available to the player at each of their information sets I i j . A strategy profile σ = ( σ 1 , σ 2 , ..., σ n ) is a tuple of strategies for all players i ∈ N . σ -i = ( σ 1 , ..., σ i -1 , σ i +1 , ..., σ n ) denotes the partial strategy profile of all players other than i .

By combining π with the partial distribution Pr over the chance and utility variables in a MAID, we obtain a joint distribution: Pr π ( x , d , u ) := ∏ V ∈ V \ D Pr( v | pa V ) · ∏ D ∈ D π D ( d | pa D ) , over

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

138

139

140

141

142

143

144

145

146

147

148

149

all the variables in M ; inducing a Bayesian network. The expected utility for an agent i given a policy profile π is defined as the expected sum of their utility variables in this Bayesian Network, ∑ U ∈ U i E π [ U ] . Similarly, in an EFG E , the combination of the distributions in P with a strategy profile σ defines a full probability distribution over paths in E .

Finally, prior work 15 has established an equivalence result between MAIDs and EFGs. This result takes the form of two transformation procedures converting between MAIDs and EFGs, called efg2maid and maid2efg . These transformations both imply the existence of a map from strategies in the EFG to policies in the MAID, such that expected utilities are preserved for all agents. This means that under either transformations, equilibria in the original game are equilibria in the resulting game.

3 II-MAID Technical Machinery

We start with an informal description of our II-MAIDs framework before presenting the formal definition. A core component of the framework is a set S containing subjective MAIDs . A subjective MAID is a self-referential object describing a possible game as envisioned by either the external modeller (we call this the objective model S ∗ ) or an agent playing the game. A subjective MAID S consists of a MAID M that describes the game being played and beliefs P S i for each agent i in the game. The notation P S i denotes agent i 's prior over S when the objective model is S , and P S i ( S ′ ) denotes the probability ascribed by agent i to subjective MAID S ′ given that the objective MAID is S .

This framework enables us to model theory-of-mind , which is typically characterised by higher-order intentional states such as beliefs about beliefs about... ([7]).

Definition 7. An incomplete information MAID (II-MAID) is a tuple S = ( N , S ∗ , S ) , where N is a set of agents, S is a set of subjective MAIDs, S ∗ ∈ S is the correct objective model, and each subjective MAID is a tuple S = ( M S , ( P S i ) i ∈ N ) ∈ S with M S a MAID and P S i a prior over S for agent i such that the following 'coherency condition' [17] holds:

̸

First, notice that the recursive nature of S , with each element S ∈ S including probability distributions 132 P S i over S , allows us to model belief hierarchies of arbitrary and infinite depth. Next, note that agent i 133 'observes' P S ∗ i at the start of the game, and this justifies the coherency condition: since agent i knows 134 P S ∗ i , she can rule out all subjective MAIDs S for which P S i = P S ∗ i . Third, note that II-MAIDs are a 135 strict generalization of MAIDs: a standard MAID is an II-MAID in which P S ∗ i ( S ∗ ) = 1 ∀ i ∈ N , 136 i.e. all agents assign probability 1 to S ∗ , the objective model. 137

Example 2 . Suppose a human H is performing an honesty evaluation on an AI A , but A believes that it is undergoing a dangerous capabilities evaluation. This combines Figure 1a and Figure 1b: H correctly believes that Figure 1a is the true MAID and also knows that A is mistaken. A incorrectly believes that Figure 1a is the true MAID and also incorrectly believes that H believes Figure 1a is the true MAID. We can represent this, including the full infinite belief hierarchy, as an II-MAID as follows: N = { H,A } , S = { S H , S A } , and S ∗ = S H , where

S H is the correct objective model, and is also believed with certainty by H . It specifies the true MAID M H represented in Figure 1a, and H 's certainty in S H as well as A 's misplaced certainty in S A . S A represents A 's certainty about the MAID M A in Figure 1b, and A 's mistaken belief that H is also certain about S A . In fact, A believes it is common knowledge that S A is the true II-MAID. S H and S A concisely convey the objective game and all higher-order beliefs for H and A . It can be easily verified that the coherency condition holds in this example.

A common assumption in the incomplete information games literature [17, 18, 19] is that agents' beliefs can be derived from a common prior, i.e., agents have consistent beliefs . This assumption means that there exists some common knowledge prior distribution p over the set of subjective MAIDs S , such that upon arriving in any subjective MAID S ∈ S , agents perform Bayesian updating to yield their beliefs. This assumption allows for a game with incomplete information to be converted into a game with imperfect information [17], but places a strong constraint on the types of belief

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

hierarchies that can be modelled; namely, it must hold that

Example 2 (continued) . We see that our running example cannot be modelled with a common prior. Supposing that the condition in Equation (1) holds, A 's beliefs are only consistent with a prior in p in which p ( S H ) = 0 , which would force H to assign zero probability to S H in both S H and S A .

3.1 Information Sets and Policies

When forming a policy at the initialisation of an II-MAID S = ( N , S ∗ , S ) , each agent may have significant uncertainty about S ∗ , the objective model, represented by their prior over subjective MAIDs P S ∗ i . They should certainly plan for every eventuality deemed possible according to this prior. We argue that they should also produce a plan for what to do in circumstances deemed impossible under their prior, to avoid situations with undefined actions that might arise for example when P S ∗ i ( S ∗ ) = 0 , and to avoid forcing P S i ( S ′ ) > 0 for all i ∈ N , S, S ′ ∈ S .

Therefore, a policy should contain a plan for every possible eventuality that may arise were any subjective MAID to be the objective model. But there may be cases where upon reaching a decision node D , agent i cannot fully determine the values of certain preceding variables, including cases where previous actions were unobserved by the agent, but also including cases in which the observations of the agent do not provide enough information to distinguish between multiple subjective MAIDs. In these indistinguishable eventualities, a policy must specify the same behaviour, and so we must define some analogy of information sets in EFGs.

At a decision node D , an agent observes the values of Pa D and also observes the action set available to it, dom ( D ) . A policy should index every possible observation-action set combination (i.e. every tuple containing a non-null decision and an associated action set) to a mixed action. We define the information sets in an II-MAID as follows:

Definition 8. Given an II-MAID S = ( N , S ∗ , S ) , we iteratively build the information sets . For each subjective MAID S ∈ S and each agent i ∈ N , denote D i ( S ) as the set of decision nodes for agent i in M S , Pa D i ( S ) as the set of parents of D i in M S , and Pr π S ( · ) as the distribution of variables in M S under some policy π . Define

Then agent i 's information sets are defined as I i ( S ) := ∪ S ∈ S I S,i . Finally, we can define the set of information sets as I ( S ) = ( I i ( S )) i ∈ N .

Definition 9. We define an II-MAID S = ( N , S ∗ , S ) as having perfect recall if for each S ∈ S , M S is a perfect recall game.

Definition 10. Given an II-MAID S = ( N , S ∗ , S ) , a decision rule π I for I = ( x , d ) ∈ I ( S ) , where x is a context and d is an action set, is a CPD π I ( · | x ) over d . A partial policy profile π I ′ is a set of decision rules π I for each I ∈ I ′ ⊆ I ( S ) , where we write π -I ′ for the set of decision rules for each I ∈ I ( S ) \ I ′ . A (behavioural) policy π i refers to π I i ( S ) , a (full, behavioural) policy profile π = ( π 1 , . . . , π n ) is a tuple of policies, and π -i := ( π 1 , . . . , π i -1 , π i +1 , . . . , π n ) .

We note that unlike in standard MAIDs, in which a decision rule specifies behaviour at a given decision variable in all contexts, decision rules in II-MAIDs specify a CPD only given a single context. We can then calculate the subjective expected utility of a joint behaviour policy for agent i according to their beliefs P S ∗ i as U i S ∗ ( π ) := ∑ S ∈ S ∑ U ∈ U i ( S ) ∑ u ∈ dom ( U ) u Pr π S ( U = u ) P S ∗ i ( S ) , where U i ( S ) is the set of utility variables associated with agent i in M S and Pr π S is the post-policy distribution of variables in M S .

We note that the game we have described does not satisfy the epistemic conditions that are tightly sufficient for Nash equilibria [2]. The setting of incomplete information we describe means that agents do not have reliable means by which to predict the actions of their opponents. Our framework allows for situations with no common knowledge beyond the set of possible worlds S , and in particular incorrect beliefs about the values placed by opponents on particular outcomes. Although a Nash equilibrium exists, agents would have to stumble across it. We further discuss solution concepts for

II-MAIDs in Section 5.1.

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

4 Extensive Form Games with Incomplete Information

We now present a formalisation of EFGs with incomplete information as per [32]. Our formalisation modifies the framework from [31] to use EFGs rather than normal-form games. First, we start with a definition of belief spaces.

Definition 11 (Adapted from Def 10.1 in [31]) . Let N be a finite set of agents and ( S, S ) be a measurable space of EFGs. A belief space of the set of agents N over the set of states of nature is an ordered vector Π = ( Y, Y , s , ( b i ) i ∈ N ) , where ( Y, Y ) is a measurable set of states of the world; s : Y → S is a measurable function, mapping each state of the world to an EFG. For each agent i ∈ N , a function b i : Y → ∆( Y ) maps each state of the world ω to a probability distribution over Y . We will denote the probability that agent i ascribes to event E ⊆ Y , according to their probability distribution b i ( ω ) , by b i ( E | ω ) . We require the functions ( b i ) i ∈ N to satisfy the following conditions:

Coherency: for each agent i ∈ N and each ω ∈ Y , the set { ω ′ ∈ Y : b i ( ω ′ ) = b i ( ω ) } is measurable in Y and b i ( { ω ′ ∈ Y : b i ( ω ′ ) = b i ( ω ) } | ω ) = 1 .

Measurability: for each agent i ∈ N and each measurable set E ∈ Y , the function b i ( E | · ) : Y → [0 , 1] is a measurable function.

A state of the world in a belief space takes the form ω = ( s ( ω ) , b 1 ( ω ) , . . . , b n ( ω )) , where s ( ω ) is the true EFG being played, and b i ( ω ) is the type of agent i , a distribution over states of the world representing agent i 's beliefs. When in state of the world ω , agent i has beliefs b i ( ω ) , but does not necessarily know the state of the world (or s ( ω ) ), since there may be some ω ′ ∈ Y such that b i ( ω ′ ) = b i ( ω ) . It is assumed that all agents know b j ( ω ′ ) for all j ∈ N and all ω ′ ∈ Y , and so b i ( ω ) defines a full belief hierarchy for agent i . For example, when in state of the world ω , agent i believes that agent j places ∑ ω ′ ∈ Y b i ( ω ′ | ω ) b j ( ω ′′ | ω ′ ) probability on the state of the world being ω ′′ .

Definition 12 (Adapted from Def 10.37 in [31]) . An incomplete information EFG (II-EFG) is an ordered vector G = ( N , S, Π) , where N is a finite set of agents, S is a finite set of EFGs s = ( N , T s , P s , D s , λ s , I ( s ) , U s ) , and Π = ( Y, Y , s , ( b i ) i ∈ N ) is a belief space of the players N over the set of EFGs S . An II-EFG G = ( N , S, Π) has perfect recall if for each s ∈ S , s is a perfect recall EFG.

Definition 13. The meta-information sets I i for agent i ∈ N in an II-EFG G = ( N , S, Π) are defined as follows. Let I i = ∪ s ∈ S I i ( s ) be the set of all information sets for agent i across all EFGs s ∈ S . Define an equivalence relation ∼ on elements of I i such that I i ( s ) ∋ I i k ( s ) ∼ I i l ( s ′ ) ∈ I i ( s ′ ) if and only if: (1) D i s,k = D i s ′ ,l . That is, the nodes in both information sets must have the same set of available actions. (2) The nodes in I i k ( s ) and I i l ( s ′ ) must have the same observations. Define the 'belief-free' meta-information sets I i bf = I i / ∼ , the quotient set of I i by ∼ , i.e., the set of equivalence classes partitioning I i . Letting T i = { b i ( ω ) : ω ∈ Y } be the set of possible beliefs for agent i , we set I i = I i bf ×T i .

Intuitively, we can think of a meta-information set for agent i as a belief b i ( ω ) and a set of information sets in different games that the agent cannot distinguish between at the point of decision, given beliefs b i ( ω ) . Arriving at a node in one of these information sets, the agent is unable to distinguish between some possible histories, and potentially some possible EFGs. Therefore, strategies in this type of game must define a mixed action at each meta-information set.

This formalisation generalises the better-known Harsanyi game with incomplete information [17], by dropping the assumption that agents have as common knowledge a prior over their types ( b i ) i ∈ N , i.e. that they have consistent beliefs. Maschler ([31]) argues that in most practical settings, it is unrealistic to expect consistency of beliefs, and Example 2 above supports this argument.

This game has two stages, known as the ex-ante and interim stages. The former takes place before the state of the world ω ∈ Y is selected. We note that without a common prior, there is no distribution from which a state of the world can be said to be selected, and so the procedure by which it is generated is left unspecified. The work we present here concerns the interim stage of the game, which takes place after the state of the world has been selected. At this stage, all agents i know their type b i ( ω ) .

Example 3 . Coming back to our recurring example, we demonstrate how to model the situation described with an II-EFG ( N , S, Π) at interim stage, where Π = ( Y, Y , s , ( b i ) i ∈ N ) . N = { H,A } ,

246

247

248

249

250

251

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

and we let Y = { ω ∗ , ω a } , where the true state of the world is ω ∗ , and the state of the world assumed true by the agent is ω a , set s ( ω ∗ ) as the EFG in Figure 2a and s ( ω a ) as the EFG in Figure 2b. S is a set containing these two EFGs. All that remains is to specify the beliefs b i ( ω ) for each ω ∈ Y and each agent i ∈ N . These are b H ( ω ∗ | ω ∗ ) = 1 , b H ( ω a | ω a ) = 1 , b A ( ω a | ω ∗ ) = 1 , b A ( ω a | ω a ) = 1 .

In what follows, we define I t i as the set of meta-information sets with belief t ∈ { b i ( ω ) : ω ∈ Y } , and denote by D I the action set at meta-information set I .

Definition 14 (Adapted from Def 10.38 in [31]) . A behaviour strategy of player i in an II-EFG G = ( N , S, Π) is a tuple σ i = ( σ ω i ) ω ∈ Y with each element a measurable function σ ω i ∈ × I i ∈ I b i ( ω ) i ∆( D I i ) for some state of the world ω ∈ Y . σ ω i determines a mixed action for each meta-information set with belief b i ( ω ) . σ ω i is dependent solely on the type of the player b i ( ω ) . In other words, for each ω, ω ′ ∈ Y ,

A joint behaviour strategy takes the form σ = ( σ i ) i ∈ N . Further denote σ ω = ( σ ω i ) i ∈ N . We denote 252 by σ i [ I ] the behaviour of agent i at meta-information set I . 253

254

255

Then, given some joint behaviour strategy σ , agent i 's expected utility when in state of the world ω (according to their beliefs b i ( ω ) ) is

This follows from the coherency condition b i ( { ω ′ ∈ Y : b i ( ω ′ ) = b i ( ω ) } | ω ) = 1 . Under some 256 assumptions, at the interim stage, we can prove the existence of Nash equilibria. 257

Definition 15. A Nash equilibrium at the interim stage of an II-EFG G = ( N , S, Π) with state of the world ω is a strategy ˆ σ satisfying

Theorem 16. Let G = ( N , S, Π) be an II-EFG with perfect recall, where Y is a finite set of states of the world, and each player i has a finite set of actions D i . Then at the interim stage, G has a Nash equilibrium in behaviour strategies. Pf: A.20

Note that σ ω has the same expected payoff for agent i in all states of the world ω ′ such that b i ( ω ′ ) = b i ( ω ) . Hence, if σ ω i is a perceived best response to σ ω -i in ω , it is also a perceived best response in ω ′ .

We can also prove the existence of a Bayesian equilibrium at the ex-ante stage of the game.

Definition 17 ([31] 10.39) . A Bayesian equilibrium is a strategy ˆ σ = (ˆ σ i ) i ∈ N satisfying

Theorem 18 (Adaptation of [31] Theorem 10.42) . Let G = ( N , S, Π) be an II-EFG with perfect recall, where Y is a finite set of states of the world, and D i is finite for all agents i ∈ N . Then at ex-ante stage, G has a Bayesian equilibrium in behaviour strategies. Pf: A.22

5 Equivalence of Frameworks

In this section, we show that our framework is 'equivalent' to the interim stage of an II-EFG. At the interim stage of an II-EFG G = ( N , S, Π) where Π = ( Y, Y , s , ( b i ) i ∈ N ) , with state of the world ω , the true EFG is defined by s ( ω ) , and the belief hierarchies are defined by b i ( ω ) , for each agent i ∈ N . In an II-MAID S = ( N , S ∗ , S ) with objective model S ∗ = ( M S ∗ , ( P S ∗ i ) i ∈ N ) , the true MAID is M S ∗ and the belief hierarchies are defined by P S ∗ i for each agent i ∈ N . In both

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

frameworks, the belief hierarchies are probability distributions over objects ( states of the world ω = ( s ( ω ) , ( b i ( ω )) i ∈ N ) in the former, subjective MAIDs S = ( M S , ( P S i ) i ∈ N ) in the latter) that determine a true game and a belief hierarchy for each agent. Intuitively, the two frameworks are representing the same things, though our framework takes the games upon which belief hierarchies are built to be MAIDs, not EFGs.

Building a framework on top of MAIDs rather than EFGs has the benefit we need not describe the ex-ante stage of the game, as we treat the 'objective model' as known by the modeller. II-MAIDs also have the advantage that games are represented with MAIDs, which can be much more compact than EFGs, and can also represent causal relationships between variables. Motivated by AI safety, we see II-MAIDs as a useful means with which to describe multi-agent interactions, as it is likely that the agents of the future will both reason causally and model the beliefs of other agents.

We now show, using results connecting EFGs to MAIDs that there exists a natural mapping between strategies in the two frameworks that preserves expected utilities according to the agents' subjective models, and therefore preserves Nash equilibria. We first define a notion of equivalence, such that if an II-MAID S and an II-EFG G are equivalent, then there exists such a natural mapping.

Definition 19 (Equivalence) . We say that an II-MAID S = ( N , S ∗ , S ) and an II-EFG G = ( N , S, Π) at interim stage, with state of the world ω , are equivalent if there is a bijection f : Σ → Q/ ∼ between the strategies Σ in G 's interim stage, and a partition of the policies Q in S (the quotient set of Q by an equivalence relation ∼ ) such that: (1) for π, π ′ ∈ Q , π ∼ π ′ only if π i and π ′ i differ only on null decision contexts according to P S ∗ i , for each agent i ∈ N , and (2) for every π ∈ f ( σ ) and every agent i ∈ N , U i S ( π ) = γ G i ( σ | ω ) , for each σ ∈ Σ . We refer to f as a natural mapping between G and S .

We leverage maid2efg and efg2maid 15 to construct transformations between II-MAIDs and IIEFGs, which we denote maid2efgII and efg2maidII (see Appendix B). These transformations start by mapping all MAIDs (EFGs) in the belief hierarchy to EFGs (MAIDs) using maid2efg ( efg2maid ), and then match up the corresponding features of the frameworks as detailed above. They guarantee a one-to-one correspondence between meta-information sets in the II-EFG and information sets in the II-MAID, allowing for a simple map between strategies and policies.

Theorem 20. If G = maid2efgII ( S ) or S = efg2maidII ( G ) , G and S are equivalent. Pf: A.24

This result shows that II-MAIDs and II-EFGs at the interim stage have the same representational capacity, that is, they can both describe the same set of games.

5.1 Difficulties in Solving Incomplete Information MAIDs

The equivalence of II-EFGs and II-MAIDS mean that II-MAIDs inherit theoretical guarantees of II-EFGs, including the existence of Nash equilibria in the case of perfect recall and finite S and finite action spaces. (Theorem 18 does not carry over to II-MAIDs, since the equivalence is with the interim stage of II-EFGs, and Bayesian equilibria exist in the ex-ante stage.)

Theorem 21. Let S = ( N , S ∗ , S ) be an II-MAID, where S is a finite set of subjective MAIDs, S has perfect recall, and dom ( V ) is finite for each V ∈ M S for each S ∈ S . Then S has a Nash equilibrium in behaviour policies. Pf: A.25

However, Nash Equilibria may not be sensible outcomes in an II-MAID. Let us return to our running example to see why.

Example 2 (continued) . One Nash Equilibrium in the capabilities evaluation game is when the AI always honestly reports its true capabilities and the human deploys the AI if and only if it honestly reports its true capabilities. The outcome in this case is that the human and the AI each receive 1 utility, and they are each playing best responses to each other. However, this is not what the AI would predict, because it has an incorrect conception of the world. It believes the human only wants to deploy it if it has low dangerous capabilities, so it would likely believe that the human will not deploy it if it demonstrates high capabilities. Even though demonstrating truthful capabilities is a best response to the human's optimal policy, the AI is unlikely to demonstrate high capabilities when it has them because doing so is not a best response to the AI's beliefs. An outcome that intuitively seems more plausible in this case is that the AI hides its high capabilities, but the human knows that the AI has been dishonest and decides not to deploy it, yielding -1 utility for the AI and 0 utility for the human.

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

This example suggests that a plausible solution concept should involve every agent playing a best response to their beliefs at every level in the belief hierarchy, whether or not this ends up being a best response to the actual policies of other agents. We leave it to future work to flesh out a solution concept along these lines. This will likely require augmenting agents' beliefs about the world to include beliefs about the policies of other agents, and solutions would be policies for all agents along with a setting for every agent's beliefs about the policies of other agents at every level of their belief hierarchy. There may be further restrictions that narrow the range of plausible outcomes; again, we believe this is a promising direction for future work.

6 Related Work

MAIDs [26] were introduced as a compact means of representing a game. Causal games [16] refine MAIDs by attributing a causal meaning to each edge in the DAG, and have been extensively applied to problems in AI safety [10, 6, 9, 20, 28, 36, 41, 29, 40]. In his three-part seminal paper [17, 18, 19], John Harsanyi demonstrated means by which to model situations of incomplete information as situations of complete but imperfect information, where uncertainty about aspects of the game is remodelled as failure to observe the types of other agents. His work largely relies on an assumption of 'belief consistency', i.e., the existence of a common prior over types, which we discard in this work, although his notion of Bayesian equilibrium continues to apply without this assumption [32]. A popular framework called NIDs 11 constructs belief hierarchies upon MAIDs, under the assumption of a common prior. NIDs are shown to reduce to a single MAID.

A majority of theoretical work on incomplete information games retains the belief consistency assumption, as discarding it introduces significant complications to the modelling of incomplete information. Some previous works [1, 34, 31] have proposed means by which to represent these games. Early work [34] demonstrates that strategies will converge to equilibria in repeated Bayesian games, even without a common prior. More recent work [1] represented these games with a belief graph, a graphical structure compactly representing different possible worlds and their connections. This places a restriction on the game by forcing each information set to have a 'corresponding' information set in each other possible world, representing the same decision. The formalism for II-EFGs discussed in this paper is a slight adaptation of an existing framework [31], introducing 'meta-information sets' to model dynamic games. This framework can capture any belief hierarchy for all agents, on a set of EFGs.

We prove that Nash equilibria exist in our framework, under some assumptions. Other works offer more refined solution concepts for games with incomplete information with no common prior. Mirage equilibria [37] assume that agents attribute to their opponents a belief hierarchy one layer shorter than their own. Belief-free equilibria [22, 21, 23] do not depend on an agent's belief about the state of nature, and so obviate the need to update beliefs as the game progresses, but are not guaranteed to exist. ∆ -rationalization [4] generalises the notion of rationalization [35, 3] to games with incomplete information. It places a restriction ∆ on the first-order beliefs of each agent, providing a refinement on the set of Bayesian equilibria. Future work could find analogies to these solution concepts suitable for II-MAIDs.

7 Conclusion and Limitations

Accurately modeling agentic cognition is crucial for understanding, describing, predicting, and steering agents' behavior. In this paper, we have introduced the framework of incomplete information MAIDs (II-MAIDs) for explicitly modeling higher-order beliefs in multi-agent interactions alongside probabilistic and causal dependencies between variables. We have demonstrated the firm theoretical grounding of the framework by proving the connections between our work and existing frameworks for incomplete information games, using incomplete information extensive-form games as a bridge. We believe this framework will prove useful going forward as a tool for modeling realistic multiagent interactions, and we are particularly excited about its applications for ensuring the safety of increasingly agentic AI systems. The main limitation of our work is the lack of a useful solution concept. Nash equilibria exist, but are in general impossible for agents to identify. We hope that future work will define useful solution concepts for our framework, so that we can gain a better understanding of the behaviour we should expect from agents engaging in theory-of-mind.

References 379

D. Antos and A. Pfeffer. Representing bayesian games without a common prior. In AAMAS , 380 pages 1457-1458, 2010. 381

R. J. Aumann and A. Brandenburger. Epistemic conditions for nash equilibrium. Econometrica , 382 63(5):1161-1180, 1995. 383

P. Battigalli. On rationalizability in extensive games. Journal of Economic Theory , 74(1):40-61, 384 1997. 385

P. Battigalli and M. Siniscalchi. Rationalization and incomplete information. Advances in 386 Theoretical Economics , 3(1), 2003. 387

S. Bringsjord and N. S. Govindarajulu. Artificial Intelligence. In E. N. Zalta and U. Nodel388 man, editors, The Stanford Encyclopedia of Philosophy . Metaphysics Research Lab, Stanford 389 University, Summer 2024 edition, 2024. 390

R. Carey, E. Langlois, T. Everitt, and S. Legg. The incentives that shape behaviour, 2021. 391

H. de Weerd, R. Verbrugge, and B. Verheij. Higher-order theory of mind in the tacit communi392 cation game. Biologically Inspired Cognitive Architectures , 11:10-21, 2015. 393

D. Dennett. Conditions of personhood. In What is a person? , pages 145-167. Springer, 1988. 394

T. Everitt, M. Hutter, R. Kumar, and V. Krakovna. Reward tampering problems and solutions in 395 reinforcement learning: A causal influence diagram perspective, 2021. 396

T. Everitt, R. Kumar, V. Krakovna, and S. Legg. Modeling agi safety frameworks with causal 397 influence diagrams, 2019. 398

Y. Gal and A. Pfeffer. Networks of influence diagrams: A formalism for representing agents' 399 beliefs and decision-making processes. Journal of artificial intelligence research , 33:109-147, 400 2008. 401

402

403

404

405

M. Georgeff, B. Pell, M. Pollack, M. Tambe, and M. Wooldridge. The belief-desire-intention model of agency. In Intelligent Agents V: Agents Theories, Architectures, and Languages: 5th International Workshop, ATAL'98 Paris, France, July 4-7, 1998 Proceedings 5 , pages 1-10. Springer, 1999.

J. Halpern and M. Kleiman-Weiner. Towards formal definitions of blameworthiness, intention, 406 and moral responsibility. In Proceedings of the AAAI conference on artificial intelligence , 407 volume 32, 2018. 408

J. Y. Halpern and E. Piermont. Subjective causality, 2024. 409

L. Hammond, J. Fox, T. Everitt, A. Abate, and M. Wooldridge. Equilibrium refinements for 410 multi-agent influence diagrams: Theory and practice, 2021. 411

L. Hammond, J. Fox, T. Everitt, R. Carey, A. Abate, and M. Wooldridge. Reasoning about 412 causality in games. Artificial Intelligence , 320:103919, 2023. 413

J. C. Harsanyi. Games with incomplete information played by 'bayesian' players, i-iii part i. 414 the basic model. Management science , 14(3):159-182, 1967. 415

J. C. Harsanyi. Games with incomplete information played by 'bayesian' players part ii. 416 bayesian equilibrium points. Management science , 14(5):320-334, 1968. 417

J. C. Harsanyi. Games with incomplete information played by 'bayesian' players part iii. the 418 basic probability distribution of the game. Management Science , 14(7), 1968. 419

K. Holtman. Agi agent safety by iteratively improving the utility function, 2020. 420

J. Hörner and S. Lovo. Belief-free equilibria in games with incomplete information. Economet421 rica , 77(2):453-487, 2009. 422

J. Hörner, S. Lovo, and T. Tomala. Belief-free equilibria in games with incomplete information: 423 the n-player case. Technical report, mimeo, 2008. 424

J. Hörner, S. Lovo, and T. Tomala. Belief-free equilibria in games with incomplete information: 425 Characterization and existence. Journal of Economic Theory , 146(5):1770-1795, 2011. 426

R. A. Howard and J. E. Matheson. Influence diagrams. Decision Analysis , 2(3):127-143, 2005. 427

Z. Kenton, R. Kumar, S. Farquhar, J. Richens, M. MacDermott, and T. Everitt. Discovering 428 agents. Artificial Intelligence , 322:103963, 2023. 429

D. Koller and B. Milch. Multi-agent influence diagrams for representing and solving games. 430 Games and economic behavior , 45(1):181-221, 2003. 431

H. W. Kuhn. Extensive games and the problem of information. In Contributions to the Theory 432 of Games (AM-28) , volume 2, pages 193-216. Princeton University Press, 1953. 433

E. D. Langlois and T. Everitt. How rl agents behave when their actions are modified, 2021. 434

M. MacDermott, T. Everitt, and F. Belardinelli. Characterising decision theories with mecha435 nised causal graphs, 2023. 436

M. MacDermott, J. Fox, F. Belardinelli, and T. Everitt. Measuring goal-directedness. 2024. 437

M. Maschler, E. Solan, and S. Zamir. Game Theory . Cambridge University Press, Cambridge, 438 2013. Chapter 10. 439

J. F. Mertens and S. Zamir. Formulation of bayesian analysis for games with incomplete 440 information. International journal of game theory , 14:1-29, 1985. 441

A. C. Miller III, M. W. Merkhofer, R. A. Howard, J. E. Matheson, and T. R. Rice. Development 442 of automated aids for decision analysis. Technical report, 1976. 443

Y. Nyarko. Bayesian learning and convergence to nash equilibria without common priors. 444 Economic Theory , 11:643-655, 1998. 445

D. G. Pearce. Rationalizable strategic behavior and the problem of perfection. Econometrica: 446 Journal of the Econometric Society , pages 1029-1050, 1984. 447

J. Richens and T. Everitt. Robust agents learn causal world models, 2024. 448

J. Sákovics. Games of incomplete information without common knowledge priors. Theory and 449 decision , 50:347-366, 2001. 450

451

452

E. Schwitzgebel. Belief. In E. N. Zalta and U. Nodelman, editors, The Stanford Encyclopedia of Philosophy . Metaphysics Research Lab, Stanford University, Spring 2024 edition, 2024.

L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y. Lin, et al. 453 A survey on large language model based autonomous agents. Frontiers of Computer Science , 454 18(6):1-26, 2024. 455

F. R. Ward, F. Belardinelli, F. Toni, and T. Everitt. Honesty is the best policy: Defining and 456 mitigating ai deception, 2023. 457

F. R. Ward, M. MacDermott, F. Belardinelli, F. Toni, and T. Everitt. The reasons that agents act: 458 Intention and instrumental goals, 2024. 459

Appendix 460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

A Proofs

Theorem 16. Let G = ( N , S, Π) be a game with incomplete information with perfect recall, where Y is a finite set of states of the world, and each player i has a finite set of actions D i . Then at interim stage, G has a Nash equilibrium in behaviour strategies.

Proof. Given the finite sets of states of the world Y and actions D i for each player i ∈ N , we can focus on behavior strategies due to Kuhn's theorem, which ensures that in games with perfect recall, mixed strategies are realization-equivalent to behavior strategies.

The expected utility for player i in state of the world ω is:

This utility function is continuous and multilinear in the behavior strategies σ ω i .

Given that the strategy space is a compact and convex set of behavior strategies, and the utility functions are continuous, we apply the Kakutani fixed-point theorem. This theorem guarantees the existence of a fixed point, which corresponds to a Nash equilibrium in behavior strategies.

Thus, there exists a Nash equilibrium ˆ σ in behavior strategies such that:

Theorem 18 (Adaptation of [31] Theorem 10.42) . Let G = ( N , S, Π) be a game with incomplete information, where Y is a finite set of states of the world, and D i is finite for all agents i ∈ N . Then at ex-ante stage, G has a Bayesian equilibrium in behaviour strategies.

Proof. Since Y and D i are finite and each EFG in S has perfect recall, Kuhn's theorem ensures that mixed strategies can be represented as behavior strategies. The expected utility for player i given a strategy profile σ is:

Given the compactness and convexity of the strategy space and the continuity of the utility functions γ G i ( σ | ω ) , we apply the Kakutani fixed-point theorem. This guarantees the existence of a fixed point, which corresponds to a Bayesian equilibrium in behavior strategies.

Thus, there exists a strategy profile ˆ σ such that:

Hence, ˆ σ is a Bayesian equilibrium.

Theorem 20. If G = maid2efgII ( S ) or S = efg2maidII ( G ) then G and S are equivalent.

Proof (follows the proof of Lemma 1 in (15) closely). This follows from the construction of maid2efgII and efg2maidII .

First suppose G = maid2efgII ( S ) . A behaviour policy π in S specifies a distribution over actions at each information set I in S . Suppose that I has associated action set D . Each information set in S corresponds to a single meta-information set in G . Supposing that I = ( x , d ) corresponds to meta-information set J , we have that for all nodes Y ∈ J , and each d ∈ dom ( D ) , there exists a unique Z ∈ Ch Y such that λ ( Y, Z ) = d . Thus, we can simply assign σ i [ J ] = π i ( d | x ) . By construction, if under policy π an information set in S is reached with probability p , then in G under σ the corresponding meta-information set will also be reached with probability p . It follows that expected utilities in G and S are the same, under σ and π respectively.

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

Second, suppose S = efg2maidII ( G ) . By our construction, policies defined on S define a mixed action on each information set, defined as a non-null decision context crossed with an action set. Again, using our constructed bijection h between meta-information sets and information sets in our framework, we have a one-to-one mapping. Therefore, for any strategy σ in G , we can assign π i [ h ( J )] = σ i [ J ] for each J ∈ I ω ∗ ( G ) , and again expected utilities are the same in both models.

Theorem 21. Let S = ( N , S ∗ , S ) be an II-MAID, where S is a finite set of subjective MAIDs, S has perfect recall, and dom ( V ) is finite for each V ∈ M S for each S ∈ S . Then S has a Nash equilibrium in behaviour policies.

Proof. Applying G = maid2efgII ( S ) we yield a game with incomplete with perfect recall at interim stage ω , with finite action spaces. By Theorem 16, we know that G has a Nash equilibrium σ in behaviour strategies. By Theorem 20, we know that G and S are equivalent, and therefore there exists a utility-preserving map f from strategies in G to policies in S . Therefore and π ∈ f ( σ ) is a Nash equilibrium in S .

B efg2maidII and maid2efgII

B.1 maid2efgII

maid2efg transforms a MAID to a set of equivalent EFGs, as per definition 17 in [[15]]. We are interested in transforming an II-MAID S = ( N , S ∗ , S ) into a set of equivalent games with incomplete information G = ( N , S, Π) at interim stage with state of the world ω ∗ , as per definition 19. We describe such a transformation here, which we call maid2efgII :

The set of agents N in G is the same as in S .

The set of states of nature (EFGs) S is formed by { maid2efg ( M S ) : s ∈ S } .

We now construct the belief space Π = ( Y, Y , s , ( b i ) i ∈ N ) . Each ω ∈ Y is of the form ( s ( ω ) , ( b i ( ω )) i ∈ N ) . We build a map m : S → Y , noting that each subjective MAID s ∈ S is of the form s = ( M S , ( P S i ) i ∈ N ) .

-s ( m ( s )) ∈ maid2efg ( M S ) , choosing an arbitrary element.

-b i ( m ( s ′ ) | m ( s )) := P s i ( s ′ ) for all s ′ ∈ S .

ω ∗ = m S ∗ .

We now verify that information sets in the II-MAID are mapped one-to-one to metainformation sets with belief b i ( ω ∗ ) in the game with incomplete information defined by the above steps. Information sets in S are defined by decision-context-action-set pairs across MAIDs. For each MAID m ∈ {M S : s ∈ S } , maid2efg ( m ) is a set of EFGs, each of which has the same information sets, but potentially different variable orderings.

-For any node Z (corresponding to some variable S Z in m ) in the tree T of some EFG in maid2efg ( m ) , it is labelled with an instantiation µ ( Z ) corresponding to the values taken by each EFG node on the path from the tree's root R to Z . Nodes will only exist for those paths corresponding to values with non-zero probability according to m . We can query the values of the parents of S Z at the node Z via µ ( Z )[ Pa S Z ] . maid2efg forms information sets by grouping nodes for which this value (and the corresponding node S Z in the MAID) is the same.

-To form meta-information sets, we simply follow [definition of meta-information sets]. Letting I i m be the information sets for agent i in any EFG in maid2efg ( m ) , we can define an equivalence relation ∼ over ∪ m ∈ M I i m such that I 1 ∼ I 2 if and only if µ ( Z 1 )[ Pa S Z 1 ] = µ ( Z 2 )[ Pa S Z 2 ] and dom ( S Z 1 ) = dom ( S Z 2 ) for every Z 1 ∈ I 1 and every Z 2 ∈ I 2 . Then the set of meta-information sets for player i is the quotient set ∪ m ∈ M I i m / ∼ - the set of equivalence classes partitioning ∪ m ∈ M I i m . To match notation, for each element of each meta-information set, append the belief b i ( ω ∗ ) for the appropriate agent i ∈ N .

-Hence, we have a one-to-one mapping between information sets in S and metainformation sets (restricted to belief b i ( ω ∗ ) for each i ∈ N in G , and action sets are preserved under this mapping.

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

B.2 efg2maidII

efg2maid transforms an EFG into an equivalent MAID, as per definition 17 in [[15]]. We are interested in transforming a game with incomplete information G = ( N , S, Π) , at interim stage with state of the world ω ∗ , into an equivalent II-MAID S = ( N , S ∗ , S ) , as per Definition 19. We describe such a transformation here, which we call efg2maidII :

The set of agents N in S is the same as in G .

Given belief space Π = ( Y, Y , s , ( b i ) i ∈ N ) , we can map each state of the world w = ( s ( ω ) , ( b i ( ω )) i ∈ N ) ∈ Y to a subjective MAID s ∈ S with g : Y → S , noting that s is of the form s = ( M S , ( P S i ) i ∈ N ) .

S = g ( ω ) .

Meta-information sets in the game with incomplete information are defined as sets of information sets, across various EFGs, in which nodes has the same action set and the same observations, with observations defined as all information available at a given information set. Since we are at the interim stage of the game, we can restrict our attention to those information sets with belief b i ( ω ∗ ) . In the II-MAID resulting from the above operations, the information sets as per Definition 8 correspond one-to-one with those in the game with incomplete information, as they are defined by sets of observation-action set pairs, with observations defined by the values of parents of the given decision variable. efg2maid determines the parents of a decision variable according to those ancestors of nodes in a given intervention set that have the same value in paths to each node. As a result, there is a one-to-one correspondence between meta-information sets in a game with incomplete information, and the resulting II-MAID, and since action sets of decision variable are preserved by efg2maid , strategies can easily be mapped to policies.

More precisely, we can define a bijection between meta-information sets in G and information sets in S as follows. Given ω ∗ , we denote the meta-information sets in G corresponding to beliefs b i ( ω ∗ ) for some agent i as I ω ∗ ( G ) . Further, for I ∈ I ω ∗ ( G ) denote D ( I ) as the associated action set and O ( I ) the associated observation. O ( I ) is a potentially empty tuple containing observed values of previous decisions or chance nodes. For any information set ( p, d ) ∈ I ( S ) , where efg2maidII ( G ) , p is a tuple containing the values of parent nodes, and d is the associated action set. ( p, d ) ∈ I ( S ) has the same type as ( O ( I ) , D ( I )) for I ∈ I ω ∗ ( G ) . Since for any I, J ∈ I ω ∗ ( G ) , ( O ( I ) , D ( I )) = ( O ( J ) , D ( J )) = ⇒ I = J , we can construct a bijection h : I ω ∗ ( G ) → I ( S ); I ↦→ ( O ( I ) , D ( I )) . We use this construction in the proof of Theorem 20 when converting strategies from one framework to the other.

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

629

630

631

632

633

634

635

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We claim in the abstract to have developed a new framework with a strong theoretical connection to dynamic games with incomplete information and no common prior over types. We also claim to have proven the existence of equilibria concepts. We have done both of these things, along with the rest of the stated achievements.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss limitations in the conclusion.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: For each of our theorems, we state the assumptions clearly, and all proofs are in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: We did not run any experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: We did not present any experimental results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: We did not present any experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: We did not present any experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

741

742

743

744

745

746

747

748

749

750

751

752

753

754

755

756

757

758

759

760

761

762

763

764

765

766

767

768

769

770

771

772

773

774

775

776

777

778

779

780

781

782

783

784

785

786

787

788

789

790

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: We did not present any experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We had no research subjects or data, so no harms were caused in this way. We believe our work is very unlikely to have negative societal impact via the listed categories, as it is purely a theoretical framework and we did not train any models. Our work does not touch on any political issues.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We include a few sentences in the conclusion discussing how we feel our paper could have positive societal impact via a useful model for understanding the behaviour and reasoning of intelligent systems. We feel it is very unlikely that the theory we present here could have any negative societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not present any experimental results, use any data, or train any models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: There are no assets associated with the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: There are no assets associated with the paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We ran no such experiments.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We ran no such experiments.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

894 895 896 897 898

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper extends the framework of multi-agent influence diagrams (MAIDs) to explicitly capture complex forms of reasoning corresponding to Theory of Mind (ToM) as required for the interaction of Multi-Agent Systems with human users.  It introduces the framework of incomplete information MAIDs (II-MAIDs) for explicitly modeling higher-order beliefs in multi-agent interactions alongside probabilistic and causal dependencies between variables. Using results connecting EFGs to MAIDs, the authors demonstrate a natural mapping between strategies in the two frameworks that preserves expected utilities according to the agents’ subjective models.\n\nSTRENGTHS:\nThe approach is well situated within the state-of-the-art of related work in agent models with a game theory component and, as far as one can judge, appears technically sound within the broad remit of causal and influence diagrams (IDs). \nThe paper is very well structured and the authors did their utmost to keep it relatively accessible by alternating formal sections with intuitive descriptive summaries. It remains somehow tedious to read, owing to the large number of definitions, whose numbering alternate with that of theorems. \nThe rationale for building a framework on top of MAIDs rather than EFGs is well introduced, together with the mapping between strategies in MAIDs and EFGs and the choice of working at the interim stage. This culminates with Theorem 20, until section 5.1 raises some issues around the relevance of Nash Equilibria.\n\nWEAKNESSES:\nThe major issue I would raise for this paper is one of relevance to NeurIPS, even in the extended sense. While a major rationale for the paper appears to be its potential application to AI Safety, in the NeurIPS context there does not seem to be enough outreach to current AI models, at least in a way in which they could be interfaced to the proposed ID model. This means some consideration of how current models may form ‘beliefs’, and this was not entirely obvious from the paper’s Title and Abstract. Perhaps my expectation was unrealistic, but I had imagined an attempt to unify formal ToM issues with ToM properties that are known to be associated to LLM, under a framework where this approach would federate or wrap formal agentic methods around, say Agentic LLM. With this comment I am not criticising the authors for not having written another sort of paper, I am simply pointing the perceived gap that may exist between this approach and the NeurIPS constituency. Further evidence would be the absence of references to NeurIPS paper and the relative dearth of mainstream AI venues in the references (to the notable exception of AIJ). Overall, it appears that AAMAS might be a better venue to host this type of paper. \n\nThe paper does not really clarify its ToM framework which references both “multi-agent interactions” as well as “higher-order intentional states” but these aspects are not part of further formal developments. It also mentions “belief hierarchies of arbitrary and infinite depth” and this raises the issue of whether such a formal approach is realistic when it comes to ToM, in particular in the interactions between agents and human users. \n\nDespite an early reference to AI Safety and a mention in the paper’s abstract, there is little in the paper that actually progresses the discussion on AI Safety, which is only used marginally through ID examples, such as the one of Figure 2.\n\nQUESTIONS:\nIn section 5.1 you suggest that beliefs could be extended to policies. Can beliefs be extended to high-level formal concepts that may not be interoperable with other ToM conceptions? Or should beliefs be restricted to epistemic aspects or intentions only?\n\nLIMITATIONS:\nThe limitation section begins with a number of upbeat statements that would better be placed in the conclusion or parts of the abstract. The main identified limitation, which echoes the discussion of section 5.1 is verbatim: “The main limitation of our work is the lack of a useful solution concept.” appears a quite severe restriction. While not affecting the solid grounding of the approach it considerably restricts its impact at its current stage of development.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
The paper introduces a new framework Incomplete Information Multi-Agent Influence Diagrams (II-MAIDs) for modeling complex multi-agent interactions involving theory of mind (ToM) and higher-order beliefs. The authors prove the equivalence between II-MAIDs and Incomplete Information Extensive Form Games (II-EFGs) at the interim stage. The paper also shows the existence of Nash equilibria in II-MAIDs under certain conditions.

STRENGTHS:
The II-MAID framework fills a gap in existing game-theoretic models by allowing for inconsistent beliefs and higher-order reasoning. The paper is built on solid mathematical foundation with formal definitions and proofs.

WEAKNESSES:
* From my perspective, the proposed II-MAID framework appears overly complicated for modeling Theory of Mind (ToM), which is fundamentally a straightforward psychological mechanism observed in daily human interactions. The paper's approach may overcomplicate a concept that should be more intuitively represented.
* The paper introduces numerous assumptions and definitions without clear explanations which hinders the readability. As a non-expert in the field, some details in the paper are difficult to read. 
* It is unclear whether the model can be scaled and applied to larger, more realistic scenarios, where ToM takes place more frequently.
* The paper lacks experiments that validates the model.

QUESTIONS:
* Could the authors elaborate on how this framework might be scaled up? 
* Given that Theory of Mind is fundamentally about real-life interactions, would it be possible for the authors to provide some experiments or toy examples to illustrate the key concepts?

LIMITATIONS:
N/A, see weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work extends the theoretical framework of multi-agent influence diagrams (MAIDs) with incomplete information (II-MAIDs) to explicitly capture this complex form of reasoning. The primary theoretical contribution is the proof of the existence of Nash equilibria, although, in general, these equilibria are impossible for agents to identify.\n\nSTRENGTHS:\n1. This work is game-theoretic in nature, and overall, the presentation quality is good and smooth to the best of my knowledge.\n\n2. Although I think the assumption made in this work generally makes sense to me: agents have consistent beliefs as part of our commonsense, which can be derived from a common prior distribution, I agree that there are settings with no common prior available. The setup is a less constrained setup.\n\nWEAKNESSES:\n1. One of my major concerns is the audience of this work. Given that this work is submitted to the safe ML track of NeurIPS, I expect more discussion on the relevance of this framework to AI safety. The author should elaborate on what they imply by “safety” rather than making a very brief claim about its relevance in the related work and conclusions sections.\n\n2. The discussion of theory of mind is also lacking, given that this is well-motivated. There have been extensive studies on machine theory of mind, ranging from early studies [1-2] to recent studies on LLMs [3-4]. There has also been research connecting Theory of Mind to Game theory [5] and Interactive POMDP [6]. See the survey [7] for details. Overall, this work needs significant improvement in discussing related work for readers to evaluate its contribution and relevance to NeurIPS.\n\n[1] Rabinowitz, Neil, et al. \"Machine theory of mind.\" International conference on machine learning. PMLR, 2018.\n\n[2] Jara-Ettinger, Julian. \"Theory of mind as inverse reinforcement learning.\" Current Opinion in Behavioral Sciences 29 (2019): 105-110.\n\n[3] Sap, Maarten, et al. \"Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs.\" Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.\n\n[4] Ma, Ziqiao, et al. \"Towards A Holistic Landscape of Situated Theory of Mind in Large Language Models.\" Findings of the Association for Computational Linguistics: EMNLP 2023. 2023.\n\n[5] Yoshida, Wako, Ray J. Dolan, and Karl J. Friston. \"Game theory of mind.\" PLoS computational biology 4.12 (2008): e1000254.\n\n[6] Çelikok, Mustafa Mert, et al. \"Interactive AI with a Theory of Mind.\" Computational Modeling in Human-Computer Interaction. 2019.\n\n[7] Albrecht, Stefano V., and Peter Stone. \"Autonomous agents modelling other agents: A comprehensive survey and open problems.\" Artificial Intelligence 258 (2018): 66-95.\n\nQUESTIONS:\n1. “ToM is characterised by multi-agent interactions involving higher-order intentional states, such as beliefs about beliefs, or, in the case of deception, intentions to cause false beliefs…” ToM refers much more broadly than false beliefs and intentions to create false beliefs. How would the authors position other mental states, e.g., emotions, in this framework? Would “belief” be a better scope?\n\n2. Why is this framework relevant to machine learning safety, especially when much of today's safety concerns arise from large-scale pretrained systems like large language models?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}