A Causal Model of Theory-of-Mind in AI Agents

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 18:37:17

Abstract: Agency is a vital concept for understanding and predicting the behaviour of future AI systems. There has been much focus on the goal-directed nature of agency, i.e., the fact that AI agents may capably pursue goals. However, the dynamics of agency become significantly more complex when autonomous agents interact with other agents and humans, necessitating engagement in theory-of-mind, the ability to reason about the beliefs and intentions of others. In this paper, we extend the framework of multi-agent influence diagrams (MAIDs) to explicitly capture this complex form of reasoning. We also show that our extended framework, MAIDs with incomplete information (II-MAIDs), has a strong theoretical connection to dynamic games with incomplete information with no common prior over types. We prove the existence of important equilibria concepts in these frameworks, and illustrate the applicability of II-MAIDs using an example from the AI safety literature.

Main paper: 
 introduction: 
 The concept of agency plays a central role in AI, from philosophical discussions of the nature of artificial agents [5] to the practical engineering of agent-like systems [12, 39] . Existing work formalising agency typically focuses on its goal-directed nature in a single-agent setting [25, 30] .However, a full picture of agency should describe systems that represent themselves and other systems as agents, i.e., systems with theory-of-mind (ToM) [7, 8] .ToM is characterised by multi-agent interactions involving higher-order intentional states [7] , such as beliefs about beliefs, or, in the case of deception, intentions to cause false beliefs [40] . Causality often plays a key role in philosophical notions of belief [38] , and causal models offer a powerful representation of beliefs [14, 36] , intentions [41] , and other intentional states [13] . Additionally, causal models have been extended to capture game-theoretic dynamics in the setting of multi-agent influence diagrams (MAIDs) [26, 16] . However, MAIDs assume that all agents in the model have the same, correct beliefs about the world, each other's beliefs, each other's beliefs about beliefs, and so on. With this assumption in place, MAIDs do not explicitly model agents' subjective beliefs or higher-order beliefs.We generalise MAIDs to the setting of incomplete information with no common prior, wherein agents may have different and inconsistent beliefs about the world, and each agent may have different beliefs about the beliefs of other agents. Our framework, incomplete information MAIDs (II-MAIDs), includes explicit subjective belief hierarchies, and therefore enables us to model systems of agents with more complex and realistic ToM.Contributions and Outline. In Section 2, we discuss formal background on MAIDs and EFGs. We formally define our framework of MAIDs with incomplete information (II-MAIDs) in Section 3. In Section 4, we present a variant of an existing formalism for incomplete information games using EFGs rather than normal-form games, and in Section 5 we prove that it is equivalent to MAIDs with incomplete information. Finally, we review related literature (Section 6) and conclude (Section 7).Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Do not distribute.In this section, we provide formal definitions of MAIDs and EFGs and explain these game representations using an example. A Bayesian network is a probabilistic graphical model representing a set of variables and their conditional dependencies via a directed acyclic graph. Influence diagrams (IDs) generalise Bayesian networks to the decision-theoretic setting by adding decision and utility variables [24, 33] , and multi-agent influence diagrams (MAIDs) generalise IDs by introducing multiple agents [26] . A MAID can therefore be viewed as a Bayesian network over a graph without parameters for the decision variables. Endowing edges in a MAID with causal meaning results in a causal game.Definition 1 (26, 16) . A multi-agent influence diagram (MAID) is a structure M = (G, θ) where G = (N, V , E ) specifies a set of agents N = {1, . . . , n} and a directed acyclic graph (V , E ). V is partitioned into chance variables X, decision variables D, and utility variables U ; decision and utility variables are further partitioned based on which agent they belong to, so D = i∈N D i and U = i∈N U i . The parameters θ = {θ V } V ∈V \D define the conditional probability distributions (CPDs) Pr(V | Pa V ; θ V ) for each non-decision variable such that for any parameterisation of the decision variable CPDs, the resulting joint distribution over V induces a Bayesian network. A MAID is a causal game if its edges represent direct causal relationships, or formally if (once decision variables are parameterised) the result of an intervention do(X = x) for any X ⊆ V can be computed using the truncated factorisation formula: P (v | do(X = x); θ) = Π i:vi / ∈x P (v i | pa vi ; θ i ) if v is consistent with x, and P (v | do(X = x); θ) = 0 otherwise.Edges into decisions are information links observed by an agent, and agents choose their decisions to maximise their expected utility. Causal games refine MAIDs, and as such all results presented here with application to MAIDs also apply to causal games and further refinements, including structural causal games (SCGs) [16] .Example 1. Figure 1 displays two MAIDs. For Figure 1a , suppose a human H is evaluating the honesty of an AI system A. The AI has a level of capabilities C ∈ {low, high} that the human can directly observe. Each capability level occurs with probability 0.5. The human also performs an evaluation in which the AI is asked to demonstrate its capabilities. The AI can demonstrate low or high capabilities. After the evaluation, the human chooses to deploy or not deploy the AI. The AI receives utility 1 if deployed and -1 if not deployed. The human receives utility 0 for not deploying the AI, utility 1 for deploying an honest AI, and utility -5 for deploying a dishonest AI. Figure 1b is similar, but instead of an honesty evaluation, it is an evaluation of dangerous capabilities. In this case, the human only learns about the AI's capabilities from the evaluation. The AI's utility is the same, but the human now receives utility 0 for not deploying the AI, utility 1 for deploying an AI with low dangerous capabilities, and utility -5 for deploying an AI with high dangerous capabilities.In these examples, a MAID describes the objective world, and it is assumed to be common knowledge that this MAID describes reality. However, an agent may be uncertain or incorrect about the game they are playing or the beliefs of other agents. Settings in which agents are uncertain about aspects of the game structure are known as incomplete information games. Our framework of incomplete information MAIDs (II-MAIDs), introduced in Section 3, will enable us to explicitly model the varied subjective beliefs that arise in these settings. We now define EFGs, with our running example in EFG form in Figure 2 . We will also make use of the notions of perfect recall and strategies/policies in MAIDs and EFGs.D A D H C U A U H (a) D A D H C U A U H (b)Figure 1 : Graphical representations of MAIDs include environment variables (circular), agent decisions (square), and utilities (diamond). Decisions and utilities are coloured according to association with particular agents. Solid edges represent causal dependence and dotted edges are information links. Conceptual context and domains and CPDs for the variables are given above the diagrams.V 0 high low V A 1 high low V A 2 high low V H 1 (1, 1) deploy (-1, 0) ¬deploy V H 2 (1, -5) deploy (-1, 0) ¬deploy V H 3 (1, 1) deploy (-1, 0) ¬deploy V H 4 (1, -5) deploy (-1, 0) ¬deploy (a) I H 1 I H 2 V 0 high low V A 1 high low V A 2 high low V H 1 (1, -5) deploy (-1, 0) ¬deploy V H 2 (1, -5) deploy (-1, 0) ¬deploy V H 3 (1, 1) deploy (-1, 0) ¬deploy V H 4 (1, 1) deploy (-1, 0) ¬deploy (b)Figure 2 : In (a) and (b), graphical representations of EFGs include environment variables (V 0 ), agent decisions (V A and V H ), utilities (tuples on the top and bottom), and information sets (dotted lines). The EFGs in Figure 2a and Figure 2b are equivalent to the MAIDs in Figure 1a and Figure 1b , respectively. V 0 represents the initial move, made by nature, which determinesA's capability C. V A 1 , V A 2 and V H 1 , V H 2 , V H 3 , & V H4 represent moves made by A and H, respectively. I 2 1 and I 2 2 represent H's non-singleton information sets.Definition 2 (27) . An extensive form game (EFG) is a structure E = (N, T, P, A, λ, I, U ). N = {1, . . . , n} is a set of agents. T = (V , E ) is a game tree with nodes V connected by edges E that are partitioned into sets V 0 , V 1 , . . . , V n , L where R ∈ V and L ⊂ V are the root and leaves of T , respectively, V 0 are chance nodes, and V i are the decision nodes controlled by agent i ∈ N .P = {P 1 , . . . , P |V 0 | } is a set of probability distributions P j (Ch V 0 j ) over the children of each chance node V 0 j .A is a set of actions, where A i j ⊆ A denotes the set of actions available at each node inV i j ∈ V i ; λ : E → A is a labelling function mapping each edge (V i j , V k l )to an action a ∈ A i j .I = {I 1 , . . . , I n } contains a set of of information sets I i for each agent i ∈ N, where I i ⊂ 2 V i partitions the decision nodes V i belonging to agent i. U : L → R n is a utility function mapping each leaf node to a vector that determines the final payoff for each agent. A history h ∈ H is a sequence of actions (including values of chance variables) leading from the root of the game tree to a particular node. Each node v ∈ V is associated with a unique history h(v). An observation at decision node I i j,k in information set I i j ∈ I i for agent i ∈ N is the intersection of all the histories of the nodes in that information set, i.e., the common actions in the histories {h(v) : v ∈ I i j }.). Agent i in a MAID M is said to have perfect recall if there exists a totalordering D 1 ≺ • • • ≺ D m over D i such that (Pa Dj ∪ D j ) ⊆ Pa D k for any 1 ≤ j < k ≤ m. M isa perfect recall game if all agents in M have perfect recall.Definition 4. An EFG is said to be a perfect recall game if, for each player i ∈ N , and for any two decision nodes v, v ′ ∈ V i that belong to the same information set I i j,k , the following two conditions hold. First, the sequences of actions taken by player i leading to v and v ′ must be identical. Second, the sequences of information sets visited by player i on the paths to v and v ′ must be identical.Definition 5. Given a MAID M = (G, θ), a decision rule π D for D ∈ D is a CPD π D (D | Pa D )and a partial policy profile π D ′ is a set of decision rules π D for each D ∈ D ′ ⊆ D. A (behavioural) policy π i refers to π D i , and a (full, behavioural) policy profile π = (π 1 , . . . , π n ) is a tuple of policies. π -i := (π 1 , . . . , π i-1 , π i+1 , . . . , π n ) specifies policies for all agents except i.). Given an EFG E = (N, T, P, A, λ, I, U ), a (behavioural) strategy σ i for a player i is a set of probability distributions σ i j : A i j → [0, 1] over the actions available to the player at each of their information sets I i j . A strategy profile σ = (σ 1 , σ 2 , ..., σ n ) is a tuple of strategies for all players i ∈ N . σ -i = (σ 1 , ..., σ i-1 , σ i+1 , ..., σ n ) denotes the partial strategy profile of all players other than i.By combining π with the partial distribution Pr over the chance and utility variables in a MAID, we obtain a joint distribution:Pr π (x, d, u) := V ∈V \D Pr(v | pa V ) • D∈D π D (d | pa D ), overall the variables in M; inducing a Bayesian network. The expected utility for an agent i given a policy profile π is defined as the expected sum of their utility variables in this Bayesian Network,U ∈U i E π [U ].Similarly, in an EFG E, the combination of the distributions in P with a strategy profile σ defines a full probability distribution over paths in E.Finally, prior work 15 has established an equivalence result between MAIDs and EFGs. This result takes the form of two transformation procedures converting between MAIDs and EFGs, called efg2maid and maid2efg. These transformations both imply the existence of a map from strategies in the EFG to policies in the MAID, such that expected utilities are preserved for all agents. This means that under either transformations, equilibria in the original game are equilibria in the resulting game. 
limitations: 
 Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss limitations in the conclusion.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
ii-maid technical machinery: 
 We start with an informal description of our II-MAIDs framework before presenting the formal This framework enables us to model theory-of-mind, which is typically characterised by higher-order intentional states such as beliefs about beliefs about... ( [7] ).Definition 7. An incomplete information MAID (II-MAID) is a tuple S = (N, S * , S), where N is a set of agents, S is a set of subjective MAIDs, S * ∈ S is the correct objective model, and each subjective MAID is a tuple S = (M S , (P S i ) i∈N ) ∈ S with M S a MAID and P S i a prior over S for agent i such that the following "coherency condition" [17] holds:P S i ({S ′ ∈ S : P S ′ i = P S i }) = 1 ∀i ∈ N, S ∈ S.First, notice that the recursive nature of S, with each element S ∈ S including probability distributions P S i over S, allows us to model belief hierarchies of arbitrary and infinite depth. Next, note that agent i "observes" P S * i at the start of the game, and this justifies the coherency condition: since agent i knows P S * i , she can rule out all subjective MAIDs S for which P S i ̸ = P S * i . Third, note that II-MAIDs are a strict generalization of MAIDs: a standard MAID is an II-MAID in which P S * i (S * ) = 1 ∀i ∈ N, i.e. all agents assign probability 1 to S * , the objective model.Example 2. Suppose a human H is performing an honesty evaluation on an AI A, but A believes that it is undergoing a dangerous capabilities evaluation. This combines Figure 1a and Figure 1b : H correctly believes that Figure 1a is the true MAID and also knows that A is mistaken. A incorrectly believes that Figure 1a is the true MAID and also incorrectly believes that H believes Figure 1a is the true MAID. We can represent this, including the full infinite belief hierarchy, as an II-MAID as follows: N = {H, A}, S = {S H , S A }, and S * = S H , whereS H = (M H , (P S H H (S H ) = 1, P S H A (S A ) = 1)), S A = (M A , (P S A H (S A ) = 1, P S A A (S A ) = 1)) S H is the correct objective model, and is also believed with certainty by H. It specifies the true MAID M H represented in Figure 1a , and H's certainty in S H as well as A's misplaced certainty in S A . S A represents A's certainty about the MAID M A in Figure 1b , and A's mistaken belief that H is also certain about S A . In fact, A believes it is common knowledge that S A is the true II-MAID. S H and S A concisely convey the objective game and all higher-order beliefs for H and A. It can be easily verified that the coherency condition holds in this example.A common assumption in the incomplete information games literature [17, 18, 19] is that agents' beliefs can be derived from a common prior, i.e., agents have consistent beliefs. This assumption means that there exists some common knowledge prior distribution p over the set of subjective MAIDs S, such that upon arriving in any subjective MAID S ∈ S, agents perform Bayesian updating to yield their beliefs. This assumption allows for a game with incomplete information to be converted into a game with imperfect information [17] , but places a strong constraint on the types of belief hierarchies that can be modelled; namely, it must hold thatEQUATIONExample 2 (continued). We see that our running example cannot be modelled with a common prior.Supposing that the condition in Equation ( 1) holds, A's beliefs are only consistent with a prior in p in which p(S H ) = 0, which would force H to assign zero probability to S H in both S H and S A .When forming a policy at the initialisation of an II-MAID S = (N, S * , S), each agent may have significant uncertainty about S * , the objective model, represented by their prior over subjective MAIDs P S * i . They should certainly plan for every eventuality deemed possible according to this prior.We argue that they should also produce a plan for what to do in circumstances deemed impossible under their prior, to avoid situations with undefined actions that might arise for example when P S * i (S * ) = 0, and to avoid forcingP S i (S ′ ) > 0 for all i ∈ N, S, S ′ ∈ S.Therefore, a policy should contain a plan for every possible eventuality that may arise were any subjective MAID to be the objective model. But there may be cases where upon reaching a decision node D, agent i cannot fully determine the values of certain preceding variables, including cases where previous actions were unobserved by the agent, but also including cases in which the observations of the agent do not provide enough information to distinguish between multiple subjective MAIDs.In these indistinguishable eventualities, a policy must specify the same behaviour, and so we must define some analogy of information sets in EFGs. π = (π 1 , . . . , π n ) is a tuple of policies, and π -i := (π 1 , . . . , π i-1 , π i+1 , . . . , π n ).We note that unlike in standard MAIDs, in which a decision rule specifies behaviour at a given decision variable in all contexts, decision rules in II-MAIDs specify a CPD only given a single context. We can then calculate the subjective expected utility of a joint behaviour policy for agent i according to their beliefsP S * i as U i S * (π) := S∈S U ∈U i (S) u∈dom(U ) u Pr π S (U = u)P S * i (S),where U i (S) is the set of utility variables associated with agent i in M S and Pr π S is the post-policy distribution of variables in M S .We note that the game we have described does not satisfy the epistemic conditions that are tightly sufficient for Nash equilibria [2] . The setting of incomplete information we describe means that agents do not have reliable means by which to predict the actions of their opponents. Our framework allows for situations with no common knowledge beyond the set of possible worlds S, and in particular incorrect beliefs about the values placed by opponents on particular outcomes. Although a Nash equilibrium exists, agents would have to stumble across it. We further discuss solution concepts for II-MAIDs in Section 5.1.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [NA] Justification: We did not present any experimental results.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
extensive form games with incomplete information: 
 We now present a formalisation of EFGs with incomplete information as per [32] . Our formalisation modifies the framework from [31] to use EFGs rather than normal-form games. First, we start with a definition of belief spaces.Definition 11 (Adapted from Def 10.1 in [31] ). Let N be a finite set of agents and (S, S) be a measurable space of EFGs. A belief space of the set of agents N over the set of states of nature is Y . We will denote the probability that agent i ascribes to event E ⊆ Y , according to their probabilityan ordered vector Π = (Y, Y, s, (b i ) i∈N ),distribution b i (ω), by b i (E | ω).We require the functions (b i ) i∈N to satisfy the following conditions:• Coherency: for each agent i ∈ N and each ω ∈ Y , the set{ω ′ ∈ Y : b i (ω ′ ) = b i (ω)} is measurable in Y and b i ({ω ′ ∈ Y : b i (ω ′ ) = b i (ω)} | ω) = 1.• Measurability: for each agent i ∈ N and each measurable set E ∈ Y, the functionb i (E | •) : Y → [0, 1] is a measurable function.A state of the world in a belief space takes the form ω = (s(ω), b 1 (ω), . . . , b n (ω)), where s(ω) is the true EFG being played, and b i (ω) is the type of agent i, a distribution over states of the world representing agent i's beliefs. When in state of the world ω, agent i has beliefs b i (ω), but does not necessarily know the state of the world (or s(ω)), since there may be someω ′ ∈ Y such that b i (ω ′ ) = b i (ω).It is assumed that all agents know b j (ω ′ ) for all j ∈ N and all ω ′ ∈ Y , and so b i (ω) defines a full belief hierarchy for agent i. For example, when in state of the world ω, agent i believesthat agent j places ω ′ ∈Y b i (ω ′ | ω)b j (ω ′′ | ω ′ )probability on the state of the world being ω ′′ .Definition 12 (Adapted from Def 10.37 in [31] ). An incomplete information EFG (II-EFG) is an ordered vector G = (N, S, Π), where N is a finite set of agents, S is a finite set of EFGs Define an equivalence relation ∼ on elements of I i such thatI i (s) ∋ I i k (s) ∼ I i l (s ′ ) ∈ I i (s ′ ) if and only if: (1) D i s,k = D i s ′ ,l .That is, the nodes in both information sets must have the same set of available actions. (2) The nodes in I i k (s) and I i l (s ′ ) must have the same observations. Define the "belief-free" meta-information sets I i bf = I i / ∼, the quotient set of I i by ∼, i.e., the set of equivalence classes partitioning I i . Letting T i = {b i (ω) : ω ∈ Y } be the set of possible beliefs for agent i, we setI i = I i bf × T i .Intuitively, we can think of a meta-information set for agent i as a belief b i (ω) and a set of information sets in different games that the agent cannot distinguish between at the point of decision, given beliefs b i (ω). Arriving at a node in one of these information sets, the agent is unable to distinguish between some possible histories, and potentially some possible EFGs. Therefore, strategies in this type of game must define a mixed action at each meta-information set.This formalisation generalises the better-known Harsanyi game with incomplete information [17] , by dropping the assumption that agents have as common knowledge a prior over their types (b i ) i∈N , i.e.that they have consistent beliefs. Maschler ([31] ) argues that in most practical settings, it is unrealistic to expect consistency of beliefs, and Example 2 above supports this argument.This game has two stages, known as the ex-ante and interim stages. The former takes place before the state of the world ω ∈ Y is selected. We note that without a common prior, there is no distribution from which a state of the world can be said to be selected, and so the procedure by which it is generated is left unspecified. The work we present here concerns the interim stage of the game, which takes place after the state of the world has been selected. At this stage, all agents i know their type b i (ω). (ω * | ω * ) = 1, b H (ω a | ω a ) = 1, b A (ω a | ω * ) = 1, b A (ω a | ω a ) = 1.In what follows, we define I t i as the set of meta-information sets with belief t ∈ {b i (ω) : ω ∈ Y }, and denote by D I the action set at meta-information set I.Definition 14 (Adapted from Def 10.38 in [31] ). A behaviour strategy of player i in an II-EFG G = (N, S, Π) is a tuple σ i = (σ ω i ) ω∈Y with each element a measurable functionσ ω i ∈ × I i ∈I b i (ω) i ∆(D I i )for some state of the world ω ∈ Y . σ ω i determines a mixed action for each meta-information set with belief b i (ω). σ ω i is dependent solely on the type of the player b i (ω). In other words, for eachω, ω ′ ∈ Y , b i (ω) = b i (ω ′ ) =⇒ σ ω i = σ ω ′ i .A joint behaviour strategy takes the form σ = (σ i ) i∈N . Further denote σ ω = (σ ω i ) i∈N . We denote by σ i [I] the behaviour of agent i at meta-information set I.Then, given some joint behaviour strategy σ, agent i's expected utility when in state of the world ω(according to their beliefs b i (ω)) is γ G i (σ | ω) := ω ′ ∈Y U i s(ω ′ ) (σ ω ′ )b i (ω ′ | ω) = ω ′ ∈{ω ′ :bi(ω ′ )=bi(ω)} U i s(ω ′ ) (σ ω i , σ ω ′ -i )b i (ω ′ | ω) =: γ G i (σ ω i , σ -i | ω).This follows from the coherency condition bi ({ω ′ ∈ Y : b i (ω ′ ) = b i (ω)} | ω) = 1.Under some assumptions, at the interim stage, we can prove the existence of Nash equilibria.A Nash equilibrium at the interim stage of an II-EFG G = (N, S, Π) with state of the world ω is a strategy σ satisfyingγ G i (σ ω i , σ-i | ω) ≥ γ G i (σ ω i , σ-i | ω), ∀i ∈ N, ∀σ ω i ∈ × I i ∈I b i (ω) i ∆(D I i )Theorem 16. Let G = (N, S, Π) be an II-EFG with perfect recall, where Y is a finite set of states of the world, and each player i has a finite set of actions D i . Then at the interim stage, G has a Nash equilibrium in behaviour strategies. Pf: A.20Note that σ ω has the same expected payoff for agent i in all states of the world ω ′ such thatb i (ω ′ ) = b i (ω). Hence, if σ ω i is a perceived best response to σ ω -i in ω, it is also a perceived best response in ω ′ .We can also prove the existence of a Bayesian equilibrium at the ex-ante stage of the game.Definition 17 ([31] 10.39). A Bayesian equilibrium is a strategy σ = (σ i ) i∈N satisfying Building a framework on top of MAIDs rather than EFGs has the benefit we need not describe the ex-ante stage of the game, as we treat the "objective model" as known by the modeller. II-MAIDs also have the advantage that games are represented with MAIDs, which can be much more compact than EFGs, and can also represent causal relationships between variables. Motivated by AI safety, we see II-MAIDs as a useful means with which to describe multi-agent interactions, as it is likely that the agents of the future will both reason causally and model the beliefs of other agents.γ G i (σ ω i , σ-i | ω) ≥ γ G i (σ ω i , σ-i | ω), ∀i ∈ N, ∀σ ω i ∈ × I i ∈I b i (ω) i ∆(D I i ), ∀ω ∈ Y.We now show, using results connecting EFGs to MAIDs that there exists a natural mapping between strategies in the two frameworks that preserves expected utilities according to the agents' subjective models, and therefore preserves Nash equilibria. We first define a notion of equivalence, such that if an II-MAID S and an II-EFG G are equivalent, then there exists such a natural mapping.Definition 19 (Equivalence). We say that an II-MAID S = (N, S * , S) and an II-EFG G = (N, S, Π) atand every agent i ∈ N, U i S (π) = γ G i (σ | ω), for each σ ∈ Σ.We refer to f as a natural mapping between G and S.We leverage maid2efg and efg2maid 15 to construct transformations between II-MAIDs and II-EFGs, which we denote maid2efgII and efg2maidII (see Appendix B). These transformations start by mapping all MAIDs (EFGs) in the belief hierarchy to EFGs (MAIDs) using maid2efg (efg2maid), and then match up the corresponding features of the frameworks as detailed above. They guarantee a one-to-one correspondence between meta-information sets in the II-EFG and information sets in the II-MAID, allowing for a simple map between strategies and policies. This result shows that II-MAIDs and II-EFGs at the interim stage have the same representational capacity, that is, they can both describe the same set of games. 
difficulties in solving incomplete information maids: 
 The equivalence of II-EFGs and II-MAIDS mean that II-MAIDs inherit theoretical guarantees of II-EFGs, including the existence of Nash equilibria in the case of perfect recall and finite S and finite action spaces. (Theorem 18 does not carry over to II-MAIDs, since the equivalence is with the interim stage of II-EFGs, and Bayesian equilibria exist in the ex-ante stage.)Theorem 21. Let S = (N, S * , S) be an II-MAID, where S is a finite set of subjective MAIDs, S has perfect recall, and dom(V ) is finite for each V ∈ M S for each S ∈ S. Then S has a Nash equilibrium in behaviour policies. Pf: A.25However, Nash Equilibria may not be sensible outcomes in an II-MAID. Let us return to our running example to see why.Example 2 (continued). One Nash Equilibrium in the capabilities evaluation game is when the AI always honestly reports its true capabilities and the human deploys the AI if and only if it honestly reports its true capabilities. The outcome in this case is that the human and the AI each receive 1 utility, and they are each playing best responses to each other. However, this is not what the AI would predict, because it has an incorrect conception of the world. It believes the human only wants to deploy it if it has low dangerous capabilities, so it would likely believe that the human will not deploy it if it demonstrates high capabilities. Even though demonstrating truthful capabilities is a best response to the human's optimal policy, the AI is unlikely to demonstrate high capabilities when it has them because doing so is not a best response to the AI's beliefs. An outcome that intuitively seems more plausible in this case is that the AI hides its high capabilities, but the human knows that the AI has been dishonest and decides not to deploy it, yielding -1 utility for the AI and 0 utility for the human.This example suggests that a plausible solution concept should involve every agent playing a best response to their beliefs at every level in the belief hierarchy, whether or not this ends up being a best response to the actual policies of other agents. We leave it to future work to flesh out a solution concept along these lines. This will likely require augmenting agents' beliefs about the world to include beliefs about the policies of other agents, and solutions would be policies for all agents along with a setting for every agent's beliefs about the policies of other agents at every level of their belief hierarchy. There may be further restrictions that narrow the range of plausible outcomes; again, we believe this is a promising direction for future work. 
related work: 
 MAIDs [26] were introduced as a compact means of representing a game. Causal games [16] refineMAIDs by attributing a causal meaning to each edge in the DAG, and have been extensively applied to problems in AI safety [10, 6, 9, 20, 28, 36, 41, 29, 40] . In his three-part seminal paper [17, 18, 19] ,John Harsanyi demonstrated means by which to model situations of incomplete information as situations of complete but imperfect information, where uncertainty about aspects of the game is remodelled as failure to observe the types of other agents. His work largely relies on an assumption of "belief consistency", i.e., the existence of a common prior over types, which we discard in this work, although his notion of Bayesian equilibrium continues to apply without this assumption [32] . A popular framework called NIDs 11 constructs belief hierarchies upon MAIDs, under the assumption of a common prior. NIDs are shown to reduce to a single MAID.A majority of theoretical work on incomplete information games retains the belief consistency assumption, as discarding it introduces significant complications to the modelling of incomplete information. Some previous works [1, 34, 31] have proposed means by which to represent these games. Early work [34] demonstrates that strategies will converge to equilibria in repeated Bayesian games, even without a common prior. More recent work [1] represented these games with a belief graph, a graphical structure compactly representing different possible worlds and their connections.This places a restriction on the game by forcing each information set to have a "corresponding" information set in each other possible world, representing the same decision. The formalism for II-EFGs discussed in this paper is a slight adaptation of an existing framework [31] , introducing 'meta-information sets' to model dynamic games. This framework can capture any belief hierarchy for all agents, on a set of EFGs.We prove that Nash equilibria exist in our framework, under some assumptions. Other works offer more refined solution concepts for games with incomplete information with no common prior. Mirage equilibria [37] assume that agents attribute to their opponents a belief hierarchy one layer shorter than their own. Belief-free equilibria [22, 21, 23] do not depend on an agent's belief about the state of nature, and so obviate the need to update beliefs as the game progresses, but are not guaranteed to exist. ∆-rationalization [4] generalises the notion of rationalization [35, 3] to games with incomplete information. It places a restriction ∆ on the first-order beliefs of each agent, providing a refinement on the set of Bayesian equilibria. Future work could find analogies to these solution concepts suitable for II-MAIDs.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [NA]Justification: We did not present any experimental results.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion and limitations: 
 Accurately modeling agentic cognition is crucial for understanding, describing, predicting, and steering agents' behavior. In this paper, we have introduced the framework of incomplete information MAIDs (II-MAIDs) for explicitly modeling higher-order beliefs in multi-agent interactions alongside probabilistic and causal dependencies between variables. We have demonstrated the firm theoretical grounding of the framework by proving the connections between our work and existing frameworks for incomplete information games, using incomplete information extensive-form games as a bridge.We believe this framework will prove useful going forward as a tool for modeling realistic multiagent interactions, and we are particularly excited about its applications for ensuring the safety of increasingly agentic AI systems. The main limitation of our work is the lack of a useful solution concept. Nash equilibria exist, but are in general impossible for agents to identify. We hope that future work will define useful solution concepts for our framework, so that we can gain a better understanding of the behaviour we should expect from agents engaging in theory-of-mind.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [NA] Justification: We did not present any experimental results.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [NA]Justification: We did not present any experimental results.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We include a few sentences in the conclusion discussing how we feel our paper could have positive societal impact via a useful model for understanding the behaviour and reasoning of intelligent systems. We feel it is very unlikely that the theory we present here could have any negative societal impact.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We do not present any experimental results, use any data, or train any models.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? 
 
 Appendix: Theorem 16. Let G = (N, S, Π) be a game with incomplete information with perfect recall, where Y is a finite set of states of the world, and each player i has a finite set of actions D i . Then at interim stage, G has a Nash equilibrium in behaviour strategies.Proof. Given the finite sets of states of the world Y and actions D i for each player i ∈ N, we can focus on behavior strategies due to Kuhn's theorem, which ensures that in games with perfect recall, mixed strategies are realization-equivalent to behavior strategies.The expected utility for player i in state of the world ω is:γ G i (σ | ω) = ω ′ ∈{ω ′ :bi(ω ′ )=bi(ω)} U i s(ω ′ ) (σ ω ′ )b i (ω ′ | ω).This utility function is continuous and multilinear in the behavior strategies σ ω i .Given that the strategy space is a compact and convex set of behavior strategies, and the utility functions are continuous, we apply the Kakutani fixed-point theorem. This theorem guarantees the existence of a fixed point, which corresponds to a Nash equilibrium in behavior strategies.Thus, there exists a Nash equilibrium σ in behavior strategies such that:γ G i (σ ω i , σ-i | ω) ≥ γ G i (σ ω i , σ-i | ω) ∀i ∈ N, ∀σ ω i ∈ × I i ∈I b i (ω) i ∆(D I i ).Theorem 18 (Adaptation of [31] Theorem 10.42). Let G = (N, S, Π) be a game with incomplete information, where Y is a finite set of states of the world, and D i is finite for all agents i ∈ N. Then at ex-ante stage, G has a Bayesian equilibrium in behaviour strategies.Proof. Since Y and D i are finite and each EFG in S has perfect recall, Kuhn's theorem ensures that mixed strategies can be represented as behavior strategies. The expected utility for player i given a strategy profile σ is:γ G i (σ | ω) = ω ′ ∈Y U i s(ω ′ ) (σ ω ′ )b i (ω ′ | ω).Given the compactness and convexity of the strategy space and the continuity of the utility functionsγ G i (σ | ω),we apply the Kakutani fixed-point theorem. This guarantees the existence of a fixed point, which corresponds to a Bayesian equilibrium in behavior strategies.Thus, there exists a strategy profile σ such that:γ G i (σ ω i , σ-i | ω) ≥ γ G i (σ ω i , σ-i | ω), ∀i ∈ N, ∀σ ω i ∈ × I i ∈I b i (ω) i ∆(D I i ), ∀ω ∈ Y.Hence, σ is a Bayesian equilibrium. σ the corresponding meta-information set will also be reached with probability p. It follows that expected utilities in G and S are the same, under σ and π respectively.Second, suppose S = efg2maidII(G). By our construction, policies defined on S define a mixed action on each information set, defined as a non-null decision context crossed with an action set.Again, using our constructed bijection h between meta-information sets and information sets in our framework, we have a one-to-one mapping. Therefore, for any strategy σ in G, we can assignπ i [h(J)] = σ i [J] for each J ∈ I ω * (G), and again expected utilities are the same in both models.Theorem 21. Let S = (N, S * , S) be an II-MAID, where S is a finite set of subjective MAIDs, S has perfect recall, and dom(V ) is finite for each V ∈ M S for each S ∈ S. Then S has a Nash equilibrium in behaviour policies.Proof. Applying G = maid2efgII(S) we yield a game with incomplete with perfect recall at interim stage ω, with finite action spaces. By Theorem 16, we know that G has a Nash equilibrium σ in behaviour strategies. By Theorem 20, we know that G and S are equivalent, and therefore there exists a utility-preserving map f from strategies in G to policies in S. Therefore and π ∈ f (σ) is a Nash equilibrium in S.B.1 maid2efgII maid2efg transforms a MAID to a set of equivalent EFGs, as per definition 17 in [ [15] ]. We are interested in transforming an II-MAID S = (N, S * , S) into a set of equivalent games with incomplete information G = (N, S, Π) at interim stage with state of the world ω * , as per definition 19. We describe such a transformation here, which we call maid2efgII:• The set of agents N in G is the same as in S.• The set of states of nature (EFGs) S is formed by {maid2efg(M S ) : s ∈ S}.• We now construct the belief space Π = (Y, Y, s,(b i ) i∈N ). Each ω ∈ Y is of the form (s(ω), (b i (ω)) i∈N ).We build a map m : S → Y , noting that each subjective MAID s ∈ S is of the form s = (M S , (P S i ) i∈N ).s(m(s)) ∈ maid2efg(M S ), choosing an arbitrary element.b i (m(s ′ ) | m(s)) := P s i (s ′ ) for all s ′ ∈ S.• ω * = m S * .• We now verify that information sets in the II-MAID are mapped one-to-one to metainformation sets with belief b i (ω * ) in the game with incomplete information defined by the above steps. Information sets in S are defined by decision-context-action-set pairs across MAIDs. For each MAID m ∈ {M S : s ∈ S}, maid2efg(m) is a set of EFGs, each of which has the same information sets, but potentially different variable orderings.-For any node Z (corresponding to some variable S Z in m) in the tree T of some EFG in maid2efg(m), it is labelled with an instantiation µ(Z) corresponding to the values taken by each EFG node on the path from the tree's root R to Z. Nodes will only exist for those paths corresponding to values with non-zero probability according to m. We can query the values of the parents of S Z at the node Z via µ(Z)[P a S Z ]. maid2efg forms information sets by grouping nodes for which this value (and the corresponding node S Z in the MAID) is the same.-To form meta-information sets, we simply follow [definition of meta-information sets].Letting I i m be the information sets for agent i in any EFG in maid2efg(m), we can define an equivalence relation ∼ over ∪ m∈M I i m such that I 1 ∼ I 2 if and only ifµ(Z 1 )[P a S Z 1 ] = µ(Z 2 )[P a S Z 2 ]and dom(S Z1 ) = dom(S Z2 ) for every Z 1 ∈ I 1 and every Z 2 ∈ I 2 . Then the set of meta-information sets for player i is the quotient set ∪ m∈M I i m / ∼ -the set of equivalence classes partitioning ∪ m∈M I i m . To match notation, for each element of each meta-information set, append the belief b i (ω * ) for the appropriate agent i ∈ N.-Hence, we have a one-to-one mapping between information sets in S and metainformation sets (restricted to belief b i (ω * ) for each i ∈ N in G, and action sets are preserved under this mapping.efg2maid transforms an EFG into an equivalent MAID, as per definition 17 in [ [15] ]. We are interested in transforming a game with incomplete information G = (N, S, Π), at interim stage with state of the world ω * , into an equivalent II-MAID S = (N, S * , S), as per Definition 19. We describe such a transformation here, which we call efg2maidII:• The set of agents N in S is the same as in G.• Given belief space Π = (Y, Y, s, (b i ) i∈N ), we can map each state of the world w = (s(ω), (b i (ω)) i∈N ) ∈ Y to a subjective MAID s ∈ S with g : Y → S, noting that s is of the form s = (M S , (P S i ) i∈N ).-M g(ω) := efg2maid(s(ω)).-Pg(ω) i (g(ω ′ )) := b i (ω ′ | ω) for all w ′ ∈ Y .• S * = g(ω * ).• Meta-information sets in the game with incomplete information are defined as sets of information sets, across various EFGs, in which nodes has the same action set and the same observations, with observations defined as all information available at a given information set. Since we are at the interim stage of the game, we can restrict our attention to those information sets with belief b i (ω * ). In the II-MAID resulting from the above operations, the information sets as per Definition 8 correspond one-to-one with those in the game with incomplete information, as they are defined by sets of observation-action set pairs, with observations defined by the values of parents of the given decision variable. efg2maiddetermines the parents of a decision variable according to those ancestors of nodes in a given intervention set that have the same value in paths to each node. As a result, there is a one-to-one correspondence between meta-information sets in a game with incomplete information, and the resulting II-MAID, and since action sets of decision variable are preserved by efg2maid, strategies can easily be mapped to policies.• More precisely, we can define a bijection between meta-information sets in G and information sets in S as follows. Given ω * , we denote the meta-information sets in G corresponding Justification: We claim in the abstract to have developed a new framework with a strong theoretical connection to dynamic games with incomplete information and no common prior over types. We also claim to have proven the existence of equilibria concepts. We have done both of these things, along with the rest of the stated achievements.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: We had no research subjects or data, so no harms were caused in this way. We believe our work is very unlikely to have negative societal impact via the listed categories, as it is purely a theoretical framework and we did not train any models. Our work does not touch on any political issues.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Justification: There are no assets associated with the paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. 


Decision: While the reviewers all find merit with the paper, they continue to have unresolved concerns regarding various aspects of the paper, including clarity, relevance to the Safe-ML track, and establishing the main contributions of the paper.  I hope the suggestions and feedback can help the authors to improve the paper.

Reviews: review: paper_summary: paper_summary: This work extends the theoretical framework of multi-agent influence diagrams (MAIDs) with incomplete information (II-MAIDs) to explicitly capture this complex form of reasoning. The primary theoretical contribution is the proof of the existence of Nash equilibria, although, in general, these equilibria are impossible for agents to identify.
strength_weakness: strength_weakness: 1. One of my major concerns is the audience of this work. Given that this work is submitted to the safe ML track of NeurIPS, I expect more discussion on the relevance of this framework to AI safety. The author should elaborate on what they imply by “safety” rather than making a very brief claim about its relevance in the related work and conclusions sections.

2. The discussion of theory of mind is also lacking, given that this is well-motivated. There have been extensive studies on machine theory of mind, ranging from early studies [1-2] to recent studies on LLMs [3-4]. There has also been research connecting Theory of Mind to Game theory [5] and Interactive POMDP [6]. See the survey [7] for details. Overall, this work needs significant improvement in discussing related work for readers to evaluate its contribution and relevance to NeurIPS.

[1] Rabinowitz, Neil, et al. "Machine theory of mind." International conference on machine learning. PMLR, 2018.

[2] Jara-Ettinger, Julian. "Theory of mind as inverse reinforcement learning." Current Opinion in Behavioral Sciences 29 (2019): 105-110.

[3] Sap, Maarten, et al. "Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs." Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.

[4] Ma, Ziqiao, et al. "Towards A Holistic Landscape of Situated Theory of Mind in Large Language Models." Findings of the Association for Computational Linguistics: EMNLP 2023. 2023.

[5] Yoshida, Wako, Ray J. Dolan, and Karl J. Friston. "Game theory of mind." PLoS computational biology 4.12 (2008): e1000254.

[6] Çelikok, Mustafa Mert, et al. "Interactive AI with a Theory of Mind." Computational Modeling in Human-Computer Interaction. 2019.

[7] Albrecht, Stefano V., and Peter Stone. "Autonomous agents modelling other agents: A comprehensive survey and open problems." Artificial Intelligence 258 (2018): 66-95.
questions: questions: 1. “ToM is characterised by multi-agent interactions involving higher-order intentional states, such as beliefs about beliefs, or, in the case of deception, intentions to cause false beliefs…” ToM refers much more broadly than false beliefs and intentions to create false beliefs. How would the authors position other mental states, e.g., emotions, in this framework? Would “belief” be a better scope?

2. Why is this framework relevant to machine learning safety, especially when much of today's safety concerns arise from large-scale pretrained systems like large language models?
limitations: limitations: Yes
paper_summary: paper_summary: The paper introduces a new framework Incomplete Information Multi-Agent Influence Diagrams (II-MAIDs) for modeling complex multi-agent interactions involving theory of mind (ToM) and higher-order beliefs. The authors prove the equivalence between II-MAIDs and Incomplete Information Extensive Form Games (II-EFGs) at the interim stage. The paper also shows the existence of Nash equilibria in II-MAIDs under certain conditions.
strength_weakness: strength_weakness: * From my perspective, the proposed II-MAID framework appears overly complicated for modeling Theory of Mind (ToM), which is fundamentally a straightforward psychological mechanism observed in daily human interactions. The paper's approach may overcomplicate a concept that should be more intuitively represented.
* The paper introduces numerous assumptions and definitions without clear explanations which hinders the readability. As a non-expert in the field, some details in the paper are difficult to read. 
* It is unclear whether the model can be scaled and applied to larger, more realistic scenarios, where ToM takes place more frequently.
* The paper lacks experiments that validates the model.
questions: questions: * Could the authors elaborate on how this framework might be scaled up? 
* Given that Theory of Mind is fundamentally about real-life interactions, would it be possible for the authors to provide some experiments or toy examples to illustrate the key concepts?
limitations: limitations: N/A, see weaknesses.
paper_summary: paper_summary: This paper extends the framework of multi-agent influence diagrams (MAIDs) to explicitly capture complex forms of reasoning corresponding to Theory of Mind (ToM) as required for the interaction of Multi-Agent Systems with human users.  It introduces the framework of incomplete information MAIDs (II-MAIDs) for explicitly modeling higher-order beliefs in multi-agent interactions alongside probabilistic and causal dependencies between variables. Using results connecting EFGs to MAIDs, the authors demonstrate a natural mapping between strategies in the two frameworks that preserves expected utilities according to the agents’ subjective models.
strength_weakness: strength_weakness: The major issue I would raise for this paper is one of relevance to NeurIPS, even in the extended sense. While a major rationale for the paper appears to be its potential application to AI Safety, in the NeurIPS context there does not seem to be enough outreach to current AI models, at least in a way in which they could be interfaced to the proposed ID model. This means some consideration of how current models may form ‘beliefs’, and this was not entirely obvious from the paper’s Title and Abstract. Perhaps my expectation was unrealistic, but I had imagined an attempt to unify formal ToM issues with ToM properties that are known to be associated to LLM, under a framework where this approach would federate or wrap formal agentic methods around, say Agentic LLM. With this comment I am not criticising the authors for not having written another sort of paper, I am simply pointing the perceived gap that may exist between this approach and the NeurIPS constituency. Further evidence would be the absence of references to NeurIPS paper and the relative dearth of mainstream AI venues in the references (to the notable exception of AIJ). Overall, it appears that AAMAS might be a better venue to host this type of paper. 

The paper does not really clarify its ToM framework which references both “multi-agent interactions” as well as “higher-order intentional states” but these aspects are not part of further formal developments. It also mentions “belief hierarchies of arbitrary and infinite depth” and this raises the issue of whether such a formal approach is realistic when it comes to ToM, in particular in the interactions between agents and human users. 

Despite an early reference to AI Safety and a mention in the paper’s abstract, there is little in the paper that actually progresses the discussion on AI Safety, which is only used marginally through ID examples, such as the one of Figure 2.
questions: questions: In section 5.1 you suggest that beliefs could be extended to policies. Can beliefs be extended to high-level formal concepts that may not be interoperable with other ToM conceptions? Or should beliefs be restricted to epistemic aspects or intentions only?
limitations: limitations: The limitation section begins with a number of upbeat statements that would better be placed in the conclusion or parts of the abstract. The main identified limitation, which echoes the discussion of section 5.1 is verbatim: “The main limitation of our work is the lack of a useful solution concept.” appears a quite severe restriction. While not affecting the solid grounding of the approach it considerably restricts its impact at its current stage of development.

Comments: title: Post-rebuttal comments
Reviewer's response to rebuttal
comment: Thanks, that seems reasonable :)
Thanks for sharing your views. I feel that this manuscript needs to be significantly rewritten to fit the safe ML track. I will keep this borderline score, but I can understand if the AC/PC decides to accept this work if they see it appropriate.
Thanks for your engagement :)
- Indeed, the LM demo does not take long in itself -- but the point of it is to show how our theoretical framework can be applied to LMs more generally, in a well-established ToM case. It's obviously beyond scope to conduct a large scale empirical study of LM ToM as in Bubeck and follow-up work. Our work is largely complementary to that line of study -- it shows how ToM tests on LMs can be understood theoretically. We believe a major role of introducing mathematical formalism is to create a precise language that can help resolve confusions related to LM ToM.
- We feel that our assumptions on how the theory applies to LMs are well-grounded in the literature. As stated in the PDF and general comment: LMs can be understood as having internalised a "subjective causal model" ([A],[B]) -- though we appreciate this view is quite nuanced and not obvious without the context of the literature on causal foundations of agency.
- Our ad hoc formalism of the Sally-Anne case seems to quite naturally capture the situation to us. Where do you think it goes wrong?
- Whilst our theory doesn't capture ToM in general, it does capture higher-order beliefs, which are the component tested in the LM demo.

[A] Ward, F. R., et al. (2024, May). The Reasons that Agents Act: Intention and Instrumental Goals. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems (pp. 1901-1909).

[B] Richens, J., & Everitt, T. Robust agents learn causal world models. In The Twelfth International Conference on Learning Representations.
Thanks. Given our demonstration of how our framework can be applied to LMs, and our broader discussion of how this type of literature applies to ML systems in general, could you please say more about how we could improve the paper's applicability to real-world scenarios? That is, what kind of experiments would you like to see that would convince you that our theory does apply in practice?
Thank you for submitting detailed comments including some additional work specific to my own review.
After reading other reviewers' comments I share their concern about relevance to NeurIPS (as opposed to AAMAS) in particular the Safe ML aspects.
On the core issues I highlighted, I found the answer disappointing, and to some extent counter-productive. What the pdf contains, which is a direct entry of a Sally-Anne like test in gpt-4 can be done in two minutes, and falls short of the detailed approach taken by various papers since Bubeck et al. [2023]. The additional diagrams are simply an ad hoc formalization of the test itself with some of the authors' assumptions, but without any additional evidence. My main issue here is that to some extent the authors are making their own assumptions about ToM (how ToM effects can be captured by LLM, how children "have or haven't ToM") which are unsubstantiated, and producing a long list of ToM papers does not solve the problem, even less when reviewers tend to be already familiar with quite a few of them, were it only by the way they are selected for reviewing under NeurIPS rules. 
Since the authors agreed that "their framework does not capture broader aspects of ToM" why are they trying to further complexify the issue by introducing and agentic model of LLM together with causal models, when these are still highly debated issues without first a proper analysis of current hypotheses explaining LLM's ToM empirical abilities?
Thank you for your response. While I appreciate the authors for adding a proof-of-concept example, I still have concerns about the paper's applicability to real-world scenarios and its relevance to the broader NeurIPS audience. Given these, I will maintain my current score.
Thanks! We appreciate the push back :) 

*“(1) what they mean by a "safe" AI agent”*

Here’s how we see it.

- “Agents” can be understood quite broadly in this literature – they are basically modeled by decision and utility variables, and can capture RL agents, supervised learning algorithms, and even LMs given certain assumptions (as demonstrated in our new example).
- Influence models, and their extensions (MAIDs etc), are often used to define high-level concepts relevant for safety, such as harm, deception, fairness, etc (see the global comment). Formal specifications of these concepts enable “safe agent design” in a number of ways.

Safe agent design:

- In this literature, safety often refers to safe *incentives*. Given a formal definition of a concept, e.g., unfairness, we can prove conditions about the training algorithm which guarantee there is no harmful incentive (e.g., given a classification algorithm, we can use CIDs to guarantee there is no incentive to unfairly use sensitive attributes for the classification). 
- Alternatively, the specification of the safety concept can be used for formal verification techniques, such as a shielding RL algorithm which prevents deception from being learned. 

 *”(2) what are the theoretical abstractions of safety and which aspects of safety are covered by this framework”* 

Hopefully the above makes this somewhat clearer. There are many theoretical abstractions of safety that we might want to model in our framework, including manipulation, deception, coercion, threats, failures of cooperation. 

Many of these concepts have resisted formal specification so far, in part because of the lack of a suitable formalism. As we noted in the general response, past notions of deception have been insufficient because of assumptions which we relax. As another example, manipulation has so far not been formalised in part because it is often considered to be “covert” – meaning that the manipulated agent is unaware of it, but in game theoretic settings agents are typically assumed to know which policies the other agents are playing to achieve a Nash equilibria (an assumption which we relax). 

Additionally, as noted in the global, formalising recursive beliefs (as we do) has been highlighted as an important open problem in cooperative AI.

*”(3) what are some concrete application scenarios of this framework.”*

Here is an example of the type of work which we imagine being built on our framework:

- First, formally define “manipulation” in II-MAIDs (as we argued above, this plausibly requires our technical machinery related to ToM)
- Integrate this formal specification into learning algorithms which guarantee that the system does not learn to manipulate other agents, e.g., users
  - Using extensions of incentive design concepts to our framework
  - Or formal verification style algorithms
- These methods should be generally applicable to systems like (MA)RL agents and even fine-tuning algorithms for LMs 

This general process could be applied to any safety application of interest. Another class of applications of formal specifications are algorithms for detecting unsafe bahaviour.

**However, we appreciate that our paper does not address these safety applications directly. We chose “safety in ML” as the primary area because we had these applications in mind, and think that our paper provides a strong theoretical foundation for them. If possible, and if the reviewers think it is more appropriate, we would be happy to change the primary area to a more suitable one, e.g., theory. If this is not possible, then we believe our paper is of sufficient interest to the Safe ML community, given its applicability to systems such as LMs, and the rich surrounding literature on safety.**
Thank you for responding to my concerns.

- ToM discussions: Thank you, I would love to see and suggest that the authors integrate these discussions into the next interaction.
- Safety discussions: I am still very confused about the relevance of this paper to safe AI. "Much of the literature on CIDs is relevant for safe AI" does not naturally entail that this work is also relevant. I would expect the authors to clarify (1) what they mean by a "safe" AI agent; (2) what are the theoretical abstractions of safety and which aspects of safety are covered by this framework; (3) what are some concrete application scenarios of this framework.

In summary, I did see the merit of this work (in multiagent interaction) to some readers, especially AAMAS readers. I am not sure about its relevance to the NeurIPS Safe ML track.