## 4 Inferring an Agent's Beliefs
### State Observability  
False belief tasks are prominent ToM assessment tasks [6]. MDPs, while providing a compact paradigm for the study of rational decision making, have a critical limitation in the context of algorithmic approaches to ToM: agents' beliefs about the state of the world are always accurate [19]. Partially observable MDPs (POMDP) are an extension of the MDP construct in which states may not be fully identified by the agent from perceptual information. Instead, agents have a belief distribution \(b(s)\) over \(\mathcal{S}\), given the evidence up to the current time step, denoting the agent's belief that a given state is the actual current state. The formal definition of a POMDP entails a tuple \((\mathcal{S},\mathcal{A},T,D,\gamma,\mathcal{R},\Omega,\mathcal{O})\). The elements that differentiate it from a MDP are a space of observations \(\Omega\) and an observation distribution \(\mathcal{O}=\Pr(o|s)\) over \(\Omega\) encoding what actual states are consistent with an observation (percept). The uncertainty stemming from incomplete state information provides a context in which to model the beliefs of an agent.  
Several IRL methods have been proposed under the POMDP formulation, including a general framework to extend existing IRL methods where agents act according to their beliefs, as opposed to the actual state, so policies are a mapping \(\pi:\Delta\to A\), where \(\Delta\) is the belief simplex in \(|S|-1\) dimensions [127]. Others specifically provide a computational implementation of ToM, using Bayesian inference to reconstruct agents' joint belief state and desires [19]. In this approach, the observer encodes this joint distribution in a dynamic Bayesian network (DBN) with world state (\(Y\)), agent state (\(X\)), percept (\(O\)), belief (\(B\)), reward (\(R\)), and action (\(A\)) variables. The world and agent states result from a decomposition of the MDP state \(S\), because the agent state \(x\in\mathcal{X}\) is fully observed, but the world state \(y\in\mathcal{Y}\) is not. The agent's belief at a given time \(t\) is a probability \(b(y)\) over \(\mathcal{Y}\) denoting the agent's belief that a given \(y\) is the actual state of the world (fixed for the entirety of a given episode). The observer's joint distribution (conditioned on a given sequence up to time \(T\geq t\)) of belief at time \(t\) and reward function can be recovered from the DBN. Beliefs are smoothed retrospectively: the observer's model of the beliefs of the agent at time \(t\) is informed by behaviour up to time \(T\geq t\), bearing similarity to how humans model other's beliefs. Earlier work by the authors under the name of Bayesian inverse planning [33; 128], is a subset of the IRL problem, wherein the reward function is known to be constrained to \(R(s)=-1\;\;\forall s\in S\setminus s_{g}\), where \(s_{g}\) is the absorbing goal state.  
To summarise, one cannot assume that agents have perfect knowledge of the effects of performing an action (i.e., their transition dynamics model may not match that of the environment), or what the actual state of the world is (i.e., their beliefs about the current state may be a more or less accurate distribution over states based on observations of the environment). Furthermore, from a ToM perspective, the observer's transition dynamics and current state models may differ from both the actual environment's and the agent's. For IRL methods that rely on the actual state and transition dynamics being given to be suitable in the ToM context, they need to be expanded to incorporate these considerations about beliefs.  
Orthogonally, there is yet another factor to be considered. Even if an agent were to have a clearly defined reward function and perfectly accurate beliefs, its behaviour may not be in accordance with them, be it due to a lack of skill or due to other extrinsic effects. Bridging the gap between desires and beliefs on the one hand and behaviour (actions) on the other are an agent's intentions.