## 2 Background
### Boltzmann Policies  
To make stochastic policies more robust to environmental changes, it is desirable that they assign nonzero probabilities to actions other than the optimal to allow for exploration. According to [PERSON] Maximum Entropy Principle (MaxEnt) [38], to find a probability distribution that minimises bias for a given partial set of information requires maximising the amount of entropy (or uncertainty) in the distribution, subject to the known information. Recall the definition of a deterministic optimal policy through the \(Q\)-function (Equation (3)). To define a stochastic policy, a distribution over action choices can be determined from their associated \(Q\)-values instead. Subject to moment matching constraints for the zeroth and first moments  
\[\begin{array}{c}\sum_{a\in\mathcal{A}}\pi(a|s)=1\ \ \ \forall s\in \mathcal{S},\\ \mathbb{E}[Q^{\pi}(s,a)]=\sum_{a\in\mathcal{A}}\pi(a|s)Q^{\pi}(s,a)\ \ \ \forall s\in \mathcal{S},\end{array}\]  
the entropy-maximising distribution is the Boltzmann distribution, or \"Boltzmann policy\" as it is known in the RL context,  
\[\pi(a|s)=\Pr(a|s,R,\pi)=\frac{1}{Z}\exp(\alpha Q^{\pi}(s,a;R)), \tag{11}\]  
with normalising constant, or partition function \(Z(s,R,\pi,\alpha)=\sum_{a^{\prime}\in\mathcal{A}}\exp(\alpha Q^{\pi}(s,a^{ \prime};R))\) and (negative) potential energy \(Q^{\pi}(s,a;R)\). The hyperparameter \(\alpha\) serves as an inverse temperature parameter defining the steepness of the policy distribution, or how \"greedy\" for optimal \(Q\)-values it is. This greediness may be understood as the level of rationality attributed to the agent by the ToM observer. This distribution is known as the Softmax function in the machine learning literature.  
A maximum entropy optimal policy can be obtained through the soft \(Q\)-function  
\[\begin{array}{c}Q^{\pi}_{\text{soft}}(s,a)=R(s,a)+\gamma\sum_{s^{\prime}\in \mathcal{S}}T(s,a,s^{\prime})V^{\pi}_{\text{soft}}(s^{\prime})\\ V^{\pi}_{\text{soft}}(s)\triangleq\log\sum_{a\in\mathcal{A}}\exp\bigl{(}Q^{\pi} _{\text{soft}}(s,a)\bigr{)},\end{array} \tag{12}\]  
where the value function is defined through the LogSumExp function [39]. Actions with higher \(Q\)-values reduce regret, which rational agents are expected to act in accordance with. An alternative and equivalent interpretation of this policy is as proportional to the exponential advantage of an action  
\[\pi(a|s)\propto\exp(Q^{\pi}(s,a)-V^{\pi}(s)). \tag{13}\]  
The Boltzmann distribution provides a smooth parametric model for the action choice distribution in a given state (i.e., a policy) that is shaped by the \(Q\)-function at the state. These qualities, along with the alignment with MaxEnt and the degree of freedom in the temperature hyperparameter, are desirable attributes in modelling rational agents. For these reasons, a sizable proportion of IRL algorithms resort to a Boltzmann assumption when characterising the policy (e.g., Sections 3.1.1.6, 3.1.2.2, 3.1.2.3, 3.1.3.2 and 3.1.6.3). It is common practice to assume the \(Q\)-function is given in a converged state or obtained through dynamic programming (value iteration) or RL methods (e.g., \(Q\)-learning). In the ToM interpretation, the accuracy of the \(Q\)-function with respect to the true values of the MDP encode, in part, the accuracy of the agent's beliefs--a core mental attitude of ToM, as discussed in Section 4. Another core mental attitude in models of ToM are desires, which are encoded as rewards in rational agent models. In the following section, we review IRL algorithms whose emphasis is on recovering these rewards, and discuss how they can provide an effective computational approach to inferring an agent's desires from their behaviour.