## 3 Inferring an Agent's Desires
### Reward Function Discrimination
#### 3.1.1 Maximum Margin Methods
```
AlgorithmMaxMargin(\(\mathcal{D}=\{s_{0},s_{1}^{(j)},\ldots,s_{H}^{(j)}\}_{j=1}^{m},\phi\)) Initialisation
1\(\Pi\leftarrow\{\}\) for\(k\gets 1\)to\(K\)do
2\(\hat{R}\leftarrow\theta^{\mathsf{T}}\phi\)\(\pi_{k}\leftarrow\)OptimalPolicy(MDP\(\backslash\)R, \(\hat{R}\)) Find best \(\pi\) under \(\hat{R}\)\(\Pi\leftarrow\Pi+\{\pi_{k}\}\)\(\tau^{\pi_{k}}\leftarrow\)GatherTrajectories(MDP\(\backslash\)R, \(\hat{R}\), \(\pi_{k}\), m)
3\(\theta\leftarrow\)UpdateWeights(\(\theta\), \(\phi\), \(\tau^{\pi_{k}}\), \(\tau^{\pi^{*}}\), \(\Pi\))
4 end for
5 return\(\hat{R}\) ProcedureUpdateWeights(\(\theta,\phi\), \(\tau^{\pi}\), \(\tau^{\pi^{*}}\))
6\(\tilde{v}^{\pi}\leftarrow\)EstimateFeatureExpectations(\(\tau^{\pi}\), \(\phi\)) Equation (16)
7\(\phi^{\pi^{*}}\leftarrow\)EstimateFeatureExpectations(\(\tau^{\pi^{*}}\), \(\phi\))
8\(L\leftarrow\sum_{\pi_{i}\in\Pi}p(\theta^{\mathsf{T}}\tilde{v}^{\pi_{i}}-\theta^ {\mathsf{T}}\tilde{v}^{\pi^{*}})\) Equation (15), (17)
9\(\theta\leftarrow\)Optimise(\(L\), \(\theta:|\theta_{i}|\leq 1\)\(i=1,\ldots,d\))
10 return\(\theta\)
```  
**Algorithm 1**Algorithm from [40]  
In the context of ToM, the given trajectories represent the observer's knowledge of the actor's behaviour. The longer the trajectories and the larger the set of trajectories (hyperparameters \(H\) and \(m\), respectively), the better the observer can be said to know the actor. The resultant \(\hat{R}\) is the observer's model of what drives the actor's behaviour (i.e., its utilities), which may be used in conjunction with policy estimates \(\pi\in\Pi\) (i.e., its probabilities) to predict its future behaviour. In Figure 1, we group these two variables together conceptually as the observer's model of the agent (green, dashed outline). The rationality of the actor is based on these two sources of information [41]. The observer requires a model of the environment (MDP\(\backslash\)R) to be able to estimate the model of the agent. This is a sensible requirement for any agent. In Algorithm 1, it is assumed to be completely faithful to the real environment (Figure 1, yellow with dashed and solid outlines, respectively). One outstanding question is the meaning of the basis functions, or environment features, \(\phi_{i}\) in the context of ToM. We place them conceptually within the observer, as depicted in Figure 1 (orange). The cardinality \(d\) of the space \(\Theta\) in which we perform the linear approximation of the reward function, and thus its expressivity, depends on how many features the observer makes use of. Intuitively, they stand for the perceptual acuity of the observer--the number of different \"stimuli\" the observer can differentiate amongst and attribute value to. They are likely to differ to that of the actor; that is, if the actor does have them in the first place--it may not know its subutilities and simply be guided by its reward function. Simple examples of features in the scenario of an agent crossing the road include whether there is a car present, the speed of the car, the state of the pedestrian crossing traffic lights, etc. Moreover, not only the features, but the state observations themselves may differ between the actor and the observer (e.g., first-person vs. third-person point of view). In [PERSON] and [PERSON] [40] they are \"given\" and fixed. As new trajectories are observed, the same algorithm can be used to update the weights if the current \(\theta\) and \(\pi_{k}\) are used instead of randomly initialising them. We call attention to the fact that, although this was not stated in the algorithm as presented, it can yield the set of policies \(\pi\in\Pi\), as well as their respective \(\tilde{v}^{\pi}\), \(\tilde{V}^{\pi}\), and different reward function estimates \(\hat{R}\) under which each of the policies were optimised.