## 3 Inferring an Agent's Desires
### Reward Function Discrimination
#### 3.1.1.5 Maximum Margin Planning  
The use of (estimated) state-action occupancy measure from demonstrations is further extended by [PERSON] et al. [49], whose goal is to find a reward function under which the optimal policy is similar to the expert's. To do so, they cast the problem as structured prediction, relying on a loss-augmented reward function \(R_{l}=\theta^{\text{T}}F_{j}+l_{j}^{T}\), where \(l_{j}\in\mathbb{R}_{+}^{|\mathcal{S}\times\mathcal{A}|}\) is a loss vector defining the cost of deviating from the expert policy for every state-action pair.  
Each demonstration may be generated in a different MDP, and is given by \(\mathcal{D}=\{(\mathcal{S},\mathcal{A},F,\mathcal{G},\mu,l)_{j}\}_{j=1}^{m}\). From this, they introduce an objective function measuring the difference in performance between a policy with occupancy \(\mu_{\theta}=\arg\max_{\mu\in\mathcal{G}_{j}}(\theta^{\mathsf{T}}F_{j}+l_{j}^{ \mathsf{T}})\mu\) (optimal under the loss-augmented reward function), and the given demonstration \(\mu_{j}\) (without loss-augmentation), based on a quadratic programming formulation (in accordance with the hinge loss form)  
\[c_{q}(\theta)=\frac{1}{m}\sum_{j=1}^{m}\beta_{j}\Big{(}\left(\theta^{\mathsf{ T}}F_{j}+l_{j}^{\mathsf{T}}\right)\mu_{\theta}-\theta^{\mathsf{T}}F_{j}\mu_{j} \Big{)}^{q}+\frac{\lambda}{2}||\theta||^{2}. \tag{22}\]  
The form of the loss function imposes a margin by which the solution obtained is better than any other possible solutions. Using the occupancy measures from the expert policy has the effect of making rewards for high occupancy state-action pairs larger, which in turn encourages similarity between the policies, as well as discouraging degenerate solutions [51]. The optimisation of the weights \(\theta\) is performed through gradient descent by using the subgradient of the objective function  
\[g_{\theta}^{q}=\frac{1}{m}\sum_{j=1}^{m}q\beta_{j}\Big{(}\left(\theta^{ \mathsf{T}}F_{j}+l_{j}^{\mathsf{T}}\right)\mu_{\theta}-\theta^{\mathsf{T}}F_{ j}\mu_{j}\Big{)}^{q-1}F_{j}(\mu_{\theta}-\mu_{j})+\lambda\theta, \tag{23}\]  
where \(\beta_{j}\) is a data-dependent normalisation coefficient, \(q\in\{1,2\}\) is a choice of slack penalty type (for \(\ell\)1- and \(\ell\)2-loss, respectively). [PERSON] and [PERSON] suggest the use of a loss vector \(l(s,a)=1-\mu_{j}(s,a)\)[35]. The (near-)optimal policy \(\pi_{\theta}\) is obtained with RL methods in the particular MDP\(\backslash\)R under the loss-augmented reward function, and provides the occupancy \(\mu_{\theta}\). Optionally, the weights \(\theta\) can be projected on to additional problem-specific constraints after every update. The weights can be learned offline from a training set \(\mathcal{D}\), or online for each observation \(\mathcal{D}_{j}\). The occupancies \(\mu_{j}\) can be obtained equivalently from trajectories \(\tau_{j}\) when the demonstrations are in said format instead.  
Prior knowledge can be included in the form of further constraints on \(\theta\), such as by explicitly penalising certain features, or regularising the learning procedure around a prior belief about \(\theta\) instead of approximately 0. Additionally, it can be included through the loss vector \(l\) if certain state-action pairs are known to be poor choices [49].  
When there is no single reward function that maximises the margin, such as when the agent's behaviour is suboptimal or no data is available for parts of the state space, this method is limited [52].