## 3 Inferring an Agent's Desires
### Reward Function Discrimination
#### 3.1.6.2 Cascaded Supervised IRL  
Subsequent work by [PERSON] et al. [79] retrieve a reward function estimate by chaining two generic supervised learning steps. First, a multinomial classification step yields a \(Q\)-function surrogate \(q\), as in [78]. If the transition dynamics for the environment are known, a reward function can be obtained directly based on the Bellman equation from this classifier:  
\[R(s,a)=q(s,a)-\gamma\sum_{s^{\prime}}T(s,a,s^{\prime})q(s^{\prime},\pi_{\text{ C}}(s^{\prime})). \tag{45}\]  
A key contribution of this work is removing the requirement of knowing the transition dynamics by approximating \(R\) through regression, as the second step in the process. Although the regressors \((s,a)\) are provided, this requires a response variable \(\hat{r}\), obtained from the Bellman equation with  
\[\{\hat{r}_{t}=q(s_{t},a_{t})-\gamma q(s_{t+1},\pi_{\text{C}}(s_{t+1}))\}_{t}. \tag{46}\]  
The resultant dataset is \(\mathcal{D}_{R}=\{(s_{t},a_{t}),\hat{r}_{t}\}_{t}\). However, samples for state-actions that differ from the expert's \((s_{k},a^{\prime}\
eq a_{k})\) are needed to reduce the regression error. The authors address this with a synthetic augmentation of the regression dataset with artificial samples \(((s_{t}=s_{k},a^{\prime}),r_{lo})_{t,\forall a^{\prime}\
eq\pi_{E}(s_{t})}\). The reward for these samples is set to ensure it is always lower than that of the expert's samples: \(r_{lo}=\min_{k}\hat{r}_{k}-1\).  
3.1.6.3 Empirical Q-Space Estimation  
[PERSON] et al. [61] provide analytical solutions to the IRL problem through constraints imposed by the policy observations, which can be optimal, perturbed, or incomplete. The so-called inverse Bellman equation  
\[R(s,a)=Q^{*}(s,a)-\gamma\sum_{s^{\prime}\in\mathcal{S}}T(s,a,s^{\prime})\sum_{ a^{\prime}\in\mathcal{A}}\pi^{*}(s,a)Q^{*}(s,a) \tag{47}\]  
shows a one-to-one relationship between \(Q\)-functions and rewards. If the transition dynamics are known, all we need to obtain a valid \(R\) is the \(Q\)-function, because the optimal policy is assumed to be either deterministic (Equation (3)) or Boltzmann (Equation (11)). Given the demonstrations and a prior on policies we can obtain an empirical Bayesian estimate of the policy \(\hat{\pi}(s,a)\). If the optimal policy is known for a given state, we have \(Q(s,\cdot)=0\) for actions that are suboptimal and uniform across the optimal actions. If the optimal policy is noisy, \(Q(s,a)=\frac{\log(\hat{\pi}(s,a))}{\alpha}+V(s)\) and we can set \(V\) arbitrarily. If no information is available for the policy at a given state, invoking the advantage function \(A(s,a)=Q(s,a)-V(s)\) we have multiple degrees of freedom: arbitrary \(V(s)\), and \(A(s,\cdot)\) constrained to be \(\leq 0\) and have at least one zero-valued element because every state has at least one optimal action.  
3.1.6.4 Direct Loss Minimisation  
[PERSON] et al. [80] propose performing direct (deterministic) policy search on a reward function \(R_{j}(\tau)=-L(\tau_{j},\tau)\) that reflects the loss between observed \((\tau_{j})\) and proposed \((\tau)\) trajectories. That is, optimise the reward parameterisation weights through\[c(\theta)=\sum_{j}^{M}R_{j}(\tau) \tag{48}\]  
where \(\tau\) is a trajectory under parameters \(\theta\) (e.g., generated by the optimal policy for \(R_{\theta}\)) in the same MDP as the demonstrations. Any off-the-shelf policy search method can be used to optimise \(\theta\), with the authors employing the covariance matrix adaptation evolutionary strategy (CMA-ES) optimiser.