## 3 Inferring an Agent's Desires
### Reward Function Discrimination
#### 3.1.7 Adversarial Methods  
[PERSON] and [PERSON] [83] introduced a model-free adversarial framework to learn a policy, which is trained through RL under a reward function obtained through MaxCausalEnt IRL. Although this work did not contribute new IRL algorithms, it proposed a new framing of the problem analogous to generative adversarial networks (GAN) [84].  
In tasks with large state-action spaces or unknown transition dynamics, the computation of the partition function in the MaxEnt objective is intractable [85]. Adversarial IRL methods [72; 86] approximate the MaxEnt objective through sampling. A synthetic policy \(\pi_{\omega}\) generates trajectories maximising an entropy-regularised policy objective \(\mathbb{E}[\sum_{t}\hat{R}_{\theta}(s_{t},a_{t})-\log\pi_{\omega}(a_{t}|s_{t})]\), whereas a binary discriminator  
\[D_{\theta}(s,a)=\frac{\exp\bigl{(}\hat{R}_{\theta}(s,a)\bigr{)}}{\exp\bigl{(} \hat{R}_{\theta}(s,a)\bigr{)}+\pi_{\omega}(a|s)} \tag{49}\]  
discerns between synthetic and expert trajectories. The two networks are trained adversarially, resulting in a reward function approximator \(\hat{R}_{\theta}\) and a policy \(\pi_{\omega}\). This shares similarities with the earlier approach in [43] (Section 3.1.1.3).  
The adversarial IRL approach has been extended for metalearning [85; 87] (Section 6); improved with an information bottleneck [88], semantic rewards [89], or end-to-end differentiability through self-attention [90]; and adapted to language-conditioned tasks [91].  
In this subsection, we have outlined the many approaches that have been proposed to discriminate between the several reward functions that could explain a given set of behavioural demonstrations. Maximum margin methods do so by attempting to maximise the margin between the chosen reward function and any other alternatives (Section 3.1.1). Probabilistic methods interpret the rewards as a random variable and the state-action pair demonstrations as evidence, framing the problem as Bayesian inference to obtain a posterior distribution over rewards (Section 3.1.2). This is extended in maximum entropy methods, which seek to account for interdependencies between action choices at the trajectory level to provide a more accurate way to select from plausible reward functions (Section 3.1.3). A gradient method is proposed to obtain a MAP estimate of the rewards without needing to integrate over the entire solution space, showing how most previous methods can be unified under this perspective (Section 3.1.4). Others, by approximating the environment by using the LMDP construct, are able to recover the reward function without needing the actions to be given in the demonstrations (Section 3.1.5). A class of more direct methods exploit the algebraic definition of the IRL problem to find solutions by means of optimisaion techniques (Section 3.1.6). Finally, adversarial methods train a synthetic policy to generate trajectories and a discriminator to discern between expert and synthetic trajectories, converging into a useful reward approximator (Section 3.1.7). All of these approaches assume the solution space for reward functions is defined a priori. In what ways may this solution space be defined? In other words, how may these reward functions be characterised?