## 3 Preliminary
### Gail  
GAIL represents a sophisticated approach that synergizes the principles of Generative Adversarial Networks (GANs) with IL. This technique is particularly efficacious in complex settings where articulating a precise reward function poses significant challenges. GAIL acquires the capability to execute specific tasks by mimicking the behaviors of experts, circumventing the need for explicit reward signals in the learning process. Within this paradigm, the expert's strategy is denoted by \(\pi_{E}\), and the learner's strategy is signified by \(\pi\). The formulation for the learner's strategy \(\pi\) encompasses the generation of expected trajectories, illustrating the process of learning through observation and emulation of expert actions. \[\mathbb{E}_{\pi}[f(s,a)]\triangleq\mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^{ t}f(s_{t},a_{t})\right] \tag{1}\]  
Here, \(s_{0}\sim\rho_{0}\), \(a_{t}\sim\pi(a_{t}|s_{t})\), and \(f\) is an arbitrary function. At the heart of GAIL lies the emulation of the expert policy's state-action occupancy measure. This measure is a critical probability distribution that captures the frequency of an agent executing certain actions in given states under a specific policy. Unlike behavior cloning, which directly replicates expert trajectories without necessitating environmental interaction and consequently faces the challenge of compounding errors--where slight inaccuracies in imitation accumulate over time--GAIL necessitates active engagement with the environment. This interaction is pivotal for GAIL as it enables the learning policy to refine its actions based on the dynamic feedback from the environment, thereby mitigating the risk of error propagation inherent in behavior cloning. The GAIL algorithm includes a discriminator D and a policy \(\pi\), where policy \(\pi\) acts as the generator in the Generative Adversarial Network. The discriminator \(D\) takes state-action pairs \((s,a)\) as input and outputs a probability between 0 and 1, indicating how likely it is that the pair \((s,a)\) originates from the agent's policy rather than the expert's policy. The goal of the discriminator is to make the output for expert data as close to 0 as possible and the imitator's output close to 1, thus clearly distinguishing between the two. Based on the above, we can define the loss function for the discriminator \(D\) as:  
\[L(\phi)=-E_{\rho_{\pi}}[\log D_{\phi}(s,a)]-E_{\rho_{E}}[\log(1-D_{\phi}(s,a))] \tag{2}\]  
where \(\phi\) represents the parameters of the discriminator. The core idea of this algorithm is to make the trajectories generated by the policy \(\pi\) be misidentified by the discriminator as coming from the expert. Using this approach, we do not need to preset a specific reward function; instead, we can define the reward \(r(s,a)\) as \(-\log(D(s,a))\). In this setting, if the discriminator \(D\) judges the state-action pair \((s,a)\) to be extremely similar to the expert's behavior, the value of \(D(s,a)\) is close to 1, and then \(-\log(D(s,a))\) will be a smaller negative value, equivalent to a higher reward. Conversely, if the value of \(D(s,a)\) is close to 0, implying that the discriminator considers the state-action pair unlike the expert's  
Figure 3: The framework of Meta-RL. behavior, then the reward \(-\log(D(s,a))\) becomes a larger negative value, representing a lower reward. Building on the theoretical framework described, we embarked on a series of comparative analyses involving three distinct algorithms, applied to the Cartpole task. The initial step entailed the generation of expert data, for which we opted for the Proximal Policy Optimization (PPO) algorithm, given the relative simplicity of the Cartpole task. From the generated trajectory, we sampled 30 state-action pairs \((s,a)\). Our comparative analysis revealed that the Generative Adversarial Imitation Learning (GAIL) algorithm required merely 80 training epochs to attain the maximum reward threshold.