# Data-Driven Policy Learning Methods from Biological Behavior: A Systematic Review
## 3 Strategy Learning from Demonstration: Imitation Learning
### Improved GAIL Algorithms  
The drawbacks of GAIL in practical applications limit its adaptation to applications with high sample acquisition costs. The problem of model collapse originates from GAN, e.g., the Generator finds a data type that easily deceives the Discriminator and thus continues to generate that type, that is, resulting in the Generator having a similar single type of output, thus depriving the generated samples of diversity. Low sample efficiency for environmental interactions relates to how the policy is learned. The stochastic policy of GAIL causes the agent to randomly select behaviors (including undesirable ones) according to the probability distribution, which may lead to low efficiency for the agent to search the environment. In addition, GAIL learns policies directly through a model-free policy instead of using environmental models. Hence, agents require many environmental interactions to maximize the reward, and the expensive training process leads to low sample efficiency. To improve the performance of imitation learning, many attempts have been made to solve these problems in addition to GAIL. The improvement in the GAIL model collapse caused by GAN is also inspired by an improved approach to GAN. Extending imitation learning from a single-model assumption to learning from multiple models can effectively avoid model collapse and satisfy the requirements of practical applications. In the case of autonomous driving, for example, an intelligent agent is expected to imitate not only the fast-driving model but also learn multiple driving models, including safety. * Conditional GAIL (cGAIL): Based on the idea of conditional GAN (cGAN) [88], i.e., when multiple model labels for imitation learning can be obtained directly from expert samples, these model label data can be directly used as conditional constraints in the process of policy and reward function construction. Hence, cGAIL [24; 25], which can perform imitation learning under multiple model conditional constraints, is proposed. * GAIL with Auxiliary Classifier (AC-GAIL): AC-GAIL [26; 27; 28] was inspired by GAN with auxiliary classifier (AC-GAN) [89], which introduced an auxiliary network model for multiple model imitation learning. AC-GAIL does not use model label data directly but uses an additional auxiliary network to train the model label data, which can extract abstract deep features from the samples and efficiently use the model label data. * Information Maximizing GAIL (InfoGAIL): Inspired by information maximizing GAN (InfoGAN) [90], InfoGAIL [29; 30] employs the idea of mutual information to judge the correlation between samples and maximizes the mutual information from the relationship between the generated samples and the randomly sampled hidden modal data to interpret the meaningful implicit modal data in the expert samples. Thus, it realizes unsupervised multimodal imitation learning and provides a certain degree of interpretability. * Variational Auto-Encoder GAIL (VAE-GAIL): Based on the principle of variational auto-encoder GAN (VAE-GAN) [91], VAE-GAIL [31; 32] replaces the generative model in the original GAIL with a variational self-encoder to obtain meaningful modalities in the sample by maximizing the mutual information between the expert samples and the implicit modal data inferred by VAE. Because of the continuity of the implicit modal data inferred by VAE, the strategies learned by VAE-GAIL have a higher level of diversity. * Wasserstein GAIL: Based on the improvement of GAN by Wasserstein distance [92], Wasserstein GAIL is proposed [33]. It employs Wasserstein distance instead of KL divergence or JS divergence to compute the distance between the generated distribution and the true distribution, i.e., the loss function of the Generator. In contrast,the difference measured by Wasserstein distance is continuous; hence, the training is smoother. Furthermore, no modal collapse was observed in the experiment [92].