# Data-Driven Policy Learning Methods from Biological Behavior: A Systematic Review
## 3 Strategy Learning from Demonstration: Imitation Learning
### Generative Adversarial Imitation Learning  
The purpose of simple BC is to match the behavior of the demonstrating teacher, and it has significant limitations in terms of environmental interactions. This defect was improved by the emergence of GAIL [21]. GAIL is used to extract strategies directly from expert demonstration data. It has the advantage of being a generative adversarial network (GAN) [85], thus outperforming BC in complex applications and gradually becoming a popular model-free IL framework.  
First, the concept of GAN is introduced. GAN is a deep generative model that generates generative samples for fitting training samples by transforming deep features abstracted from random input noise to the underlying features, which can be regarded as the inverse process of deep classification models. The deep generative model, which aims to generate realistic samples, is called a \"Generator\", and the deep classification model, which aims to discriminate the source of the samples, is called a \"Discriminator\". They are trained adversarially by playing a game with each other such that the samples generated by the Generator can fit the real training samples well.  
Similarly, GAIL adopts the idea of adversarial training. It utilizes two neural networks to represent the strategy and reward function and continuously optimizes the parameters of these two networks by adversarial training. In other words, the strategy that outputs the action according to the input state is considered a Generator, and the reward function that outputs the reward value according to the input generation samples is considered a Discriminator. The learning process of the strategy and the solving process of the reward function are the training processes of the Generator and the Discriminator, respectively. The parameters are updated during the training process using a gradient  
\[\min_{\pi}\max_{D}\mathbb{E}\pi[log(D(s,\,a))]+\mathbb{E}\pi_{E}[log(1-D(s,\,a ))], \tag{1}\]  
where \(\pi\) is the policy to be learned, \(\pi_{E}\) is the policy of the experts, and \(D(s,\,a)\) is the probability determined by the Discriminator. The Generator aims to minimize the cross-entropy loss of the generative samples, whereas the Discriminator aims to maximize the cross-entropy loss of the discriminative model. In addition, the learning objective of GAIL is to optimize the strategy directly instead of computing the reward function before solving it; thus, it avoids the high complexity of the computation process in IRL (Section 4) and can adapt to IL tasks with large-scale complex behaviors.  
Based on the GAN framework, the strategy and reward function in GAIL can automatically extract abstract features from training samples with higher representational capability,which alleviates the dependence of IL on large datasets. Moreover, GAIL significantly reduces computational complexity by learning strategies directly from data and, thus, can better adapt to the complexity of real applications. The method has exhibited excellent performance in autonomous driving [22]. Nevertheless, GAIL also faces the problems of model collapse [86] and low sample efficiency in terms of environmental interactions [23; 87].