# Data-Driven Policy Learning Methods from Biological Behavior: A Systematic Review
## 3 Strategy Learning from Demonstration: Imitation Learning
### Behavior Cloning  
As a basic form of IL, simple BC can be viewed as a supervised learning method that transforms the imitation problem into a learning process to map states to actions. While supervised learning uses labeled training data, BC uses the training dataset \(\{(s_{1},\,a_{1}),\,\cdots,\,(s_{n},\,a_{n})\}\), which consists of demonstration examples in the form of \((state,\,action)\), where \(n\) represents the length of the demonstration trajectory. BC uses deep neural networks to train the demonstration data and learns implicitly from it. Thus, obtaining realistic behavioral strategies depends heavily on accurately mapping decision pairs in the dataset.  
Most BC studies learn strategies and understand behaviors from human or animal demonstrations. Here, the human or animal providing the behavioral demonstration is defined as the \"teacher\", and the computer or robot is defined as the \"agent\". Learning from observation (LfO) demonstration data can be categorized into (1) first-person and (2) third-person views:  
1. In the first-person view LfO, the demonstrating teacher records the states and actions during the activity through the worn sensors. The agent can directly imitate the teacher's decision from the first-person view without transforming the demonstration data. This approach has been extensively used in humanoid robotics and autonomous driving, such as imitation learning using the VR teleoperation approach [8], and end-to-end autonomous driving BC [9; 10]. The advantage of this approach is that it does not require the remapping of demonstration data and can provide accurate measurements. However, because of the size of the sensors and the customization environment, collecting demonstration behaviors by mounting sensors on the teacher cannot be used in all cases.
2. The third-person view LfO collects demonstration data through external behavioral observation, which requires the perspective transformation of the observed data, i.e., mapping the recorded data from the third perspective to the agent from the first perspective [11; 12]. It is primarily based on external cameras and thus can be applied to robot manipulation without being limited to customized experimental environments [12; 13]. Compared to the former method, the third-person view LfO is more flexible regarding the applicable environments; however, the collected demonstration data may not be sufficiently precise.  
Dataset aggregation (DAgger) improves the interaction between the algorithm and the environment through data augmentation and aggregation based on simple BC [84]. DAgger is an online learning algorithm that continuously enables acquired strategies to interact with the environment to generate new data. Subsequently, the newly acquired data are aggregated to improve the dataset. It can learn expert behavioral strategies in complex environments, such as autonomous driving tasks [14; 15] and strategy learning in complex 3D game scenarios [16].  
In the practical application of strategy learning, IL with a combination of RL is employed in addition to the above methods. This integrates the advantages of both algorithms. This approach first performs IL with demonstration data as pre-training and then uses RL in addition [4; 17; 18]. It is difficult for policies learned via IL to cope with situations not encountered in demonstration data; however, this method compensates for this shortcoming by combining RL. In addition, pre-training through IL avoids the limitation of manually designing reward functions for RL and significantly reduces the RL training time. Therefore, a strategy learning method that combines IL and RL has become popular and is a powerful alternative to RL. This enables robots to perform long-horizon tasks in challenging simulated kitchen environments [18] and also mimics the agile locomotion skills of animals [4].