# Data-Driven Policy Learning Methods from Biological Behavior: A Systematic Review
## 3 Strategy Learning from Demonstration: Imitation Learning  
By maximizing the cumulative rewards from the environment, RL learns strategies to achieve specific goals [81]. It has been extensively employed to solve various control problems involving the interactions between agents and environments. However, experts  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Category** & **Sub-Category** & **References** \\ \hline \multirow{3}{*}{Imitation Learning (IL)} & Behavior Cloning (BC) & [4; 8; 9; 10; 20] \\  & Generative Adversarial Imitation Learning (GAIL) & [21; 22] \\  & Improved GAIL Algorithms & [2; 23; 24; 25; 26; 27; 28; 29; 30; 31; 32; 33; 34; 35; 36; 37] \\ \hline \multirow{3}{*}{Inverse Reinforcement Learning (IRL)} & Maximum Margin Method & [5; 38] \\  & Entropy Optimization Method & [3; 50; 39; 3; 51; 3; 52; 3; 53; 54; 55; 56; 57; 58; 59; 60; 61; 62; 63] \\ \cline{1-1}  & Bayesian Optimization Method & [51; 62; 63] \\ \cline{1-1}  & Adversarial Method & [64; 65; 66; 67; 68] \\ \hline \multirow{3}{*}{Causal Policy Learning (CPL)} & Traditional Structural Causal Model (SCM) Modeling & [69; 70; 71; 72; 73] \\ \cline{1-1}  & Other SCM Modeling Methods & [74; 75; 76] \\ \hline \hline \end{tabular}
\end{table}
Table 1: Overall framework of this review.  
have designed reward functions used for strategy learning. Although they are feasible for handling relatively well-defined tasks (e.g., Go games [82]); in more complicated environments, it can be extremely difficult to artificially design a reasonable reward function for animal or human behavior. However, IL can avoid this problem by directly learning from the demonstration, that is, by imitating the object behavior in the given tasks.  
IL takes the observed state and action sequences as decision data and trains the agent to learn the mapping relationships between each set of decision data. Thus, we obtain an optimal policy model to perform the task in the demonstration. IL does not rely on designing explicit reward functions; it simplifies the process of teaching agents to learn by providing demonstrations to them. Thus, it has several advantages in achieving complex tasks with minimal expert knowledge [83]. Integrated with high-speed sensors that enable instantaneous data collection, IL has great potential for applications in bio-inspired robots, autonomous driving, and VR games.  
In this section, we review policy-learning methods using IL. These IL methods include behavior cloning (BC), generative adversarial imitation learning (GAIL), and improved GAIL techniques. BC learns the mapping of behaviors from states to actions through a supervised learning approach using labeled behavioral demonstration data as the training set, which allows IL to be most simple. Unlike BC, GAIL trains policies and reward functions by generating adversaries that can learn policies directly from demonstrations and has stronger representational power in IL involving environmental interactions. However, the improved GAIL methods improve learning and increase the efficiency of sample utilization based on the original GAIL algorithm, making it adaptable to real-world problems.