# Data-Driven Policy Learning Methods from Biological Behavior: A Systematic Review
## 6 Discussion  
### Algorithmic Property Analysis  
Policy learning enables the generation of expert behavior when the reward function is unknown or difficult to specify. This review examined three approaches based on data-driven policy-learning algorithms. The features of the algorithms reviewed in this paper are summarized in Table 2. Based on the literature review results, combined with their respective recent improvements, we selected relatively popular algorithms among the three groups of approaches: BC, MaxEnt-IRL, DAgger, GAIL, and AIRL. Furthermore, we discuss the approaches based on task requirements, data availability, and computational cost. The robustness in high-dimensional spaces, dataset quality, and dynamic systems is also discussed. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Category** & **Sub-Category** & **Algorithm** & **Task Requirements** & **Data Availability** & **Computational Cost** \\ \hline \multirow{8}{*}{BC} & \multirow{4}{*}{DAgger} & Simple BC & Simple tasks with clear, static goals and easily accessible expert demonstrations & The quality, quantity, and diversity of data sets are important & Low computational cost \\ \cline{3-5}  & & & & \\ \cline{3-5}  & & & & \\ \cline{3-5}  & & & & \\ \cline{3-5}  & & & & \\ \cline{3-5}  & & & & \\ \hline \multirow{8}{*}{IL} & \multirow{4}{*}{AC-GAIL} & \multirow{4}{*}{Tasks that require learning diverse behaviors} & Compared with GAIL, explicit modal labels are required for diverse behavior learning & Similar to GAIL but the computational cost depends on conditional information \\ \cline{3-5}  & & & & \\ \cline{3-5}  & & & & \\ \cline{3-5}  & & & & \\ \cline{3-5}  & & & & \\ \cline{3-5}  & & & & \\ \cline{3-5}  & & & & \\ \cline{3-5}  & & & & \\ \cline{3-5}  & & & & \\ \cline{3-5}  & & & & \\ \cline{3-5}  & & & & \\ \cline{3-5}  & & & & \\ \cline{3-5}  & & & & \\ \cline{3-5}  & & & & \\ \cline{3-5}  & & & & \\ \cline{3-5}  & & & & \\ \cline{3-5}  & & & & \\ \cline{3-5}  & & & & \\ \cline{3-5}  & & & & \\ \hline \hline \multirow{8}{*}{IL} & \multirow{4}{*}{Maximum Margin Method} & \multirow{4}{*}{MLP} & High dimensional tasks where the reward function is required and hard to determine, suitable for complex and dynamic environments & Can learn an alternative policy of the expert policy, the robustness of suboptimal demonstration is poor & High computational cost owing to model-based algorithm and reward function searching \\ \cline{3-5}  & & & & & \\ \cline{3-5}  & & & & \\ \cline{3-5}  & & & & \\ \cline{3-5}  & & & & \\ \cline{3-5}  & & & & \\ \cline{3-5}  & & & & \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Comparison of properties for reviewed policy learning algorithms. Simple BC employs expert presentation datasets for supervised state-action mapping. BC is suitable for tasks where the presentation data are easily accessible and can completely cover the state-action space. Although computationally efficient, it is highly dependent on the quality and diversity of large datasets. Subtle inaccuracies can eventually lead to poor prediction results, known as covariate shifts and causal confusion. Hence, it is limited in dynamic environments and suboptimal dataset performance. When the task focuses more on the dynamics of the environment, IRL methods can recover the reward function from the presentation data that explains the behavior, helping the agent to make predictions in a dynamic system. In contrast, MaxEnt-IRL exploits the maximum entropy principle and learns strategies outperforming suboptimal expert demonstrations. MaxEnt-IRL has a high generalization ability to avoid many problems in BC. However, MaxEnt-IRL inevitably incurs high computational costs because the function search is ill-posed. This limits the computational efficiency in high-dimensional scenarios. Extending this method to large-scale datasets and environments remains challenging for the current IRL research. IRL is fundamentally about learning a cost function to explain expert behavior rather than telling the agent how to act.