# Data-Driven Policy Learning Methods from Biological Behavior: A Systematic Review
## 6 Discussion
### Existing Challenges and Development Trends  
In general, we believe that the current trend is to combine different techniques to bridge the barriers between algorithms. The latest IL approaches focus on achieving generalization capabilities across different domains with meta-learning. IRL techniques focus on combining algorithms with adversarial frameworks. Causal structure invariance or invariant causal features can help improve the generalization ability of CPL across various tasks by meta-learning or domain transfer. In addition, we analyze the challenges, their corresponding solutions, and future work methods in existing policy learning approaches from demonstration samples, diversification strategies, and computational costs.  
* Data availability: Many studies face difficulties in accessing demonstration data when using data-driven policy learning, including obtaining large amounts of high-quality demonstration data, and the accessibility or interactivity of demonstration experts. Therefore, it is important to be able to take advantage of datasets to the maximum, i.e., learning through a limited amount of non-perfect demonstration data. Combining generative adversarial methods [64], self-supervision, meta-learning, transfer learning, or domain adaptation [119] is a new trend in the development of policy learning methods to solve the problems of difficulty in collecting expert data and reducing the dependence on large datasets.
* Learning diverse behaviors: Limited by algorithms, many methods currently can only learn a single behavioral strategy from expert demonstrations. Learning multimodal behaviors from data is still a challenge in current research. As mentioned in Section 3. C, some improved GAIL methods can effectively learn multimodal behavioral strategies. Besides, other algorithms are also considering learning diverse behaviors by combining VAE (e.g., BC with VAE [120; 121; 122]) or dividing multiple subtasks (i.e., multi-task learning) [123; 124].
* High computational cost: Although the development of GPUs has eased the pressure of high computational cost, for most IRLs developed in model-based environments, DAgger involving multiple iterations, and certain GAIL algorithms using models, the high computational costs might limit their application in large-scale datasets. There have been proposed approaches that consider improving function search methods [64] or using model-free algorithms [23] to improve computational efficiency. Further computational cost reduction is also a priority that needs to be addressed in future work.
* Transition to real-world applications: Most algorithms are based on the following ideal assumptions: (1) the agent can access complete information about the environment; (2) the information observed by the expert and the agent is consistent. These are difficult to realize in practice. Simple BC can apply low-dimensional tasks to reality, while most policy learning methods are still tested in simulated environments. Reducing the gap between the simulated and real environments requires the generalization ability of the algorithm. Moreover, tasks involving HCI place more importance on algorithmic security. Therefore, using partially observable MDPs (POMDPs) [32; 119], causal modeling of variables [69], and sim-to-real transfer techniques [125] will be key to extending policy learning to real-world problem solutions.