# Data-Driven Policy Learning Methods from Biological Behavior: A Systematic Review
## 5 Learning Based on Interpretability and Safety: Causal Policy Learning
### Causal Policy Learning  
The IL and IRL mentioned in Sections 3 and 4, respectively, exhibit excellent performance in behavioral strategy learning methods by learning from the demonstration and learning the reward function of behavior. In these machine learning-based policy-learning methods, although more input data can make the model estimate more accurately, it is difficult to ensure that the estimation result is unbiased [114]. This is because IL and IRL follow the assumption that the observations demonstrated by the expert and those available to the agent match when implementing policy learning. However, in the real world, it is difficult to satisfy this assumption fully; that is, the expert may not observe all covariates. Under such conditions, it is difficult for IL and IRL to perform as desired. A causal inference framework improves the accuracy and interpretability of machine learning. CPL incorporates causal learning frameworks. It aims to learn causal relationships and strategies from the observed data by estimating the causal effects of actions on potential future states and considering potential confounders in the data. The common approach in CPL is to use SCMs to represent the causal relationships between variables. These models allow agents to perform counterfactuals on the variables of interest and estimate their causal effects. An agent can thereby use this causal knowledge to guide the process of adaptive and flexible behavioral generation. The traditional CPL framework for SCM [69; 70; 71; 72; 73] first defines the SCM \(M\) as \(\{U,\,V,\,F,\,P(u)\}\) to capture the causal relationship between the variables of interest. It divides the endogenous variable \(V\) into observed variables \(O\) and latent variables \(L\), where \(O\subseteq V\) and \(L=V\setminus O\). Thus, SCM \(M\) can be implemented to model the unobserved nature of endogenous variables. The expected reward and reward of the expert demonstrated are denoted by \(\mathbb{E}[Y\mid do(\pi)]\) and \(\mathbb{E}[Y]=1\), respectively, where \(\pi\) denotes the policy. The function \(do()\) represents the intervention or action being performed in causal inference. In policy learning tasks, the expected reward \(\mathbb{E}[Y\mid do(\pi)]\) cannot be determined directly from the observed data or DAG because the reward \(Y\) is typically latent. Additionally, there are cases in which confounders or other variables exist. For example, for ordered variables \((X,\,Y)\) in a DAG, exploring the causal effects between \(X\to Y\) can be difficult if there are variables that point to or influence \(X\) that are difficult to measure in the experiment. The backdoor criterion is used to address this problem [70]. It states that the set of variables \(Z\) satisfies the backdoor criterion for ordered variables \((X,\,Y)\) when no element of the set \(Z\) is a successor node of \(X\) and \(Z\) blocks all paths between \(X\) and \(Y\) that towards \(X\). In subsequent refinements, the backdoor criterion applicable to single-stage decision problems is extended to the sequential backdoor criterion by considering the time dependence between variables; that is, \(X\) affects \(Y\) at different points in time and can be viewed as a recursive form of the single-stage backdoor criterion [70]. If \(Z\) satisfies the ordered backdoor criterion for ordered variables \((X,\,Y)\), then the causal effect between \(X\to Y\) can be deduced. \[P(Y\mid do(X))=\sum_{Z}P(Y\mid X,\,Z)P(Z) \tag{17}\]  
Figure 3: Example of a highway driving causal diagram, where \(X\) and \(Y\) represent the action (shaded red) and reward (shaded blue), respectively. Input covariates are shaded in light red (CC BY 4.0 image credit: [69]). Strategy learning is then guided by the collected observed data and estimated causal effects.