# Data-Driven Policy Learning Methods from Biological Behavior: A Systematic Review
## 4 Explaining Behavior via Reward Function: Inverse Reinforcement Learning
### Entropy Optimization Method  
Although MMP has enabled the replication of expert trajectories, this maximum margin method-based approach suffers from labeling bias in practical applications. The estimated reward function will be biased towards actions or trajectories consistent with the expert's demonstrated behavior while ignoring other potentially desirable alternatives [105]. The over-preference for expert behavior when learning reward functions from demonstrations makes diverse strategy exploration challenging. Hence, a series of IRL methods employ entropy optimization. The policy with the highest reward derived by MMP is not necessarily the policy with the highest probability among the entropy optimization methods. To reduce this label bias, the maximum entropy IRL (MaxEnt-IRL) [106] learns the reward function using a probabilistic method to solve the ill-posed problem of IRL. MaxEnt-IRL uses the maximum entropy principle [107] to solve this uncertainty, which allows for estimating the minimum deviation from the given information when only partial information is available. The probability of an observed expert trajectory is weighted by the estimated reward:  
\[Pr(\tau\mid w)=\frac{1}{Z}e^{\alpha\mathbb{E}(\tau,w)}, \tag{12}\]  
where \(\tau\) represents the trajectory, \(w\) is the weight vector, and \(Z\) is the normalizing constant. \(\alpha\) indicates the confidence level at which the expert selects an action with a high reward value. Thus, the optimal weight vector \(w\) is derived by maximizing the likelihood of the observed trajectory using the maximum-entropy approach:  
\[w^{*}=arg\max_{w}\sum_{\tau\in D}\log Pr(\tau\mid w), \tag{13}\]  
where \(D\) represents the demonstration. MaxEnt-IRL uses the maximum entropy approach to select the reward function based on the premise that the calculated strategy reward is comparable to the expert strategy reward. Despite the superiority of the probabilistic approach, MaxEnt-IRL remains limited to the representational capacity of linear reward functions. On this basis, the maximum entropy deep IRL (MEDIRL) [39] extends the representation of the reward function to nonlinearity. MEDIRL employs the same basic algorithm as MaxEnt-IRL; however, it represents the reward function through a flexible neural network, and the features in the reward function do not need to be set manually. However, MaxEnt-IRL does not adequately consider stochastic transformation models. Thus, the maximum causal entropy IRL (MCE-IRL) extends the maximum entropy principle to side information [40; 41; 108]. It focuses on inferring an agent's preferences from observed behaviors and further understanding the causality between actions and outcomes. MCE-IRL allows stochastic transformation models. Hence, it considers the causality between actions and outcomes while estimating the reward function. IRL algorithms based on the maximum entropy framework are model-based. In contrast, relative entropy IRL (RE-IRL) is a model-free algorithm [5; 42; 109]. It overcomes the computational cost problem by minimizing the KL scatter between the learned and real trajectories to determine the optimal policy. Therefore, it is more suitable for IRL tasks that use large datasets. These probability-based maximum entropy methods enable experts to demonstrate strategies as suboptimal and, therefore, learn potentially optimal models that are more flexible in practical applications without the concern of less-than-perfect demonstration samples. For autonomous driving problems, the MaxEnt-IRL can learn the reward function [43; 44] from driving data, including the vehicle speed, distance from the front vehicle, and relative speed. It has also been used to learn pedestrian navigation [3; 45], animal behavior [46; 47; 48], and human preference [49], which provided new inspirations for HCI developments and the understanding of behavior mechanisms. On this basis, MEDIRL can handle large datasets [50]. Moreover, as a model-free algorithm, RE-IRL is capable of learning different strategy styles [5].