# Data-Driven Policy Learning Methods from Biological Behavior: A Systematic Review
## 4 Explaining Behavior via Reward Function: Inverse Reinforcement Learning
### Maximum Margin Method  
The original IRL algorithm assumes a problem consistent with MDP, where a policy can decide an action for the next state using the current state-action pair. The reward function must be estimated from a portion of the basis functions that represent the important features of the states using a linear approximation. Subsequently, we generalize to more unknown states as follows:  
\[R(s)=\alpha_{1}\phi_{1}(s)+\alpha_{2}\phi_{2}(s)+\cdots+\alpha_{n}\phi_{n}(s)= \alpha\phi(s), \tag{5}\]  
where each basis function \(\phi_{i}(s)\) maps the state \(s\) to a scalar value, and the coefficient \(\alpha\) determines the contribution of each basis function to the state. According to Equations (2) and (5), the value function \(v_{\pi}(s_{t})\) can be expressed as  
\[v(\pi)=\mathbb{E}[\sum_{t=0}^{\infty}\gamma^{t}\sum_{i=1}^{n}\alpha_{i}\phi_{i }(s_{t})\mid\pi], \tag{6}\]  
and is further simplified to the following form:  
\[v(\pi)=\sum_{i=1}^{n}\alpha_{i}\mathbb{E}[\sum_{t=0}^{\infty}\gamma^{t}\phi_{ i}(s_{t})\mid\pi]=\sum_{i=1}^{n}\alpha_{i}\mu_{i}(\pi)=\alpha\mu(\pi). \tag{7}\]  
It is worth mentioning that IRL is an ill-posed problem according to its definition. Multiple optimal policies may exist under the same reward function, and multiple reward functions may correspond to the same optimal policy. When the optimal policy is unknown, the original IRL algorithm estimates it using multiple expert demonstrations, also known as apprenticeship learning (AL) [100]:  
\[\hat{\mu}(\pi^{E})=\frac{1}{n}\sum_{i=1}^{n}\mu(\pi^{i}). \tag{8}\]  
For the reward function to have only one corresponding optimal policy, AL solves the ill-posed problem using the maximum margin method, that is, maximizing the difference between the first and second optimal policies. The process of optimizing the reward function can be represented as follows:  
\[\max\{t_{i}=\min_{j\in\{0,1,\ldots,i-1\}}\alpha(\mu(\pi^{E})-\mu(\pi^{j}))\}, \tag{9}\]  
where \(t_{i}\) denotes the minimum value of the transition difference in the expected rewards among all previous strategies up to step \(i-1\). By maximizing this term, a strategy significantly better than any previous strategy is identified to make it as close as possible to the expert strategy, as shown in Figure 2A. The optimization process of this algorithm is randomized, and thus, may not be sufficiently accurate in the face of expert behavior with deterministic behavior. Rather than returning to an optimal policy, AL uses the maximum margin method to select a randomly returned alternative policy with performance approximately similar to expert behavior. Thus, maximum margin planning (MMP) provides a learning method that can directly reproduce expert behavior [101]. MMP avoids the ill-posed problem by introducing a loss function \(L(\pi)\). The difference from AL is the margins in the MMP scale with the loss function. \(\max_{\pi^{\prime}}\) denotes the maximum value of all strategies, except \(\pi\). In other words, \(L(\pi)\) penalizes strategies with a small margin between the expected value and the expected value of the best strategy, thus selecting more robust strategies with larger margins:  
\[\max_{\pi}\{v_{\pi}(s_{0})-\min_{s^{\prime}}v_{\pi}(s^{\prime})\}, \tag{10}\]  
\[L(\pi)=\max_{\pi^{\prime}}[v_{\pi}(s_{0})-v_{\pi^{\prime}}(s_{0})]. \tag{11}\]  
In the optimization process, MMP employs the \(q_{\pi}(s,\,a)\) function to estimate the expected reward for each action in each state. It is continuously updated iteratively, such that the action with the highest reward is selected to obtain an optimal policy. In subsequent improvements, LEARCH was introduced and extended the learning of the reward function to nonlinear using an exponential function gradient descent method for optimization [102], as shown in Figure 2B. For some complex tasks (e.g., autonomous helicopter aerobatic flight, which needs to consider other factors such as dynamics in addition to the target trajectory), the sample tra  
Figure 2: Direct illustrations of the maximum margin method.