## 4 Methodology
### SHapley Additive exPlanations  
We might want to know why the vehicle accelerates or decelerates under the driving of the PPO algorithm. In other words, the agent observes the state that consists of some elements. Some of the elements may spur the vehicle to accelerate, while others may slow down the vehicle. Meanwhile, the attribution of the features (i.e., the element in the state vector) is different for the agent to make decisions. Considering safety and reliability, we should build an explanation module to describe the behavior of the agent.  
It is known that feature attribution is a common approach to analyzing a trained machine learning model, whereas the RL is a branch of machine learning. Conventionally, the machine learning community takes tree-based models as one kind of the interpretable methods [48]. Nevertheless, this paper deploys a DRL model, which can only be explained in a post-hoc manner. We assume that \(F\) is a machine learning model that maps some input \(x=(x_{1},\ldots,x_{n})\in R^{n}\) to the target values. The feature attribution of the prediction at \(x\) relative to a baseline input \(x_{b}\) can be denoted with a vector \(A_{F}(x,x_{b})=(fa_{1},fa_{2},\ldots fa_{n})\in R^{n}\) where \(fa_{i}\) represents the contribution of \(x_{i}\) to the prediction \(F(x_{i})\). In this case, the Shapley value becomes an effective method to distribute the total gains of a cooperative game to a coalition of cooperating players [49]. More specifically, taking a coalitional game with \(n\) players and a function \(F\) that maps the subset of players to the real numbers \(f:2^{N}\to R\) as an example, the attribution amount of player \(i\) to the game is:  
\[A_{f}(i)=\sum_{C\subseteq N\setminus\{i\}}\frac{|C|!(n-|C|-1)!}{n!}(f(C \cup\{i\})-f(C)), \tag{26}\]  
where \(f(C)\) is the worth of coalition \(C\), illustrating the total expected sum of payoffs the members of \(C\) can derive from the cooperation. The gist of the equation is: the difference between value function with and without player \(i\) is computed for each coalition \(C\) without player \(i\), so the contribution of player \(i\) is equal to the weighted difference \(\frac{|C|!(n-|C|-1)!}{n!}\). For the RL-based eco-driving problem, we formulate a game for the control state at each timestep. The \"total gains\" denote the acceleration of the vehicle, while the \"players\" is the set of the state in the MDP model. The Shapely-value-based approach tends to approximate Shapley values by examining the effect of removing an element under all possible combinations of the presence or absence of the other elements in the state vector.  
However, computing the Shapely value for every feature is still challenging. On the other hand, LIME (Local Interpretable Model-Agnostic Explanations) [50] is proposed to be an additive feature attribution method that learns an explainable model like a decision tree or linear regression model \(w(x^{\prime})\) to approximate the target black-box model locally, where \(z^{\prime}\) is a simplified binary input vector. Then the simplified features are transformed into the original algebraic space so that the corresponding target value is computed by \(w(z^{\prime})=f(h_{x}(z^{\prime}))\), where \(h_{x}\) is a function that maps the simplified input to the original input. The Kernel SHAP method draws lessons from LIME and computes the weight for each element in \(z^{\prime}\). A SHAP value estimates the change in predicted output, from the base value \(E[f(x)]\) to \(f(x)\), attribute to every single feature \(i\). More precisely, the SHAP value can be \(E\big{[}f(z^{\prime})\big{|}z^{\prime}_{1}=x_{1}\big{]}-E[f(z^{\prime})]\) for feature with \(i=1\).