# Interpretable Reinforcement Learning for Sequential Strategy Prediction in Language-Based Games
## 3 Methods
### Decision Logic and Reward Function  
Based on the Enhanced-DDPG framework, the model estimates the state-action value \(Q(s,a)\) using the critic network and selects optimal actions through an \(\epsilon\)-greedy strategy to balance exploration and exploitation (with \(\epsilon\) initial set to 0.3 and gradually decaying to 0.05). The model selects the action that maximizes \(Q(s,a)\) within the valid action subset \(A_{valid}\), as defined in Equation (5):  
\[a^{*}=\underset{a\in A_{valid}}{\underset{a\in A_{valid}}{\max}}Q(s,a). \tag{5}\]  
The model prioritizes experience replay based on the absolute values of rewards, accelerating convergence by focusing on critical patterns (e.g., high-reward or high-penalty samples).  
Additionally, SHAP values are employed to quantify the contribution of each state feature to the model's prediction, enabling the distribution of the global reward across individual feature dimensions. A novel reward function is proposed, as shown in Equation (7). During actor network updates, SHAP values constrain the policy gradient direction, ensuring that critical features influence action selection as intended.  
The original reward \(r(s_{t},a_{t})\) represents the traditional immediate reward returned by the environment after executing an action in state \(s_{t}\). Then, \(r(s_{t},a_{t})\) incorporates both positive rewards and negative penalties, where each newly revealed correct letter yields a reward of one point, correctly guessing the target word yields ten points, guessing a word not in the dictionary incurs a two-point penalty, reusing a previously attempted word incurs a one-point penalty, and making a guess that provides no new clues incurs a 0.5-point penalty.  
SHAP values quantify feature importance by calculating the marginal contribution of each feature, following the Shapley value concept from game theory. Specifically, for a target sample \(x_{target}\), the baseline prediction value \(E[f(X)]\) is calculated from the background dataset. Then, for all feature subsets \(S\), perturbed inputs are generated. The SHAP value for feature \(i\) is obtained by comparing the prediction differences when including or excluding \(i\), followed by a weighted summation, as shown in Equation (6):  
\[\phi_{i}=\sum_{S\subseteq X\smallsetminus\{i\}}\frac{(|S|)!(n-|S|-1)!}{n!}[f(S \cup\{i\})-f(S)], \tag{6}\]  
where \(\phi_{i}\) represents the SHAP value of feature \(i\), \(S\) denotes feature subsets excluding \(i\), \(n\) is the total number of features, and \(f(S)\) is the model prediction based on subset \(S\).  
The proposed reward function includes a SHAP compensation term \(\sum_{i}SHAP_{i}(s_{t})\), which quantifies the contribution of each feature to the model's prediction. An adjustment parameter \(\lambda\) balances the SHAP compensation term with the original reward to achieve interpretability objectives, as shown in Equation (7):  
\[R_{i}=r(s_{t},\;a_{t})+\lambda\cdot\sum_{i}SHAP_{i}(s_{t}). \tag{7}\]  
Finally, soft updates are applied to blend the main network parameters into the target networks, ensuring smooth target tracking. This training process repeats until the policy converges to approximately optimal actions.