# Interpretable Reinforcement Learning for Sequential Strategy Prediction in Language-Based Games
## 3 Methods
### Model Structure
For this study, the model structure is shown in Figure 1 and the model algorithm process is shown in Algorithm 1. The following is a detailed description of the main steps in the algorithm table:  
```
Require: Observation space \(S\), action space \(A\) Actor network \(\pi(S;\theta)\), target actor network \(\pi_{t}(S;\theta_{t})\) Critic network \(Q(S,A;\varphi)\), target critic network \(Q_{t}(S,A;\varphi_{t})\) Experience buffer size \(M\), discount factor \(\gamma\), noise model \(\mathcal{N}\) Target update method(smoothing factor \(\tau\) or update frequency) Ensure: Trained actor network\(\pi(S;\theta)\) 1: Initialize critic \(Q(S,A;\varphi)\) with random parameters \(\varphi\) 2: \(\varphi_{t}\ \leftarrow\ \varphi\) 3: Initialize actor \(\pi(S;\theta)\) with random parameters \(\theta\) 4: \(\theta_{t}\ \leftarrow\ \theta\) 5: Initialize experience buffer with capacity \(M\) 6: for each training time step do 7: Observe current state \(S\) 8: \(A=\pi(S;\theta)+\mathcal{N}\) 9: Execute action \(A\), observe reward RandnextstateS\({}^{\prime}\) 10: Store experience \((S,A,R,S^{\prime})\) in the buffer 11: Sample a mini - batch of \(M\) experiences \((S_{i},A_{i},R_{i},S_{i}^{\prime})\) 12: for each experience in the mini - batch do 13: if \(S_{i}^{\prime}\) is a terminal state then 14: \(y_{i}=R_{i}\) 15: else 16: \(y_{i}=R_{i}+\gamma Q_{t}\big{(}S_{i}^{\prime},\pi\big{(}S_{i}^{\prime};\theta_ {t}\big{)};\varphi_{t}\big{)}\) 17: end if 18: end for 19: Update critic parameters \(\varphi\) by minimizing the loss \(L\) across all ampled experiences 20: Compute gradients \(\
abla_{A}Q(S_{i},A_{i};\varphi)\) and \(\
abla_{\theta}\pi(S_{i};\theta)\) 21: Update actor parameters \(\theta\) via sampled policy gradient 22: if smoothing then 23: \(\theta_{t}\ \leftarrow\ \tau\theta+(1-\tau)\theta_{t}\) 24: \(\varphi_{t}\ \leftarrow\ \tau\varphi+(1-\tau)\varphi_{t}\) 25: elseif periodic then 26: Every \(k\) steps: \(\theta_{t}\ \leftarrow\ \theta\), \(\varphi_{t}\ \leftarrow\ \varphi\) 27: end if 28: end for
```  
**Algorithm 1** The algorithm of the proposed Enhanced-DDPG. First, the parameters of two neural networks (i.e., the parameters \(\omega\) of the critic network \(Q_{\omega}(s,a)\) and the parameters \(\theta\) of the actor network \(\mu_{\theta}(s)\)) are randomly initialized to speed up convergence. Next, to maintain consistent targets during training, we create target networks \(Q_{\omega^{\prime}}\) and \(\mu_{\theta^{\prime}}\) with the same architecture as the main networks and initialize them by copying the current parameters from the main networks. Then, we construct an experience replay buffer \(D\) with capacity \(M\) to store the agent's experiences. When the buffer reaches its limit, it removes the oldest samples using a FIFO strategy to maintain diversity and timeliness. In each training step, the actor network outputs an action \(a_{t}\) based on the current state \(s_{t}\), which is then perturbed by noise from a predefined model to produce \(a_{t}^{explore}\), encouraging sufficient exploration. After executing \(a_{t}^{explore}\), the environment returns the immediate reward \(r_{t}\) and next state \(s_{t+1}\). The experience \(\left(s_{t},a_{t}^{explore},r_{t},s_{t+1}\right)\) is stored in the replay buffer for future training. During the update phase, a mini-batch of samples is randomly drawn from the buffer. The target value as Equation (4) is computed using the target networks, where \(\gamma\) is the discount factor:  
\[y_{i}=r_{i}+\gamma\cdot Q_{\omega^{\prime}}(s_{i+1},\mu_{\theta^{\prime}}(s_{ i+1})). \tag{4}\]  
The critic network is updated by minimizing the mean squared error loss L, with actor parameters fixed and only critic parameters \(\omega\) updated via gradient descent to make estimated Q-values closer to targets. After update the critic, the actor parameters \(\theta\) are updated using policy gradients. The gradient is estimated by the product of \(\
abla_{a}Q_{\omega}(s,a)|_{a=p_{\theta}(s)}\) and \(\
abla_{\theta}\mu_{\theta}(s)\), with adjustments via backpropagation.