# Interpretable Reinforcement Learning for Sequential Strategy Prediction in Language-Based Games
## 3 Methods
### Model Structure  
Enhanced-DDPG is a model-free reinforcement learning method for continuous-action spaces [7; 8; 9; 10; 11; 12]. By combining deterministic policy gradient and deep learning techniques, Enhanced-DDPG performs well in solving continuous-control problems. The following is the algorithm flow of our Enhanced-DDPG:  
This approach aligns with reinforcement learning principles where action spaces are dynamically constrained by environmental states. Firstly, the parameters of the critic network and actor network are randomly initialized. Specifically, the parameters \(\varphi\) of the critic network \(Q(S,A;\varphi)\) and the parameters \(\theta\) of the actor network \(\pi(S;\theta)\) are initialized with random values. Furthermore, the parameters of the target critic network \(\varphi_{t}\) are set to be equal to \(\varphi\). Similarly, the parameters of the target actor network \(\theta_{t}\) are initialized to match \(\theta\). Additionally, Enhanced-DDPG creates a cyclic experience replay buffer to store and reuse past experiences, which improves the efficiency of learning by breaking the temporal correlations in the data. During the training phase, the Enhanced-DDPG algorithm follows an iterative process. For the current observed state \(S\), an action \(A\) is selected using the actor network. To encourage exploration, random noise from a predefined noise model is added to the action. The selected action \(A\) is then executed in the environment, which returns a corresponding reward \(R_{i}\) and the next state \(S^{\prime}\). The experience tuple \((S,\ A,\ R,\ S^{\prime})\) is stored in the replay buffer for later use in batch updates. To update the network parameters, the algorithm randomly samples a mini-batch of experiences \((S,\ A,\ R,\ S^{\prime})\) from the replay buffer. For each sampled experience, the algorithm computes a target value \(y_{i}\). If \(S^{\prime}_{i}\) is a terminal state, then \(y_{i}\) is simply set to \(R_{i}\). Otherwise, the next action is computed using the target actor network, and its value is evaluated using the target critic network, leading to the following target computation as Equation (3), where \(\gamma\) is the discount factor:  
\[y_{i}=R_{i}+\gamma Q_{t}\big{(}S^{\prime}_{i},\ \pi_{t}\big{(}S^{\prime}_{i}; \theta_{t}\big{)};\varphi_{t}\big{)}. \tag{3}\]  
The parameter is updated by minimizing the critic network of all sampling experiences with loss function L, adopting the method of gradient descent. Meanwhile, the parameters of the actor network are updated based on the sampled policy gradient to maximize the expected discounted cumulative long-term reward. The process involves calculating the gradient of the critic's network output to the action, and the gradient of the actor's network output to the parameter. The key step of Enhanced-DDPG algorithm is to update the target network, which plays an important role in improving the stability of learning. Enhanced-DDPG supports three target update methods, including smooth update, periodic update, and periodic smooth update. In smooth update, the target parameters are updated at each time step using a smoothing factor \(\tau\). Periodic updates involve the target parameters being updated after a fixed interval without smoothing (\(\tau=1\)). The periodic smooth update combines these two methods, updating after a fixed interval using a smoothing factor. Through this iterative training process, Enhanced-DDPG algorithm continuously optimizes the actor and critic network to achieve effective policy learning in the continuous-action space. Its advantage lies in its continuous control that can handle the high dimension problem, and by introducing the target network, effectively solve the stability problem in the process of learning. The Enhanced-DDPG algorithm and its variants continue to play an important role with the deepening of the application of reinforcement learning in automatic decision-making, autonomous driving, and other fields.