# Optimizing Reinforcement Learning Using a Generative Action-Translator Transformer
## 2 Background
### Reinforcement Learning Algorithms  
Currently, mainstream reinforcement learning algorithms train agents based on modeling the environment with MDP. When agents need to make actions interacting with the environment, they either follow the current policy or use the current value function. In deep reinforcement learning, policies and value functions are often parameterized by variables in deep neural networks. Subsequently, gradient-based methods are employed for optimization. Loss is calculated using a loss function, followed by backpropagation to update network parameters until the value or policy function meets the training objectives. Based on the mentioned training approaches, mainstream reinforcement learning methods can be categorized into three major classes: Value-Based, Policy-Based, and Actor-Critic algorithms that simultaneously rely on both policy and value functions. **Value-Based:** This method requires us to define a value function. For tasks with a small state space and action space, we can use a tabular form to represent the value function, such as Q-Learning. It optimizes a state-action value table where each data entry represents the immediate reward obtained by taking action \(a\) in state \(s\), denoted as \(Q(s,a)\). The core update formula is as follows:  
\[Q(s_{t},a_{t})\gets Q(s_{t},a_{t})+\alpha[r_{t}+\gamma\max_{a_{t+1}}Q(s_{t +1},a_{t+1})-Q(s_{t},a_{t})] \tag{2}\]  
Here, \(\alpha\) represents the update step size, and the value function \(Q(s_{t},a_{t})\) is continuously updated through iterations until convergence. Based on the action-value function, the optimal action in a certain state can be selected. In high-dimensional and complex environments, using a table alone is insufficient to represent all states and actions. Therefore, we use neural networks to fit the value function, leading to the birth of the classic deep reinforcement learning algorithm, Deep Q-Network (DQN) [14], which uses parameters \(\theta\) to approximate the value function. Subsequent Value-Based algorithms mainly focus on further optimizing the direction of reducing errors and improving sample learning efficiency. **Policy-Based:** Unlike value-based methods, policy-based methods directly learn parameterized policies \(\pi_{\theta}\). This allows us to avoid the need to solve the value function for each action in the space. Therefore, this method is suitable for solving problems with high-dimensional or continuous action spaces. It intuitively improves the performance of the policy \(\pi_{\theta}\) in the parameter space through gradient ascent methods and policy iteration updates, achieving the maximization of cumulative rewards. For example, the Policy Gradient (PG) [15] algorithm has the objective function to maximize the expected return by adjusting \(\theta\), represented as follows:  
\[J(\pi_{\theta})=E_{\pi\sim\tau}[R(\tau)] \tag{3}\]  
In the above equation, \(\tau\) represents a complete trajectory from the beginning to the end, \(\pi\) represents the current policy of the agent, and \(\pi\sim\tau\) represents the action trajectory of the agent sampled under this policy. For this maximization problem, we use the gradient ascent algorithm to find its maximum value, and the optimization formula for parameter \(\theta\) is as follows:  
\[\theta^{*}=\theta+\alpha\
abla J(\pi_{\theta}) \tag{4}\]  
The essence of policy-based algorithms lies in determining the gradient of the final return function \(J(\pi_{\theta})\) with respect to \(\theta\), commonly referred to as the policy gradient. This forms the basis for the optimization of various mainstream policy-based algorithms like PPO, TRPO, etc. **Actor-Critic:** This approach, which integrates both policy and value, concurrently learns an actor function and a critic function. The actor function serves as the agent's policy function \(\pi(\cdot|s)\), determining the action the agent will take in the current state. Meanwhile, the critic function functions as the state value function, providing an assessment of the action taken by the agent [16].