# Optimizing Reinforcement Learning Using a Generative Action-Translator Transformer
## 3 Action-Translator Transformer
### Model Formulation
#### 3.2.3 Model Details  
**Target Reward:** As we employ offline reinforcement learning data, traditional reinforcement learning sample data are stored in the format of \((s_{t},a_{t},r_{t})\). However, the Transformer model is designed to train sequence data containing contextual information. Therefore, it is necessary to establish a connection between the data and, simultaneously, segment the data into appropriate lengths for the Transformer network to facilitate learning. The representation of offline reinforcement learning trajectory data are shown in the equation below:  
\[\tau=(\dots,s_{t-1},a_{t-1},r_{t-1},s_{t},a_{t},r_{t},s_{t+1},a_{t+1},r_{t+1},\dots) \tag{10}\]  
where \(s\) represents the state information at a certain moment, \(a\) represents the action information at a certain moment, and \(r\) represents the instant reward information at a certain moment. Such trajectory data can only represent the relevant information at each moment, and it cannot be used as the training data for the Transformer model to learn contextual information. Therefore, we need to process the data. We retain the original forms of \(s\) and \(a\) in the trajectory data, and process \(r\). We represent \(r\) in a new form, denoted as \(\hat{R}\), calculated as follows:  
\[\hat{R_{t}}=r_{t}+r_{t+1}+r_{t+2}+\dots+r_{end} \tag{11}\]  
The information represented by \(\hat{R_{t}}\) is the total reward value available from the current moment to the end of the trajectory. This transformation establishes a link between the data at different moments, and the trajectory history data are processed into the following form:  
\[\hat{\tau}=(\hat{R_{0}},s_{0},a_{0},\hat{R_{1}},s_{1},a_{1},\dots,\hat{R_{t}}, s_{t},a_{t}) \tag{12}\]Although the Transformer can handle long sequences of text, it is also sensitive to sequence length limitations. Therefore, we need to preprocess the trajectory data into segments of length \(K\) that the model can accept. In cases where the original trajectory is shorter than \(K\), we fill the remaining positions with zeros. **Embedding:** After processing, the data can meet the training requirements of the Transformer. Following the flow of a text translation task, we need to encode the processed data. We use a fully connected layer to encode the state \(s\), action \(a\), and target reward \(\hat{R}\). Utilizing the dual-end structure of the Transformer, we concatenate \(\hat{R}\) and \(s\) to form a sequence on the encoder side, encoding it as observable environmental information during action translation. On the decoder side, the sequence composed of \(a\) is encoded as the predicted output control of our model. We input the real action information as the target prediction into the decoder side, because we employ the Teacher-Forcing training method, a parallel training method of Transformer, which will be explained in detail later. Here, we only introduce the input form. Both the input-side coding layer and the output-side coding layer use a fully connected layer structure, as shown in Figure 6.  
**Positional Encoding:** During the training process of the Transformer network model, it is essential to introduce positional information to enable the model to perceive the relative positions of elements in the sequence. As our reinforcement learning trajectory data already includes temporal information, i.e., it corresponds to a specific time step, we can directly use the time step as our positional information. Considering a complete trajectory as a timeline, sequentially numbered starting from zero, we input the position encoding function to generate the positional encoding information. Our position encoding method involves creating a coding matrix that maps a certain range of numbers to unique vectors, which serve as the encoded positional information.