# Optimizing Reinforcement Learning Using a Generative Action-Translator Transformer
## 3 Action-Translator Transformer
### Model Formulation
#### 3.2.4 Training and Inference  
The whole training process of ATT we can summarize as Algorithm 1. ```
1:Input: Data set \(D=\{(s_{t},a_{t},r_{t})\}\)
2:\(\hat{R}_{t}\gets r_{t+1}+r_{t+1}+r_{t+2}+\cdots+r_{end}\)
3: Initialize weight \(\theta\), training iterations \(I\), step size \(\eta\), batch size \(B\), the state embedding matrix \(W_{s}\in\mathbb{R}^{(d_{k}+d_{s})\times B}\), the action embedding matrix \(W_{a}\in\mathbb{R}^{d_{k}\times B}\)
4:for\(i=1\)to\(I\)do
5: Stack \((\hat{R},s)\)
6: Embedding: \((\hat{R},s)_{emb}\gets W_{s}\times(\hat{R},s)\)
7: Embedding: \(a_{emb}\gets W_{a}\times a\)
8: Positional Encoding
9:\(Encoders\leftarrow(\hat{R},s)_{emb}+P(t)\)
10:\(Decoders\gets a_{emb}+P(t)\)
11: Masked
12: Train
13: Output \(a^{\prime}\)
14:\(loss\gets L(a^{\prime},a)\)
15:\(\theta\leftarrow\theta+\eta\
abla L(\theta)\)
16:endfor
17:return\(\theta\)
```  
**Algorithm 1** ATT training, optimize network parameters based on offline trajectories  
At the beginning of training, for the offline dataset samples, firstly, \(r\) in the sample data are calculated as \(\hat{R}\), and the samples are processed to a specific length. After encoding and introducing the position information, they are processed by Mask and then entered into the Transformer for training using the Teacher-forcing method. The prediction of the action \(a^{\prime}\) is then compared with the original input action sequence \(a\) to calculate the loss for optimizing the model. The complete pre-training process of ATT is outlined above. The loss function used here is given by Equation (8). The Inference process of ATT, we can summarize as Algorithm 2. The Inference of the ATT model is mainly derived using self-regression, where for each prediction, the model's output is used as input for the next prediction. By initializing the model with the necessary information, we continuously utilize the model to predict actions and generate trajectories. Eventually, we obtain the target trajectory based on our initial settings. During this process, the inputs to the model include the cumulative target rewards, where we set the maximum possible reward value \(\hat{R}\) (as the ultimate goal of reinforcement learning is to achieve the maximum reward in the current environment) based on prior knowledge or environmental information. The initial state of the environment \(s\) is obtaineddirectly by initializing the environment. We initialize the action \(a\) on the decoder side and input all this information into the encoder and decoder sides according to the input mode during the training process after padding operation. Specifically, \(\hat{R}\) and \(s\) are input into the encoder side as global observable information and then passed to the decoder side after going through the Mask and Encoders. ```
1:\(Initializeenv()\)
2:Initialize \(\hat{R},s,a,step=0,length,G=0\)
3:Padding
4:while\(s\) not equal to end or step<lengthdo
5:Positional Encoding
6:\(Encoders\leftarrow(\hat{R},s)+P(t)\)
7:\(Decoders\gets a+P(t)\)
8:Masked
9:Output \(a_{next}\)
10:\(r_{next},s_{next}\leftarrow env(a_{next})\)
11:\(\hat{R}\leftarrow\hat{R}-r_{next}\)
12:\(G\gets G+r_{next}\)
13:\(step\gets step+1\)
14:endwhile
15:return\(G\)
```  
**Algorithm 2** ATT inference, generate full trajectories based on target rewards  
The next action is predicted at the decoder side based on the input historical action information \(a\) and the global observable information from the encoder side. After the model outputs the next action \(a_{\text{next}}\), we execute \(a_{\text{next}}\) in the current environment. At this point, our environment changes to \(s_{\text{next}}\) due to the agent's interaction behavior, and we obtain the instant reward \(r_{\text{next}}\). Based on this instant reward, we update the expected reward sum \(\hat{R}_{t}\) and then add the newly obtained \(a_{\text{next}},\hat{R}_{\text{next}},s_{\text{next}}\) to the existing sequence as the observable data for predicting the next action.