# Boosting Deep Reinforcement Learning Agents with Generative Data Augmentation
## 5 Experiments  
### Setup and Hyperparameter Tuning  
The proposed algorithm is implemented in the Python programming language. Specifically, the TensorFlow [49] (v2.11) library is employed for the development and training of the models and the Gym [50] environment is used for the RL interaction process. The data augmentation and the agent's training are treated as independent tasks and are then combined to form the complete agent. The experiments are carried out on three heterogeneous games of the Gym's Atari 2600 environment:  
* **Boxing:** The agent controls a boxer in a fighting ring and is rewarded for hitting its opponent while a negative reward is obtained for getting hit. There are 18 possible actions in this game. * **Pong:** The classic pong game, where the agent controls a paddle and tries to deflect the ball away from its own goal and send it into its opponents goal. Points are scored accordingly every time the ball ends up in one of the goals. In this game, the action space size is six. * **Riverraid:** the agent controls a jet over a river, aiming to avoid or destroy enemy objects. The action space size is 18. Only positive rewards are available in this game. The states fed to the augmentation and agent networks are subjected to several preprocessing steps. Initially, every image is downsampled to \(84\times 84\) pixels and converted to grayscale. Frame skipping is also performed in a way that only every fourth frame is taken into account, as the difference between two actually consecutive frames is very slight. Afterwards, four consecutive frames (that remained after skipping) are concatenated to form a state representation, so that the required information concerning the motion of objects is provided. Finally, all rewards are clipped to \(\{-1,0,1\}\). An example state of each game before and after applying the preprocessing steps is depicted in Figure 5.  
Several values are tested for the confidence threshold of the predicted actions, as well as the weight (percentage) of each data category on the mixed batches. Regarding the agent network, it consists of three convolutional layers, followed by a dense layer before the output layer. The replay buffer's size is 100,000 samples, while the synthetic buffer contains 20,000 generated samples. Complete implementation details of the agent model are provided in Table A2. Figure 6 depicts the accuracy of the action-predicting model depending on the confidence threshold for each game. As the selecting condition becomes more strict, the number of samples that can be used for the agent's training is reduced. This explains, to a certain degree, the decrease in accuracy after a certain threshold (which is expected to increase along with the confidence value), since in this case it is based on a small number of samples. In this respect, there is a tradeoff between the accuracy of the model and the amount of samples to be rejected, as a very small number of synthetic samples may cause overfitting and negatively affect the agent's training. Therefore, for each game several confidence thresholds have been tested for the respective action-predicting model in order to determine the best configuration. Figure 7 shows the average reward achieved by different configurations of the proposed agent on the Boxing game. Specifically, Figure 7a summarizes the performance of the agent for several confidence thresholds and synthetic data weight equal to 0.1, while Figure 7b outlines the respective results for synthetic data weight value 0.2. The reward is calculated every 50,000 timesteps and is averaged over 50 runs. As expected, the confidence value greatly affects the agent's learning. Higher thresholds lead to a higher accuracy of the prediction model and a lower percentage of incorrect synthetic samples. In this respect, the performance increase along with the increase in the confidence, as highlighted in Figure 7a, seems quite natural.