# Experience Replay Optimisation via ATSC and TSC for Performance Stability in Deep RL
## 5 Results and Discussion  
In this section, we commence our discussion with careful observation of the results from both frameworks. We then compare the performance of the two frameworks with a critical interest in the retention strategy used, the sampling ratio selected, and the choice of buffer size. In Table 4, we give clarity to the notations that are used in the presentation of the results. The discussion in this section commences with careful observation of the results from both frameworks, followed by a performance comparison of the two frameworks with a \begin{table}
\begin{tabular}{c c} \hline \hline
**Notation** & **Explanation** \\ \hline H1 & The first replay buffer \\ H2 & The second replay buffer \\ F1 & First-In-First-Out (FIFO) \\ F2 & Enhanced FIFO \\ F1F1 & Both H1 and H2 use F1 \\ F1F2 & H1 uses F1 and H2 uses F2 \\ F2F2 & Both H1 and H2 use F2 \\ F2F1 & H1 uses F2 and H2 uses F1 \\ R1 & Experiences are selected from H1 and H2 with a ratio of 1:1 \\ R2 & Experiences are selected from H1 and H2 with a ratio of 9:1 \\ R3 & Experiences are selected from H1 and H2 with a ratio of 1:9 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: List of major notations. \begin{table}
\begin{tabular}{c c} \hline \hline
**Hyperparameter** & **Value** \\ \hline Batch size & 64 \\ Actor learning rate & \(1\times 10^{-4}\) \\ Critic learning rate & \(1\times 10^{-3}\) \\ Gamma & 0.99 \\ Eval steps & 100 \\ Eval episodes & 10 \\ Environment complexity (seed) & \(1\times 10^{6}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Some parameters used in the implementation of the two frameworks. critical interest in the retention strategy used, the sampling ratio selected, and the choice of buffer size. Firstly, we conduct multiple experiments on Framework 1 across the three sampling ratios, the four dual retention strategies, and the four replay buffer sizes. Since the aim of the agent is to accumulate rewards, the frameworks are evaluated based on the mean returns, which is a summation of all rewards during an episode (subsequences of agent-environment interactions) divided by the number of episodes. The bigger the return, the better the performance. Equations (5) and (6) show how the mean return is computed:  
\[G_{t}=R_{t+1}+R_{t+2}+R_{t+3}+\ldots+R_{T} \tag{5}\]  
\[\text{Mean return}=\frac{G_{t}}{Number\ of\ episodes} \tag{6}\]  
where \(G_{t}\) is the return, \(R_{t+1}\) is the reward at time step \(t\), and \(T\) is the final time step. From Figure 5a-d, we observe that, while the sampling ratio of 1:1 records the worst return, the sampling ratio of 9:1 has the best and most consistent performance across the buffers and retention strategies. The output in these figures shows that Framework 1 is not heavily influenced by the retention strategy but by the sampling ratio. Thus, a higher selection frequency from a buffer with more diverse and recent transitions is essential for the stable performance of the RL agent. Additionally, the sampling ratio of 1:9 produces a rise in performance across the four retention strategies as the buffer size increases. The performance of the 1:1 sampling ratio declines for the F1F2 and the F2F2 strategies as the buffer size moves from \(1\times 10^{5}\) to \(1\times 10^{6}\). Figure 5: Performance of Framework 2 across multiple buffer sizes, retention strategies, and sampling ratios. (**a**) Framework 1 is evaluated using the FIFO for both replay buffers; (**b**) Framework 1 is evaluated using FIFO for the first replay buffer and enhanced FIFO for the second replay buffer; (**c**) Framework 1 is evaluated using enhanced FIFO and FIFO for the first and second replay buffers, respectively; (**d**) Framework 1 is evaluated using enhanced FIFO for both replay buffers. The second set of experiments follows the same setting as the previous experiments but includes a TSC threshold. The implementation begins with a reward threshold and continues with a TDE threshold.