# Experience Replay Optimisation via ATSC and TSC for Performance Stability in Deep RL
## 5 Results and Discussion
After multiple implementations of Framework 2 with reward thresholds from \(-1\) to \(-5\), the reward of \(-1\) is selected, since it generates the median return across the four retention mechanisms. Subsequently, an equal number of multiple experiments are conducted, and the results are shown in Figure 6a-d. Again, the sampling ratio of 9:1 displays the best and most consistent performance across the four buffer sizes. The ratio of 1:1 still demonstrates the minimum performance with an apparent declining return on the F1F2 retention strategy. This scenario occurs for buffer sizes from \(1\times 10^{4}\) to \(1\times 10^{6}\). Thus, it is instructive to note that an equal rate of experience selection from both replay buffers, especially in the first framework, is not healthy for the successful training of the RL agent. After identifying the superiority of Framework 2 over Framework 1, an evaluation of the former is repeated but with a TDE threshold. The implementation occurs with the exact buffer sizes and retention strategies but with a 9:1 sampling ratio and a threshold of \(|\,\mathrm{TDE}\,|<1\). This approach is more flexible and easily applicable for continual learning; it does not require the foreknowledge of a reward function to implement the TSC threshold. Focusing on our objectives and upon establishing that the sampling ratio of 9:1 has the best performance in both frameworks, the memory and retention strategy results are extracted  
Figure 6: Performance of Framework 2 across multiple buffer sizes, retention strategies, and sampling ratios. (**a**) Framework 2 is evaluated using the FIFO retention strategy for both replay buffers; (**b**) Framework 2 is evaluated using FIFO for the first replay buffer and enhanced FIFO for the second replay buffer; (**c**) Framework 2 is evaluated using enhanced FIFO and FIFO for the first and second replay buffers, respectively; (**d**) Framework 2 is evaluated using enhanced FIFO for both replay buffers. and analysed. Tables 5 and 6 present the compared results of Framework 1 and the two versions of Framework 2. We ascertain the framework with the most stable results across buffer sizes by computing the average returns across the four buffer sizes. As shown in Table 5, the F2F1 retention strategy proves superior and serves as a basis for comparing the performance of the frameworks across all four buffer sizes. Despite their competitive performance, as indicated in Figure 7 and Table 7, the reward version of the proposed framework outperforms the others. This is because the TSC ensures that more relevant transitions are selected for storage and training. It equally ensures a better distribution of transitions to prevent overfitting and underfitting during the agent's training. In effect, it boosts the generalisation of the framework. Additionally, the ATSC coupled with the dual memory promote complementary and stable learning. While the dual memory offers a more diverse storage of transitions, the ATS timeously alternate the selection of these transitions for learning. It is also insightful to note that a wrong choice of retention strategy or sampling ratio can hamper the performance of the framework. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Frameworks** & \(\mathbf{1\times 10^{6}}\) & \(\mathbf{1\times 10^{5}}\) & \(\mathbf{1\times 10^{4}}\) & \(\mathbf{1\times 10^{3}}\) \\ \hline Framework 1 & \(-188.28\) & \(-156.43\) & \(-182.50\) & \(-182.76\) \\ Framework 2 (Reward) & \(-155.43\) & \(-155.46\) & \(-179.53\) & \(-192.42\) \\ Framework2 (TDE) & \(-157.58\) & \(-160.97\) & \(-173.80\) & \(-188.39\) \\ \hline \hline \end{tabular}
\end{table} 
Table 6: The performance of the 9:1 sampling ratio on the frameworks across multiple buffer sizes using the F2F1 retention strategy.