# Experience Replay Optimisation via ATSC and TSC for Performance Stability in Deep RL
## 3 General Framework for ERO-ATSC
### ERO and Alternating ERO
#### 3.1.2 Alternating Transition Selection Framework  
Since [PERSON] [6] introduced experience replay in his reinforcement learning research, many strategies have evolved for experience selection and retention. Most of these strategies use a single buffer. However, in recent times, researchers have been exploring the use of dual memory for experience replay. Consequently, we design and investigate an alternating selection strategy that harnesses the strength of ERO and dual memory. Figures 3 and 4 show the design of two experience replay optimisation with alternating transition selection control (ERO-ATSC) frameworks; Figure 4 has a TSC but Figure 3 does not. In the first framework, the double experience replay (DER) strategy, proposed by [PERSON] et al. [17], is combined with the ERO's replay policy. We store transitions from the environment into only H1. When replay begins in H1, the replayed transitions from H1 are stored in H2. Subsequently, the ATSC uses a predetermined ratio to sample a mini-batch of transitions from either H1 or H2. The mini-batch is further vectorised, prioritised by the replay policy, and uniformly sampled to train the agent. Afterwards, the replay policy takes feedback from the training environment to assess its performance. \begin{table}
\begin{tabular}{c c} \hline \hline
**Notation** & **Explanation** \\ \hline \(\phi\) & Function approximator \\ \(B\) & Replay buffer \\ \(B_{i}\) & A transition in the replay buffer B \\ \(\phi^{b}\) & Parameters of the function approximator \\ \(f_{B_{i}}\) & Feature vector \\ \(\lambda\) & Priority score \\ \(r^{\prime}\) & Cumulative reward \\ \(r^{c}_{\pi}\) & Cumulative reward of current policy \\ \(r^{c}_{\pi^{\prime}}\) & Cumulative reward of previous policy \\ \(B^{s}\) & A specified batch size of sampled transitions \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Symbols and notations used in this section. Figure 2: An illustration of experience replay optimisation (ERO). A specified batch size of transitions is sampled from the replay buffer, using a replay policy (deep neural network), to train the agent. When the buffer is full, the First-In-First-Out (FIFO) retention strategy is applied to ensure that older transitions are replaced with newer ones. Conversely, in the second framework, as indicated in Figure 4 and elaborated with Algorithm 1, we alter the DER strategy with the addition of a TSC mechanism. We store only  
Figure 4: Framework 2: this framework combines experience replay optimisation (ERO) with a dual memory and an alternating transition selection control (ERO-ATSC) with a transition storage control (TSC) mechanism. Transitions from the environment are stored in only H1. When a specified mini-batch size is reached, the replayed transitions from H1 are stored in H2. The ATSC uses a defined ratio to sample a mini-batch of transitions from either H1 or H2 into the replay policy network. These transitions are vectorised and prioritised and uniformly sampled at random to train the agent. Feedback from the training environment is sent to the replay policy network for policy evaluation. Figure 3: Framework 1: this framework combines experience replay optimisation (ERO) with a dual memory and an alternating transition selection control (ERO-ATSC) without a transition storage control (TSC) mechanism. Transitions from the environment are stored in only H1. When a specified mini-batch size is reached, the replayed transitions from H1 are stored in H2. The ATSC uses a defined ratio to sample a mini-batch of transitions from either H1 or H2 into the replay policy network. These transitions are vectorised and prioritised and uniformly sampled at random to train the agent. Feedback from the training environment is sent to the replay policy network for policy evaluation. transitions that meet the TSC threshold into H2 instead of the replayed data from H1. We store transitions from the environment into the replay buffers H1 or H2 or both depending on how well they meet the storage threshold.