# Experience Replay Optimisation via ATSC and TSC for Performance Stability in Deep RL
## 2 Related Works
### Experience Selection Strategies and Algorithms  
The effectiveness of experience selection or sampling is rated not by the sample outcome, but also by the strategies implemented to produce the needed outcome. Irrespective of the experience selection strategy adopted, the sampling mechanism may be uniform or prioritised but not sequential; sequential sampling will generate highly correlated experiences and eventually prevent the agent from learning. However, a uniform or prioritised sampling will reduce the cost of training, break the correlation between transitions, and improve sampling efficiency and learning stability. Many selection strategies have been developed and combined with some RL algorithms to solve simple and complex problems. The DQN algorithm of [PERSON] et al. [8] combines a deep convolutional neural network with the standard ER and the Q-learning [27] method to solve the challenging Atari 2600 games; in this approach, the experiences are selected uniformly at random. Each experience in the replay buffer has an equal probability or chance of being selected for replay. For a set of experiences, E(e\({}_{1}\), e\({}_{2}\),..., e\({}_{n}\)), the probability of any experience is given as Pi = 1/n. Contrary to uniform sampling, some selection strategies draw inspiration from the prioritised sweeping [28] technique and, therefore, assign higher selection probability to experiences that are deemed important and can generate good returns. Prioritised strategies assign importance to experiences based on temporal difference error (TDE) [9], reward [13; 15], frequency of selection [17], distribution of transitions [13], and even the similarities between states [16]. While some selection strategies are fully prioritised (only the highest priority transitions are selected), others have a parameter to regulate the level of prioritisation. The parameterised prioritisation makes it flexible to scale from uniform to fully prioritised sampling. [PERSON] et al. [9] present a prioritised experience replay (PER), where experiences with higher TDE are selected frequently. This situation can result in a loss of diversity but can be addressed with stochastic prioritisation. However, prioritisation can create a bias,which can also be corrected with importance sampling. While [PERSON] et al. [29] extend prioritised experience replay to a distributed setting, [PERSON] et al. [15] use a replay policy (neural network) combined with both uniform and prioritised selection techniques to select transitions for replay. [PERSON] and [PERSON] handle state-action space diversity by over-sampling under-represented experiences in a non-diverse state-action space. [PERSON] et al. [15; 16] also sample experiences based on the similarities between the states of those experiences and the state of the agent. They equally focus on experiences that are frequently visited. Just as multiple neural networks are used in some RL algorithms to enhance training stability [8; 11; 15; 26; 30; 31], recent works in ER are exploring the use of a dual-memory architecture. The dualism may come in the form of long and short memory [32] or main and cache [33]. It may also be differentiated based on the sources of the replay data or the ratio of selection from the dual memory. [PERSON] et al. [32] rely on the complementary learning system of the human brain (interaction between the cortical and hippocampal networks) to design a dual memory (short-term and long-term) replay architecture. However, their design was implemented in a discrete state-action space. [PERSON] and [PERSON] [33] use the main memory to store diverse experiences and cache memory to manage the frequently used experiences to train the neural network. On the other hand, a ratio can be employed to alternate the sampling of relevant experiences from one memory and recent experiences from another memory [17]. Table 1 shows a tabulated summary of some RL algorithms and their adopted ER strategies.