# Fresher Experience Plays a More Important Role in Prioritized Experience Replay
## 4 Algorithm  
In this work, we respectively embed FPER in Dueling DQN and DDPG to evaluate its performance. Both algorithms follow the corresponding paradigm, which are shown in Algorithms 1 and 2.  
```
1:Input: mini-batch \(k\), learning rate \(\eta\), replay buffer size \(N\), exponent \(\alpha\), episode \(M\), freshness discounted factor \(\mu\)
2:Initialize action-value network \(Q(s,a,\phi)\), target network \(Q^{\prime}(s,a,\phi^{\prime})\)
3:Initialize replay memory \(\mathcal{H}=\emptyset\), count array \(C\), lifetime pointer \(L\), \(\Delta=0\)
4:for episode = 1, \(M\)do
5: Observe initial state \(s_{1}\)
6:for\(t=1\), \(T\)do
7: Choose \(a_{t}\sim\pi_{\phi}(s_{t})\), obtain reward \(r_{t}\) and new state \(s_{t+1}\)
8: Store experience \((s_{t},a_{t},r_{t},s_{t+1})\) in position \(L\) with maximal priority \(p_{t}=\max_{i<t}p_{i}\)
9: Reset \(C(L)=0\)
10:if\(t>N\)then
11:\(L\) points to the position with lowest absolute TD-error
12:for\(j=1\), \(k\)do
13: Sample experience \(j\sim P(j)=p_{j}^{\pi}/\sum_{j}p_{j}^{\pi}\)
14: Update count array \(C(j)\gets C(j)+1\)
15: Compute importance-sampling weight \(w_{j}\) and TD-error \(\delta_{j}\)
16: Update experience priority \(p_{j}\leftarrow\mu^{C(j)}|\delta_{j}|+\epsilon\)
17: Accumulate weight-change \(\dot{\Delta}\leftarrow\Delta+w_{j}\cdot\delta_{j}\cdot\
abla_{\phi}Q(s,a,\phi)\)
18:endfor
19: Update parameters \(\phi\leftarrow\phi+\eta\cdot\Delta\), reset \(\Delta=0\)
20: From time to time update target network parameters \(\phi^{\prime}\leftarrow\phi\)
21:endif
22:endfor
23:endfor
```  
**Algorithm 1** Dueling DQN with FPER  
Figure 1: Replay buffer with lifetime pointer.  
```
1:Input: mini-batch \(k\), learning rates \(\eta_{1}\) and \(\eta_{2}\), replay buffer size \(N\), exponent \(\alpha\), episode \(M\), freshness discounted factor \(\mu\), updating rate of the target network \(\lambda\)
2:Initialize action-value network \(Q(s,a,w)\), actor network \(\pi(s,v)\), target network \(Q^{\prime}(s,a,w^{\prime})\) and \(\pi^{\prime}(s,v^{\prime})\)
3:Initialize replay memory \(\mathcal{H}=\mathcal{O}\), count array \(C\), lifetime pointer \(L\), \(\Delta=0\)
4:for episode = 1, \(M\)do
5: Initialize a random noise process \(\Phi\)
6: Observe initial state \(s_{1}\)
7:for\(t=1\), \(T\)do
8: Add noise \(\Phi_{t}\) in the exploration policy and select action \(a_{t}\) according to the new policy
9: Obtain reward \(r_{t}\) and new state \(s_{t+1}\)
10: Store experience \((s_{t},a_{t},r_{t},s_{t+1})\) in position \(L\) with maximal priority \(p_{t}=\max_{i<t}p_{i}\)
11: Reset \(C(L)=0\)
12:if\(t>N\)then
13:\(L\) points to the position with lowest TD-error
14:for\(j=1\), \(k\)do
15: Sample experience \(j\sim P(j)=p_{j}^{\alpha}/\sum_{j}p_{j}^{\alpha}\)
16: Update count array \(C(j)\leftarrow C(j)+1\)
17: Compute importance-sampling weight \(w_{j}\) and TD-error \(\delta_{j}\)
18: Update experience priority \(p_{j}\leftarrow\mu^{C(j)}|\delta_{j}|+\epsilon\)
19:endfor
20: Minimize the loss function to update action-value network: \(L=\frac{1}{K}\sum_{j}w_{j}\delta_{j}^{2}\)
21: Compute policy gradient to update the actor network: \(\
abla_{\eta}J\approx\frac{1}{K}\sum_{j}\
abla_{\eta}Q(s,a,w)|_{S_{t}=s_{j},A _{t}=\pi(s_{j},v)}\
abla_{v}\pi(s,v)|_{S_{t}=s_{j}}\)
22: From time to time adjust parameters of target network \(Q^{\prime}(s,a,w^{\prime})\) and \(\pi^{\prime}(s,v^{\prime})\) with updating rate \(\lambda\)
23:endif
24:endfor
25:endfor
```  
**Algorithm 2** DDPG with FPER