SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Kumar et al. [2020] presents Distribution Correction (DisCor), a form of corrective feedback to make learning dynamics more steady. DisCor computes the optimal distribution and performs a weighted Bellman update to re-weight data distribution in the replay buffer. Inspired by DisCor, Regret Minimization Experience Replay (ReMERN) Liu et al. [2021] estimates the suboptimality of the Q value with an error network. Yet, Hong et al. [2022] uses Topological Experience Replay (TER) to organize the experience of agents into a graph that tracks the dependency between Q-value of states.
While PER was initially proposed as an addition to DQN-style agents, Hou et al. [2017] have shown that PER can be a useful strategy for improving performance in Deep Deterministic Policy Gradients (DDPG) Lillicrap et al. [2016]. Another recent strategy to improve sample efficiency was to introduce losses from the transition dynamics along with the TD error as the priority Oh et al. [2022]. Although this has shown improvements, it involves additional computational complexity since it also requires learning a reward predictor and transition predictor for the environment. Our proposal does not require training additional networks and hence is similar in computational complexity to PER. This makes it very simple to integrate into any existing algorithm. Wang and Ross [2019] propose an algorithm to dynamically reduce the replay buffer size during training of SAC so that the agent prioritizes recent experience while also ensuring that updates performed using newer data are not overwritten by updates from older data. However, they do not distinguish between points based on learn-ability and only assume that newer data is more useful for the agent to learn.
D.2 Off-Policy Algorithms
Off-policy algorithms are those that can learn a policy by learning from data not generated from the current policy. This improves sample efficiency by reusing data collected by old versions of the policy. This is in contrast to on-policy algorithms such as PPO Schulman et al. [2017], which after collecting a batch of data and training on it, discard those samples and start data collection from scratch. Recent state-of-the-art off-policy algorithms for continuous control include Soft Actor Critic (SAC) Haarnoja et al. [2018] and Twin Delayed DDPG (TD3) Fujimoto et al. [2018]. SAC learns two  $Q$  networks together and uses the minimum of the  $Q$  values generated by these networks for the Bellman update equation to avoid over estimation bias. The  $Q$  target update also includes a term to maximize the entropy of the policy to encourage exploration, a formulation that comes from Maximum Entropy RL Ziebart et al. [2008]. TD3 is a successor to DDPG Lillicrap et al. [2016] which addresses the overestimation bias present in DDPG in a similar fashion to SAC, by learning two  $Q$  networks in parallel, which explains the "twin" in the name. It learns an actor network  $\mu$  following Eq. 4 to compute the maximum over  $Q$  values. TD3 proposes that the actor networks be updated at a less frequent interval than the  $Q$  networks, which gives rise to the "delayed" name. In discrete control, Rainbow Hessel et al. [2017] combines several previous improvements over DQN, such as Double DQN van Hasselt et al. [2016], PER Schaul et al. [2016], Dueling DQN Wang et al. [2016], Distributional RL Bellemare et al. [2017] and Noisy Nets Fortunato et al. [2018].
E DeepMind Control Suite
We choose 9 environments from the DeepMind Control Suite Tassa et al. [2018] for testing the performance of ReLo on continuous control tasks. Each agent was trained on proprioceptive inputs from the environment for 1M frames with an action repeat of 1. The training curves for the baselines and ReLo are given in Fig. 8.
Table 7: Comparison of PER and ReLo on the DMC benchmark
BASELINE	PER	LABER	RELO
CHEETAH RUN	761.89 ± 112.38	831.87 ± 38.90	579.80 ± 60.61	660.29 ± 141.22
FINGER SPIN	966.68 ± 29.40	975.40 ± 6.75	884.46 ± 20.23	978.78 ± 14.46
HOPPER HOP	264.75 ± 37.90	217.35 ± 113.79	90.85 ± 57.76	247.81 ± 51.01
QUADRUPED RUN	612.67 ± 143.90	496.42 ± 216.01	544.80 ± 41.84	833.92 ± 81.05
QUADRUPED WALK	831.92 ± 74.34	766.30 ± 200.86	716.61 ± 270.36	942.64 ± 9.75
REACHER EASY	983.06 ± 2.70	981.58 ± 6.33	947.20 ± 14.46	979.08 ± 11.02
REACHER HARD	955.08 ± 38.52	935.08 ± 47.94	951.08 ± 6.70	956.80 ± 38.73
WALKER RUN	759.13 ± 23.91	755.49 ± 64.35	551.81 ± 58.41	795.14 ± 42.52
WALKER WALK	943.67 ± 30.28	957.38 ± 8.24	863.89 ± 112.60	963.28 ± 5.03
Figure 8: Training curves of environments from the DeepMind Control Suite. Performance is evaluated for 10 episodes over 5 random seeds.
F OpenAI Gym Environments
We evaluate agents for 1M timesteps on each environment and similar to DM Control, they are trained using proprioceptive inputs from the environment. The hyperparameters for this benchmark are shared with those used for the DM Control Suite experiments.
Table 8: Comparison of PER and ReLo on OpenAI Gym environments
BASELINE	PER	RELO
GYM HALFCHEETAH	9579.60 ± 1331.00	9549.42 ± 917.92	11590.63 ± 670.36
GYM HOPPER	2795.18 ± 659.19	2175.09 ± 456.11	2527.93 ± 648.61
GYM WALKER	2854.20 ± 623.69	735.16 ± 474.89	3514.39 ± 676.80
G MinAtar
We evaluate the baselines against all 5 environments in the MinAtar suite Young and Tian [2019]. A visualization of a few environments from the suite is presented in Fig. 5. Each agent receives
Figure 9: Training curves of environments from the OpenAI Gym benchmark. Performance is evaluated for 10 episodes over 5 random seeds.
the visual observations from the environment and is trained for 5M frames following the evaluation methodology outlined in Young and Tian [2019]. The training curves are given in Fig. 10.
Figure 10: Training curves of environments from the MinAtar benchmark. Performance is evaluated using a running average over the last 1000 episodes over 5 random seeds.
Table 9: Comparison of PER and ReLo on the MinAtar benchmark
BASELINE	PER	LABER	RELO
ASTERIX	12.54 ± 1.08	16.16 ± 1.02	15.51 ± 1.11	15.68 ± 0.89
BREAKOUT	9.36 ± 0.29	8.88 ± 0.72	8.81 ± 0.61	8.98 ± 0.75
FREEWAY	52.80 ± 0.35	52.75 ± 0.22	41.98 ± 20.99	53.25 ± 0.37
SEAQUEST	16.13 ± 2.88	6.02 ± 1.92	14.63 ± 2.98	18.13 ± 1.25
SPACE INVADERS	45.36 ± 1.65	37.36 ± 4.45	43.67 ± 3.17	38.54 ± 2.60
H Arcade Learning Environment
We evaluate agents on a compute-constrained version of the Arcade Learning Environment Bellemare et al. [2013], training each agent for 2M frames. We chose the standard 24 environments from the suite for our evaluation. ReLo is competitive with PER Schaul et al. [2016] in the tested environments. The training curves for the Temporal Difference Error and the rewards are given in Fig. 11 & Fig. 12 respectively.
I Gridworld
We implement a simple GridWorld for the experiments that highlights the drawbacks of PER with TD loss prioritization in Section 5.6. It consists of a  $7 \times 7$  grid. A visualization of the grid is given in Fig. 13. The start state of the agent, represented by the blue point, is at the top left and the goal state is at the bottom right, represented by the green point. The agent is represented by the black point. The
Figure 11: Temporal difference loss curves for Rainbow (with PER) and Rainbow with ReLo. Rainbow with ReLo achieves lower loss compared to PER, showing that ReLo is able to prioritize samples with reducible loss. The dark line represents the mean and the shaded region is the standard deviation over 3 seeds.
Figure 12: Training curves of 24 environments from the ALE benchmark. Performance is evaluated for 10 episodes over 5 random seeds.
agent gets a reward of  $+2$  when it reaches the goal state, but does not receive reward anywhere else other than the stochastic point. The red point marks the state with a corresponding reward uniformly sampled from  $[-0.5, 0.5]$ . The locations of these points is fixed and does not change during training or evaluation. The state that the agent receives is  $(x, y)$  where  $x$  and  $y$  are the coordinates of the agent's location,  $x, y \in [-3, 3]$  and  $x, y \in \mathbb{Z}$ . The agent has 4 actions, to move up, down, left or right which will deterministically move the agent in that direction by 1 unit. If the agent is at the edge of the grid and takes an action that will move it out of the  $7 \times 7$  grid, then it remains in the same location.
Table 10: Comparison of Rainbow with PER and Rainbow with ReLo on the ALE benchmark
RAINBOW	RAINBOW W/ RELO
ALIEN	1278.70 ± 223.14	1352.90 ± 535.70
AMIDAR	376.20 ± 81.07	410.10 ± 85.63
ASSAULT	2241.78 ± 648.17	2617.37 ± 555.97
ASTERIX	3214.50 ± 323.70	3352.00 ± 431.98
BANKHEIST	526.80 ± 277.83	641.80 ± 284.67
BOXING	76.65 ± 14.58	76.21 ± 11.59
BREAKOUT	82.03 ± 46.10	68.36 ± 45.39
CHOPPERCOMMAND	2794.00 ± 732.87	4974.00 ± 2801.93
DEMONATTACK	25500.50 ± 16311.26	29294.30 ± 15905.25
FREEWAY	32.58 ± 0.31	32.35 ± 0.25
FROSTBITE	2850.60 ± 1553.34	3532.30 ± 1270.73
GOPHER	5336.60 ± 1023.75	5679.80 ± 1199.68
HERO	9907.15 ± 2108.71	8830.30 ± 1894.51
JAMESBOND	810.00 ± 412.00	902.00 ± 469.40
KANGAROO	8904.00 ± 3879.97	8091.00 ± 3618.47
KRULL	6553.18 ± 803.15	6718.67 ± 799.47
KUNGFUMASTER	29371.00 ± 8525.69	23654.00 ± 12360.40
MSPACMAN	2094.70 ± 614.58	1755.80 ± 374.32
PONG	3.18 ± 9.02	5.96 ± 6.45
PRIVATEEYE	100.00 ± 0.00	100.00 ± 0.00
QBERT	8382.00 ± 2935.16	10900.25 ± 2704.23
ROADRUNNER	29333.00 ± 9465.78	29222.00 ± 8696.91
SEAQUEST	1377.20 ± 362.57	1848.80 ± 788.85
UPNDOWN	17065.40 ± 7637.25	21241.50 ± 8599.97
Figure 13: A top down view of the GridWorld. The agent is the black point. It starts at the blue point and the goal state is the green point. The red point represents the location of the stochastic reward.
J Forgetting Experiment
We visualize the gridworld used for the forgetting experiment in Fig. 14. The training curves are given in Fig. 15.
K Correlation between Validation TD Error and Policy Performance
We study the correlation between validation TD error and the performance of the policy and are able to reproduce the findings of Li et al. [2023], showing that there is indeed high correlation between the validation TD error and the performance. The results are in Table 11. We performed a paired t-test and bold entries where the results are significant.
Figure 14: Visualization of Grid Room. Cyan is goal state in Room A, green is goal state in Room B. Goal A gives a reward of  $+1$  and Goal B gives  $+5$ .
Figure 15: Performance curves of Uniform sampling, PER and ReLo on the forgetting task. Success rates are averaged over 60 seeds and shaded regions indicate standard deviation. Once the agent is no longer able to collect data about Task A (>100K steps), performance on the Task A decreases, but ReLo exhibits the least degradation in performance on Task A while still learning to solve Task B.
Table 11: Validation TD Error and Policy Performance on the DMC benchmark
METHOD	TD ERRORBEST	RETURNBEST
CHEETAHRUN	PER	PER
FINGERSPIN	RELO	RELO
HOPPERHOP	RELO	BASELINE
QUADRUPEDRUN	RELO	RELO
QUADRUPEDWALK	LABER	RELO
REACHEREASY	RELO	BASELINE
REACHERHARD	BASELINE	RELO
WALKERRUN	RELO	RELO
WALKERWALK	RELO	RELO
L Instability of PER in Mujoco Environments
We believe that degraded performance of the PER agent in Mujoco is because of instability in learning caused by rapidly varying value estimates. To test this hypothesis we studied the Walker2d environment, where PER obtains a mean reward of 700, compared to the baseline (uniform sampling) which obtains a mean reward of 2800. This is surprising since PER does learn to perform well in WalkerWalk, which is the DMC equivalent of Walker2d (they have similar agent morphologies).
An important difference in the two environments is that Walker2d has early termination (episode ends when the walker falls down) while WalkerWalk does not. This meant that the early terminations could make predicting the value of the fail state distribution difficult as the walker falls in different ways. There could be a lot of noise in the reward at that stage which can make the TD estimate noisy too.
We hypothesize that since PER only samples datapoints proportional to the TD error, these states would be repeatedly sampled as the noisy estimate would make the TD error high. This means it would be less likely to prioritize the samples corresponding to good behavior which could have lower TD error than the noisy fail states. We looked at the value estimate of the fail state in the baseline, PER and ReLo agent and observed this was the case. This is presented in Table 12 and estimates are calculated over 40 episodes.
Table 12: Variance in Value Estimate of Fail State Distribution.
METHOD	MEAN (CIS)	MIN	MAX
BASELINE	29.055 (-62.551, 120.662)	-87.8392	258.484
PER	149.982 (-76.382, 376.346)	-115.291	793.237
RELO	18.225 (-0.307, 36.756)	-14.4796	49.37
There is high variance in the predicted value of the fail state for PER, meaning that invariably the TD error for these points would be high. But further training on noisy points does not help and instead makes the problem worse, causing the value estimate to diverge. This can cause instabilities in training and potentially derail learning.
Finally we created a modified version of Walker2d without early terminations and PER achieves much better performance (Mean score of 1943.65 compared to 700.5 in the original Walker2d) in this environment, validating our hypothesis. Besides removing early termination, other parameters of the environment and the hyperparameters of the PER agent were the same in both experiments. We also looked at the value of the initial states (the environment is randomly initialized so there is an initial state distribution) in Table 13 and PER has higher variance in the predicted value even here.
METHOD	MEAN	MIN	MAX
BASELINE	227.916 (215.415, 240.417)	215.929	252.337
PER	249.379 (177.265, 321.494)	177.968	337.663
RELo	203.893 (195.357, 212.428)	193.014	214.811
Table 13: Variance in Value Estimate of Fail State Distribution.
This analysis adds credence to our hypothesis that PER suffers from high variance in value estimates which hurt learning. Additionally, the experiments also show that ReLo has the least variance in the predicted value of the state (initial or fail state), highlighting how ReLo is a more stable prioritization scheme.