Before revision:  Previous prioritization methods based on temporal-difference (TD ) errorare highly heuristic and divergent from the objective of RL . In this work, we analyze the optimal prioritization strategy that can minimize the regret of RL policytheoretically. Our theory suggests that the data with higher  TD error, better on-policiness and more corrective feedback should be assigned with higher weights during sampling.
After revision:  Previous prioritization methods based on temporal-difference (TD ) errorare highly heuristic and divergent from the objective of RL . In this work, we analyze the optimal prioritization strategy for Bellman update that can directly maximize the return of the policy. The theory suggests that the data with higher TD error, better on-policiness and more corrective feedback should be assigned with higher weights during sampling.