SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
J _ {Q _ {\beta}} = \mathbb {E} _ {\bar {s} _ {t} \sim \rho^ {\bar {\pi}}, a _ {t} \sim \bar {\pi}} \left[ \mathbb {E} _ {\mathbb {P} (s _ {t} | \bar {s} _ {t})} \left[ Q _ {\beta} ^ {\bar {\pi}} \left(s _ {t}, a _ {t}\right) \right] - \bar {Q} ^ {\bar {\pi}} \left(\bar {s} _ {t}, a _ {t}\right) \right] ^ {2}. \tag {15}
$$
Let the augmented  $Q$ -value be a soft  $Q$ -value; then, substituting  $\bar{Q}^{\bar{\pi}}(s_t,a_t)$  into (15) with the target value in (13), Equation (15) can be expanded to the following equation:
$$
\begin{array}{l} J _ {Q _ {\beta}} = \mathbb {E} _ {\bar {s} _ {t} \sim \rho (\bar {s}), a _ {t} \sim \bar {\pi}} \left[ \mathbb {E} _ {\mathbb {P} (s _ {t} | \bar {s} _ {t})} [ Q _ {\beta} ^ {\bar {\pi}} (s _ {t}, a _ {t}) - R (s _ {t}, a _ {t}) ] - \gamma \mathbb {E} _ {\bar {s} _ {t + 1} \sim \bar {P} (\cdot | \bar {s} _ {t}, a _ {t}), a _ {t + 1} \sim \bar {\pi}} [ \bar {Q} ^ {\bar {\pi}} (\bar {s} _ {t + 1}, a _ {t + 1}) \right. \\ \left. - \alpha \log \bar {\pi} \left(a _ {t + 1} \mid \bar {s} _ {t + 1}\right) \right] ^ {2}. \tag {16} \\ \end{array}
$$
Subsequently, we replace the augmented state-based soft  $Q$ -value with the expectation of beta  $Q$ -values:
$$
\bar {Q} ^ {\bar {\pi}} \left(\bar {s} _ {t + 1}, a _ {t + 1}\right) \approx \mathbb {E} _ {\mathbb {P} \left(s _ {t + 1} \mid \bar {s} _ {t + 1}\right)} \left[ Q _ {\beta} ^ {\bar {\pi}} \left(s _ {t + 1}, a _ {t + 1}\right) \right] \tag {17}
$$
by fitting the augmented  $Q$ -value to the belief projection. Then, the objective can be rewritten as:
$$
\begin{array}{l} J _ {Q _ {\beta}} = \mathbb {E} _ {\bar {s} _ {t} \sim \rho (\bar {s}), a _ {t} \sim \bar {\pi} (\bar {s} _ {t})} \left[ \mathbb {E} _ {\mathbb {P} (s _ {t} | \bar {s} _ {t})} \left[ Q _ {\beta} ^ {\bar {\pi}} (s _ {t}, a _ {t}) - R (s _ {t}, a _ {t}) \right] \right. \tag {18} \\ \left. - \gamma \mathbb {E} _ {\bar {s} _ {t + 1} \sim \bar {P}, a _ {t + 1} \sim \bar {\pi}} \Big [ \mathbb {E} _ {\mathbb {P} (s _ {t + 1} | \bar {s} _ {t + 1})} [ Q _ {\beta} ^ {\bar {\pi}} (s _ {t + 1}, a _ {t + 1}) ] - \alpha \log \bar {\pi} (a _ {t + 1} | \bar {s} _ {t + 1}) \Big ] \right] ^ {2}. \\ \end{array}
$$
Now, our objective has changed to finding the beta  $Q$ -values, where their expectation best represents the augmented state-based  $Q$ -values. We can also rewrite Equation (14) using the belief projection
instead of using the augmented  $Q$ -values:
$$
\begin{array}{l} \bar {\pi} _ {\text {n e w}} = \underset {\bar {\pi} ^ {\prime}} {\arg \min } \mathrm {D} _ {\mathrm {K L}} \left( \right.\bar {\pi} ^ {\prime} (\cdot | \bar {s} _ {t}) \left\| \right. \frac {\exp \left(\frac {1}{\alpha} \mathbb {E} _ {\mathbb {P} \left(s _ {t} \mid \bar {s} _ {t}\right)} \left[ Q _ {\beta} ^ {\bar {\pi} ^ {\text {o l d}}} \left(s _ {t} , \cdot\right)\right]\right)}{\bar {Z} ^ {\bar {\pi} ^ {\text {o l d}}} \left(\bar {s} _ {t}\right)}\left. \right) (19) \\ = \arg \min  _ {\bar {\pi} ^ {\prime}} \sum_ {a _ {t \in \mathcal {A}}} \bar {\pi} ^ {\prime} \left(a _ {t} | \bar {s} _ {t}\right) \left(\log \bar {\pi} \left(a _ {t} | \bar {s} _ {t}\right) - \frac {1}{\alpha} \mathbb {E} _ {\mathbb {P} \left(s _ {t} | \bar {s} _ {t}\right)} \left[ Q _ {\beta} ^ {\bar {\pi} _ {\text {o l d}}} \left(s _ {t}, a _ {t}\right) \right] + \log \bar {Z} ^ {\bar {\pi} ^ {\text {o l d}}} (\bar {s} _ {t})\right) (20) \\ = \arg \min  _ {\bar {\pi} ^ {\prime}} \sum_ {a _ {t} \in \mathcal {A}} \bar {\pi} ^ {\prime} \left(a _ {t} \mid \bar {s} _ {t}\right) \left(\log \bar {\pi} \left(a _ {t} \mid \bar {s} _ {t}\right) - \frac {1}{\alpha} \mathbb {E} _ {\mathbb {P} \left(s _ {t} \mid \bar {s} _ {t}\right)} \left[ Q _ {\beta} ^ {\bar {\pi} _ {\text {o l d}}} \left(s _ {t}, a _ {t}\right) \right] + \mathbb {E} _ {\mathbb {P} \left(s _ {t} \mid \bar {s} _ {t}\right)} \left[ \log Z ^ {\bar {\pi} ^ {\text {o l d}}} \left(s _ {t}\right) \right]\right) (21) \\ = \arg \min  _ {\bar {\pi} ^ {\prime}} \mathbb {E} _ {\mathbb {P} \left(s _ {t} \mid \bar {s} _ {t}\right)} \left[ \sum_ {a _ {t \in \mathcal {A}}} \bar {\pi} ^ {\prime} \left(a _ {t} \mid \bar {s} _ {t}\right) \left(\log \bar {\pi} \left(a _ {t} \mid \bar {s} _ {t}\right) - \frac {1}{\alpha} Q _ {\beta} ^ {\bar {\pi} _ {\text {o l d}}} \left(s _ {t}, a _ {t}\right) + \log Z ^ {\bar {\pi} ^ {\text {o l d}}} \left(s _ {t}\right)\right) \right] (22) \\ = \arg \min  _ {\bar {\pi} ^ {\prime}} \mathbb {E} _ {\mathbb {P} \left(s _ {t} \mid \bar {s} _ {t}\right)} \left[ \right. \mathrm {D} _ {\mathrm {K L}} \left( \right.\bar {\pi} ^ {\prime} (\cdot \mid \bar {s} _ {t}) \left\| \right. \frac {\exp \left(\frac {1}{\alpha} Q _ {\beta} ^ {\bar {\pi} _ {\text {o l d}}} \left(s _ {t} , \cdot\right)\right)}{Z ^ {\bar {\pi} _ {\text {o l d}}} \left(s _ {t}\right)}\left. \right)\left. \right], (23) \\ \end{array}
$$
where  $Z^{\bar{\pi}_{\mathrm{old}}}(s_t)$  is a normalizing function for the distribution  $Q^{\bar{\pi}_{\mathrm{old}}}(s_t, \cdot)$ . Equation (21) holds because  $\log \bar{Z}^{\bar{\pi}_{\mathrm{old}}}(\bar{s}_t)$  and  $\mathbb{E}_{\mathbb{P}(s_t|\bar{s}_t)}[\log \bar{Z}^{\bar{\pi}_{\mathrm{old}}}(s_t)]$  are independent of  $\bar{\pi}'$ .
Using Equation (23), we can update the policy by minimizing the expectation of the Kullback Leibler-divergence of the policy and exponential of the beta  $Q$ -value. Notably, we need not evaluate the augmented state-based soft  $Q$ -value, which causes the state-space explosion problem when the delayed timestep is large. By contrast, we evaluate the beta  $Q$ -values and improve the policy using these values.
The beta  $Q$ -value is expected to converge more stably and quicker than the augmented state-based  $Q$ -value because its input state space is considerably smaller than the augmented space, which also helps to obtain an improved policy with higher quality.
In the continuous state setting, the parameterized beta  $Q$ -function  $Q_{\theta ,\beta}^{\bar{\pi}}(s_t,a_t)$  can be approximately computed by minimizing the squared-residual in Equation (18) denoted by  $J_{Q_{\beta}}(\theta)$  with the aid of replay memory  $\mathcal{D}$  [24]:
$$
\begin{array}{l} J _ {Q _ {\beta}} (\theta) = \mathbb {E} _ {(\bar {s} _ {t}, s _ {t}, a _ {t}, r _ {t}, \bar {s} _ {t + 1}, s _ {t + 1}) \sim \mathcal {D}} [ \frac {1}{2} (Q _ {\theta , \beta} ^ {\bar {\pi}} (s _ {t}, a _ {t}) - (r _ {t} + \gamma \mathbb {E} _ {a _ {t + 1} \sim \bar {\pi}} [ Q _ {\theta , \beta} ^ {\mathrm {t a r}} (s _ {t + 1}, a _ {t + 1}) ] \\ \left. - \alpha \log \bar {\pi} _ {\phi} \left(a _ {t + 1} \mid \bar {s} _ {t + 1}\right)\right]\left. \right) ^ {2} ], \\ \end{array}
$$
where  $r_t = R(s_t, a_t)$ ,  $Q_{\theta, \beta}^{\mathrm{tar}}$  is a target network for  $Q_{\theta, \beta}^{\bar{\pi}}$  [24], and  $\bar{\pi}_{\phi}$  is a parameterized policy.
In the policy improvement stage, the policy can be trained by minimizing the objective  $J_{\bar{\pi}}(\phi)$ , which can be formalized from Equation (23) as:
$$
J _ {\bar {\pi}} (\phi) = \mathbb {E} _ {(\bar {s} _ {t}, s _ {t}) \sim \mathcal {D}} \left[ \mathbb {E} _ {a _ {t} \sim \bar {\pi} _ {\phi}} \left[ \alpha \log \bar {\pi} _ {\phi} (a _ {t} | \bar {s} _ {t}) - Q _ {\theta , \beta} ^ {\bar {\pi}} (s _ {t}, a _ {t}) \right] \right]. \tag {25}
$$
By iteratively minimizing  $J_{Q_{\beta}}(\theta)$  and  $J_{\bar{\pi}}(\phi)$ , the actor and critic networks can be trained for a delayed environment. This is the BPQL algorithm. Note that the first and second subscripts of  $Q_{\theta, \beta}$  refer to the neural network weights and the beta  $Q$ -values, respectively. Actually, the beta  $Q$ -values depend on the neural network weights. Nevertheless,  $\theta$  and  $\beta$  are written together for clearer notation. Only  $\theta$  is the variable set to be optimally taken through learning.
In the BPQL algorithm, expanded transition tuples that include augmented states are stored in the replay memory  $\mathcal{D}$ . To construct the augmented state, we use a temporary buffer  $\mathcal{B}$  in which the original states and action history are stored. The details of BPQL are summarized in Appendix A.
We compared the performance of the BPQL algorithm with the following three baselines:
Further details of the implementation of these algorithms are given in Appendix D.
Figure 2: Performance curves of each algorithm for the Walker2d-v3 task. We repeated the test for this task five times with different random seeds. The mean over the results is shown by the solid line, and the shaded area represents the standard deviation. As the delay increases, the proposed algorithm BPQL significantly outperforms other algorithms in terms of asymptotic performance and sample efficiency.
Table 1: Results of MuJoCo benchmark tasks for one million interactions. Each task was evaluated in the delayed environment setting for 3,6, and 9 delayed timesteps  $d$ . All tests were repeated with five different random seeds. All results are shown with the standard deviation over the five trials denoted by  $\pm$ .
Environment	HalfCheetah-v3	Walker2d-v3	Hopper-v3	Swimmer-v3	InvertedPendulum-v2	Reacher-v2
d	Algorithm
3	Normal SAC	-276.2±93.1	672.2±518.8	290.9±86.7	32.8±3.7	16.9±13.7	-28.23±4.66
Delayed-SAC	4182.7±609.7	4463.2±434.0	2821.7±609.1	73.9±32.6	916.5±142.1	-3.95±0.53
Augmented SAC	6054.0±1045.5	3453.3±462.6	2732.6±858.9	47.6±2.1	983.6±32.6	-3.95±0.51
BPQL (ours)	8100.1±543.4	4538.5±271.3	2922.5±671.2	88.0±36.6	998.1±3.6	-3.80±0.51
6	Normal SAC	-288.7±50.7	38.6±29.4	68.3±34.1	32.1±6.7	10.1±2.9	-38.2±3.18
Delayed-SAC	2660.9±492.3	1.0±4.0	1289.2±1071.7	58.2±14.6	929.1±141.7	-4.02±0.50
Augmented SAC	2012.6±835.0	3028.4±383.2	2100.0±752.7	43.4±3.8	1000.0±0.0	-4.05±0.48
BPQL (ours)	6334.6±245.3	4551.9±759.4	3336.0±200.3	93.0±31.8	983.4±33.1	-3.81±0.51
9	Normal SAC	-294.0±46.7	26.4±8.5	69.0±12.5	26.3±3.6	16.1±5.9	-37.36±3.42
Delayed-SAC	1764.3±203.3	2.6±6.4	513.7±642.2	77.6±34.1	505.8±333.2	-4.01±0.51
Augmented SAC	1297.2±265.9	1562.9±1075.9	1497.8±747.7	38.3±4.0	1000.0±0.0	-4.39±0.54
BPQL (ours)	5887.5±270.5	4104.3±428.7	2993.4±566.7	93.5±34.6	985.6±28.6	-3.86±0.50
- Augmented approach: Augmented SAC
- Model-based approach: Delayed-SAC
Normal SAC
In the augmented approach, we solve the control problem by adopting an augmented state space, which facilitates treating the CDMDP as a regular MDP. This is a popular and widely used method for an agent learning in a delayed environment [34, 27]. The policy and critic in the augmented state approach are trained by minimizing the objectives  $\bar{J}_{\bar{Q}}(\theta)$  and  $\bar{J}_{\bar{\pi}}(\phi)$ :
$$
\begin{array}{l} \bar {J} _ {\bar {Q}} (\theta) = \mathbb {E} _ {(\bar {s} _ {t}, a _ {t}, r _ {t}, \bar {s} _ {t + 1}) \sim \mathcal {D}} [ \frac {1}{2} (\bar {Q} _ {\theta} ^ {\bar {\pi}} (\bar {s} _ {t}, a _ {t}) - (r _ {t} + \gamma \mathbb {E} _ {a _ {t + 1} \sim \bar {\pi}} [ \bar {Q} _ {\theta} ^ {\operatorname {t a r}} (\bar {s} _ {t + 1}, a _ {t + 1}) ] \\ \left. - \alpha \log \bar {\pi} _ {\phi} (a _ {t + 1} | \bar {s} _ {t + 1}) ])\right) ^ {2} ], \\ \end{array}
$$
$$
\bar {J} _ {\bar {\pi}} (\phi) = \mathbb {E} _ {(\bar {s} _ {t}) \sim \mathcal {D}} [ \mathbb {E} _ {a _ {t} \sim \bar {\pi} _ {\phi}} [ \alpha \log \bar {\pi} _ {\phi} (a _ {t} | \bar {s} _ {t}) - \bar {Q} _ {\theta} ^ {\bar {\pi}} (\bar {s} _ {t}, a _ {t}) ]. \tag {27}
$$
We named this augmented-based approach Augmented SAC.
In the model-based approach, we use the delayed-Q algorithm [10], which uses a forward model to estimate the timely proper state. In this approach, the agent takes an action based on the predicted current state from the dynamic model using the last observed state and action history, i.e.,  $(s_{t - d},a_{t - d},a_{t - d + 1},\dots ,a_{t - 1})$ . The model repeats the one-step transition prediction  $d$  times to
Figure 3: (a) Result of tests on noisy version of InvertedPendulum-v2. The noises added to the actuator are drawn from a normal distribution with a mean of zero and standard deviations of 0.1 ((a)-left) and 0.15 ((a)-right), respectively. The range of action for the actuator is [-3, 3]. (b) The performance curves of BPQL and Augmented SAC with different hidden sizes  $\in\{(256,256),(512,512),(1024,1024),(256,256,256)\}$  of  $Q$ -network.
predict the current state. In the learning stage, we train the critic and policy using true transition tuples, i.e.,  $(s_t, a_t, r_t, s_{t+1})$ . The original study used double deep  $Q$ -networks (DDQN) [33] as the base learning algorithm. However, because DDQN is only applicable to discrete action spaces, we have used SAC for our base learning algorithm, so named this model-based approach Delayed-SAC.
Lastly, in the Normal SAC approach, the agent naively uses delayed feedback for training without any modification.
5.1 Performance Comparison
We tested the algorithms on several tasks using the MuJoCo benchmark [31] and evaluated their performances in environments with different numbers of delayed timesteps. Figure 2 shows that the augmented and model-based approaches are inappropriate for environments in which the delayed timestep is large, whereas the proposed BPQL algorithm exhibits significantly better performance in a long-delayed environment. Table 1 lists the overall experimental results, confirming that BPQL outperforms the conventional approaches by a wide margin in environments ranging from a short  $(d = 3)$  to a long delay  $(d = 9)$ .
5.2 Stochastic Environment
We evaluated BPQL and other baselines on the noisy version of the InvertedPendulum-v2 environment, where Gaussian noises were added to the actuator such that the acting becomes stochastic for the same input for the actuator. The added noises to the actuator are sampled from normal distribution with zero mean and standard deviations of 0.1 and 0.15. In this stochastic environment, BQPL has also shown better performance than conventional algorithms, and the difference gap in performance increased as the level of stochasticity grew.
5.3 Performance comparison of augmented approach with various capacity of  $Q$ -network
We conducted additional ablation study to investigate whether the size of  $Q$ -network was too small to learn and extract important features from the augmented state. We tested the performance of Augmented SAC with different hidden size of the  $Q$ -network, including (256,256), (512,512), (1024,1024), and (256,256,256). The results of this additional experiment show that the size of the  $Q$ -network does not significantly affect the performance of the augmented SAC as shown in Figure 3-(b). This confirms that addressing the state space explosion problem is a crucial factor in training the agent in a delayed environment.
In this study, we proposed a novel model-free algorithm BPQL for a constant delayed environment. BPQL evaluates the beta  $Q$ -function based on the original state-space size rather than evaluating the
See Appendix C for the details of these environments.
augmented  $Q$ -function, which helps the parameterized  $Q$ -function learn more stably and converge faster. In regards to the MuJoCo benchmark for continuous control tasks, BPQL achieves significantly better performance than the augmented state approach, which is a popular and widely used algorithm for delayed environments. Our results show that BPQL provides a promising avenue for handling delayed environments, such as real-world robot control environments where sensing and actuator delay exist, or a communication system with a narrow bandwidth.
As BPQL cannot be applied to a random delayed environment, it would be meaningful to extend our work to an environment with randomly delayed feedback. Ensemble learning of the beta  $Q$ -functions and augmented state-based policies that cover the entire delay range could be one possible approach. Furthermore, real-world applications using the proposed algorithm is an exciting direction for future work.
This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019-0-01906, Artificial Intelligence Graduate School Program (POSTECH))
[1] Mridul Agarwal and Vaneet Aggarwal. Blind decision making: Reinforcement learning with delayed observations. Pattern Recognition Letters, 150:176-182, 2021. doi: 10.1016/j.patrec.2021.06.022.
[2] Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine, 34(6):26-38, 2017. doi: 10.1109/MSP.2017.2743240.
[3] Richard Bellman. A markovian decision process. Journal of mathematics and mechanics, pages 679-684, 1957.
[4] Dimitri P Bertsekas. Dynamic Programming and Optimal Control, 2nd Edn, Vols. 1 and 2. Athena Scientific, Belmont, MA, 2001.
[5] Dimitri P Bertsekas. Approximate policy iteration: A survey and some new methods. Journal of Control Theory and Applications, 9(3):310-335, 2011.
[6] Yann Bouteiller, Simon Ramstedt, Giovanni Beltrame, Christopher Pal, and Jonathan Binas. Reinforcement learning with random delays. In Proceedings of the Ninth International Conference on Learning Representations, Online, 2021.
[7] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.
[8] Baiming Chen, Mengdi Xu, Liang Li, and Ding Zhao. Delay-aware model-based reinforcement learning for continuous control. Neurocomputing, 450:119-128, 2021. doi: 10.1016/j.neucom.2021.04.015.
[9] Petros Christodoulou. Soft actor-critic for discrete action settings. arXiv preprint arXiv:1910.07207, 2019.
[10] Esther Derman, Gal Dalal, and Shie Mannor. Acting in delayed environments with nonstationary Markov policies. arXiv preprint arXiv:2101.11992, 2021.
[11] Vlad Firoiu, Tina Ju, and Josh Tenenbaum. At human speed: Deep reinforcement learning with action delay. arXiv preprint arXiv:1810.07286, 2018.
[12] Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Proceedings of the 35th International Conference on Machine Learning, pages 1587-1596, Stockholm, Sweden, 2018. PMLR.
[28] John Schulman, Sergey Levine, Philipp Moritz, Michael Jordan, and Pieter Abbeel. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning, pages 1889-1897, Lille, France, 2015. JMLR.
[29] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
[30] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of Go without human knowledge. Nature, 550(7676):354-359, 2017. doi: 10.1038/nature24270.
[31] Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033, Vilamoura-Algarve, Portugal, 2012. IEEE. doi: 10.1109/IROS.2012.6386109.
[32] John N Tsitsiklis and Benjamin Van Roy. Analysis of temporal-difference learning with function approximation. Advances in Neural Information Processing Systems, 9:1075-1081, 1996.
[33] Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double Q-learning. In Proceedings of the AAAI conference on Artificial intelligence, volume 30(1), pages 2094-2100, Phoenix, AZ, 2016.
[34] Thomas J Walsh, Ali Nouri, Lihong Li, and Michael L Littman. Learning and planning in environments with delayed feedback. Autonomous Agents and Multi-Agent Systems, 18(1): 83-105, 2009. doi: 10.1007/s10458-008-9056-7.
A Architecture and Pseudo Code of BPQL
Figure 4: BPQL architecture: The agent selects an action using an augmented state, which is obtained from the temporary buffer, as an input of the policy. At every time  $t > 2d$ , a transition tuple  $\bar{s}_{t - d}, s_{t - d}, a_{t - d}, r_{t - d}, \bar{s}_{t + 1 - d}, s_{t + 1 - d}$  is stored in the replay memory, and the policy and beta critic are trained by minimizing  $J_{\bar{\pi}}$  and  $J_{Q_{\beta}}$ , respectively.
Algorithm 1 Belief-Projection-Based  $Q$ -learning (BPQL)
1: initialize a policy  $\bar{\pi}_{\phi}(a|\bar{s})$  , beta critics  $Q_{\theta ,\beta}(s,a)$  , target beta critics  $Q_{\theta ,\beta}^{\mathrm{tar}}(s,a)$  , an empty replay memory  $\mathcal{D}$  , a temporary buffer  $\mathcal{B}$  , delayed timesteps  $d$  , learning rate for the beta critic  $\lambda_{Q_{\beta}}$  learning rate for the policy  $\lambda_{\bar{\pi}}$  , and soft update ratio  $\tau$    
2: for episode  $= 1$  to  $E$  do   
3: for  $t = 1$  to  $H$  do   
4: if  $t <   d$  then   
5: executes action  $a_{t}$  randomly or No-Op' action   
6: put  $a_{t}$  to  $\mathcal{B}$    
7: else if  $t = d$  then   
8: choose action  $a_{d}$  randomly or No-Op' action   
9:  $s_1\gets \mathrm{Env}(a_d)$    
10: put  $s_1$  and  $a_{d}$  to  $\mathcal{B}$    
11: else   
12: get  $s_{t - d},a_{t - d},\ldots ,a_{t - 1}$  from  $\mathcal{B}$    
13:  $\bar{s}_t\gets (s_{t - d},a_{t - d},\dots ,a_{t - 1})$    
14:  $a_t\gets \pi_\phi (\bar{s}_t)$    
15:  $s_{t + 1 - d},r_{t - d}\gets \mathrm{Env}(a_t)$    
16: put  $s_{t + 1 - d}$  and  $a_{t}$  to  $\mathcal{B}$    
17: if  $t > 2d$  then   
18: get  $s_{t - 2d},s_{t - 2d + 1},s_{t - d},a_{t - 2d},\ldots ,a_{t - d}$  from  $\mathcal{B}$    
19:  $\bar{s}_{t - d}\gets (s_{t - 2d},a_{t - 2d},\ldots ,a_{t - d - 1})$    
20:  $\bar{s}_{t - d + 1}\gets (s_{t - 2d + 1},a_{t - 2d + 1},\ldots ,a_{t - d})$    
21: store  $(\bar{s}_{t - d},s_{t - d},a_{t - d},r_{t - d},\bar{s}_{t + 1 - d},s_{t + 1 - d})$  in  $\mathcal{D}$    
22: pop  $s_{t - 2d},a_{t - 2d}$  from  $\mathcal{B}$    
23: end if   
24: end if   
25: end for   
26: for each gradient step do   
27:  $\omega \leftarrow \omega -\lambda_{Q_\beta}\nabla_\theta J_{Q_\beta}(\omega)$    
28:  $\phi \leftarrow \phi -\lambda_{\overline{\pi}}\nabla_{\phi}J_{\overline{\pi}}(\phi)$    
29:  $\bar{\omega}\leftarrow \tau \omega +(1 - \tau)\bar{\omega}$    
30: end for   
31: end for
B Proof of Proposition 3.2
Proposition 3.2 Let the projection operator on the column-space of the belief matrix  $\mathbf{B}$  with respect to the weighted Euclidean norm  $||\cdot ||_{\mathbf{W}}$  be  $\Pi_{\mathbf{W}}$ , where  $\mathbf{W} = [\rho^{\bar{\pi}}(\bar{s}_1),\rho^{\bar{\pi}}(\bar{s}_2),\dots,\rho^{\bar{\pi}}(\bar{s}_j)]$  is a steady-augmented state probability vector, and the Markov chain be irreducible  $i,e$ ,  $\rho^{\bar{\pi}}(\bar{s}_k) > 0$  for all  $k\in \{1,2,\ldots ,j\}$ . Then the combined operator  $\Pi_{\mathbf{W}}\bar{T}^{\bar{\pi}}$  is  $\gamma$ -contraction with respect to  $||\cdot ||_{\mathbf{W}}$ .
Proof. Let  $\mathbf{V}_{\mathrm{proj.}}^{1}$  and  $\mathbf{V}_{\mathrm{proj.}}^{2}$  be an arbitrary vector in  $\mathbb{R}^j$ ,  $\bar{P}^{\bar{\pi}}(\bar{s}'|\bar{s})$  be a transition probability of  $\bar{s} \to \bar{s}'$  when the agent follows the policy  $\bar{\pi}$ , and  $V_{k}^{1,\mathrm{proj.}}$ ,  $V_{k}^{2,\mathrm{proj.}}$  be the k-th element of the  $\mathbf{V}_{\mathrm{proj.}}^{1}$  and  $\mathbf{V}_{\mathrm{proj.}}^{2}$  respectively.