SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jangwon Kim
jangwonkim@postech.ac.kr
Hangyeol Kim
hangyeol.kim@koreaaero.com
Jiwook Kang2
jiwook.kang@koreaaero.com
Jongchan Baek<sup>3</sup>
jcbaek@etri.re.kr
Soohee Han
soohee.han@postech.ac.kr
We present a novel actor-critic algorithm for an environment with delayed feedback, which addresses the state-space explosion problem of conventional approaches. Conventional approaches use an augmented state constructed from the last observed state and actions executed since visiting the last observed state. Using the augmented state space, the correct Markov decision process for delayed environments can be constructed; however, this causes the state space to explode as the number of delayed timesteps increases, leading to slow convergence. Our proposed algorithm, called Belief-Projection-Based  $Q$ -learning (BPQL), addresses the state-space explosion problem by evaluating the values of the critic for which the input state size is equal to the original state-space size rather than that of the augmented one. We compare BPQL to traditional approaches in continuous control tasks and demonstrate that it significantly outperforms other algorithms in terms of asymptotic performance and sample efficiency. We also show that BPQL solves long-delayed environments, which conventional approaches are unable to do.
Deep reinforcement learning (RL) methods have been successfully applied to diverse domains, such as decision-making tasks and robotic control problems [23, 2]. The deep RL framework has shown its potential by mastering the highly complex board game Go and defeating the professional Go player Lee [30]. It has also demonstrated superhuman performance, even in games where a perfect model is not provided, such as Atari video games [24]. Furthermore, modern deep RL algorithms have achieved significant performance improvements in continuous control domains, such as robotic locomotion [28, 29, 14]. Recently, many attempts have been made to apply RL-based control in not only simulations but also the real-world domain [13, 16, 22, 26]. Remarkable examples include applying the RL method to a quadrupedal robot to learn walking gaits and plan the motion of robotic manipulation in the real world [14, 17].
Despite recent progress in RL methods, adapting RL algorithms to the real world remains challenging for many reasons. Delayed feedback from the environment is a particular challenge. For example, latency may occur when the controller attempts to communicate with an agent if the agent is located far from the controller or if a large quantity of data, such as high-resolution images, must be transmitted. Furthermore, hardware issues can cause sensing or actuator delays. This delayed feedback may hinder achieving the control objectives.
<sup>1</sup>Computational Control Engineering Lab., Pohang University of Science and Technology. <sup>2</sup>Korea Aerospace Industries, Ltd. <sup>3</sup>Electronics and Telecommunications Research Institute.
Therefore, controlling signal delay is essential for applying RL algorithms to real-world domains. In this study, we propose a novel approach called belief projection-based  $Q$ -learning (BPQL) to overcome a constant delayed environment in which feedback is delayed by fixed timesteps. BPQL addresses the state-space explosion problem, which commonly occurs in conventional approaches, and achieves better performance than conventional approaches in terms of asymptotic performance and sample efficiency.
A Markov decision process (MDP) is defined as a 5-tuple  $(\mathcal{X},\mathcal{A},R,P,\gamma)$ , where  $\mathcal{X}$  is the state space,  $\mathcal{A}$  is the action space,  $R:\mathcal{X}\times \mathcal{A}\mapsto \mathbb{R}$  is the reward function,  $P:\mathcal{X}\times \mathcal{A}\times \mathcal{X}\mapsto [0,1]$  is the transition kernel, and  $\gamma \in (0,1)$  is a discount factor. The policy  $\pi (\cdot |s)$  maps the state-to-action distribution. In the standard RL framework, the agent chooses an action to maximize the discounted cumulative rewards by interacting with an environment defined as an MDP.
Let the distribution of the initial state be  $\rho_0$ . Then, the expected discounted cumulative reward from policy  $\pi$  is given as:
$$
\eta (\pi) = \mathbb {E} [ \sum_ {t = 0} ^ {\infty} \gamma^ {t} R (s _ {t}, a _ {t}) ], \tag {1}
$$
where initial state  $s_0\sim \rho_0,a_t\sim \pi (\cdot |s_t)$  , and  $s_{t + 1}\sim P(\cdot |s_t,a_t)$
From the standard definitions, the value and  $Q$ -value functions are obtained as:
$$
V ^ {\pi} \left(s _ {t}\right) = \mathbb {E} \left[ \sum_ {k = 0} ^ {\infty} \gamma^ {k} R \left(s _ {t + k}, a _ {t + k}\right) \mid s _ {t} \right] \tag {2}
$$
$$
Q ^ {\pi} \left(s _ {t}, a _ {t}\right) = \mathbb {E} \left[ \sum_ {k = 0} ^ {\infty} \gamma^ {k} R \left(s _ {t + k}, a _ {t + k}\right) \mid s _ {t}, a _ {t} \right], \tag {3}
$$
where  $\forall k\geq 0,a_{t + k}\sim \pi (\cdot |s_{t + k})$  and  $s_{t + k + 1}\sim P(\cdot |s_{t + k},a_{t + k})$
Both value functions satisfy the Bellman equation [3]:
$$
V ^ {\pi} \left(s _ {t}\right) = \mathbb {E} _ {a _ {t} \sim \pi \left(\cdot \mid s _ {t}\right)} \left[ R \left(s _ {t}, a _ {t}\right) + \gamma \mathbb {E} _ {s _ {t + 1} \sim P \left(\cdot \mid s _ {t}, a _ {t}\right)} \left[ V \left(s _ {t + 1}\right) \right] \right] \tag {4}
$$
$$
Q ^ {\pi} \left(s _ {t}, a _ {t}\right) = R \left(s _ {t}, a _ {t}\right) + \gamma \mathbb {E} _ {s _ {t + 1} \sim P (\cdot | s _ {t}, a _ {t})} [ V (s _ {t + 1}) ]. \tag {5}
$$
In the standard RL framework, we assume that the reward and next state only depend on the immediately previous state and action based on the Markov assumption. However, in an environment where the observation is delayed, the next state does not depend on the immediately previous state but on an older state and action history. This delay property forces the environment to be a partially observable MDP, not an MDP.
In this study, we consider the observation-delayed environment but not the action-delayed environment. However, observation and action delays are intrinsically linked [18, 34], which allows us to control the action-delay problem in the same manner as the observation-delay problem.
2.2 Constant Observation-Delayed Environment
In a delayed feedback environment, the agent receives time-delayed feedback from the environment, which makes it difficult to choose a timely and correct action at each timestep. In conventional control theory, signal delay is often handled by augmenting the state as a combination of the last observed state and history of actions for the delayed timesteps [20, 25].
This augmented state can also be used in RL frameworks. Considering a constant delayed feedback system, an environment with delayed feedback can be formalized by a constant delayed MDP (CDMDP) [34]. The CDMDP is defined as a 6-tuple  $(\mathcal{X},\mathcal{A},R,P,\gamma ,d)$ , where  $d$  is the number of timesteps between the execution of an action and receipt of feedback from that action (i.e., delayed timesteps).
Figure 1: In a constant delayed environment, the agent receives delayed state  $s_{t+1-d}$  and reward  $r_{t-d}$  after executing action  $a_t$ . These values are the actual feedback from previous action  $a_{t-d}$  and state  $s_{t-d}$ , where  $d$  is the number of delayed timesteps.
The CDMDP is known to be reducible to an MDP  $(\mathcal{S},\mathcal{A},\bar{\mathcal{R}},\bar{P},\gamma)$ , where  $\mathcal{S} = \mathcal{X}\times \mathcal{A}^d$  is an augmented state space,  $\bar{\mathcal{R}}:\mathcal{S}\times \mathcal{A}\mapsto \mathbb{R}$  is the reward mapping, and  $\bar{P}:\mathcal{S}\times \mathcal{A}\times \mathcal{S}\mapsto [0,1]$  is the transition kernel [18, 34]. This makes it possible to treat the CDMDP as a regular MDP. From this perspective, some model-free approaches, referred to as augmented state approaches, have been proposed [4, 34, 27, 6]. These use the augmented state, which is constructed by concatenating the last observed state and previous actions since visiting the observed state. However, the augmented state space grows exponentially as the number of delayed timesteps  $d$  increases. The curse of dimensionality makes the augmented state approach tractable for environments with only a relatively short delay.
Model-based approaches have also been proposed for handling delayed environments [34, 15, 11, 8, 1]. In these approaches, the agent predicts the current state by simulating the environment model and selects an action based on the predicted state. Increasing the accuracy of the transition model is key to this approach. However, in a complex, stochastic environment with a long time delay, the rapid growth of model error increases the difficulty of training the agent.
To the best of our knowledge, the closest work to ours is the study [1] that proposes a model-based algorithm expectation maximization  $Q$ -learning (EMQL). This algorithm, however, can only be used in discrete states and action spaces, whereas our proposed algorithm can be used in continuous state-action spaces without constraints. Additionally, although EMQL can suffer from dynamic model errors, our algorithm avoids this because it is a completely model-free approach.
As aforementioned, delayed environments have been mainly handled with two methods: the complete information method (e.g., the augmented state method) and the state estimation methods (e.g., the model-based method). As inherent drawbacks, the first one suffers from the curse of dimensionality and the second requires accurate model-based state estimation. Therefore, the research objective of this work is to resolve these two drawbacks together, i.e., find a model-free approach that is basically free of dimensionality curses. This is why the proposed algorithm was born.
3 Dynamics in Augmented State Space
In this section, we analyze the augmented state approach for the constant delayed environment. Figure 1 illustrates the interaction between an agent and a delayed environment. At each time  $t$ , the agent executes action  $a_{t}$  and receives the delayed state  $s_{t+1-d}$  and reward  $r_{t-d}$ , where  $d$  is the number of delayed timesteps. These feedback values are the response from action  $a_{t-d}$ . This misalignment of state, action, and reward makes it difficult to find an optimal policy.
We have observed that the CDMDP can be treated as a regular MDP by replacing the state space in the CDMDP with the augmented state space. The augmented state is constructed by concatenating the last observed state and previous actions and is denoted by  $\bar{s}$ .
More formally, in an environment delayed by  $d$  timesteps, the augmented state  $\bar{s}_t$  is constructed by concatenating the last observed state and previous  $d$  actions:
$$
\bar {s} _ {t} = \left(s _ {t - d}, a _ {t - d}, a _ {t - d + 1}, \dots , a _ {t - 1}\right). \tag {6}
$$
The policy that receives the augmented state as input is called the augmented state-based policy, denoted as  $\bar{\pi} (\cdot |\bar{s})$ .  $\rho^{\bar{\pi}}(\bar{s})$  is the agent's steady-augmented state distribution for augmented state-based policy  $\bar{\pi}$ . Furthermore, we refer to the value of the augmented state as the augmented state-based value, denoted as  $\bar{V}^{\bar{\pi}}(\bar{s}_t)$ . Similarly, the  $Q$ -value of the augmented state is called the augmented state-based  $Q$ -value, denoted as  $\bar{Q}^{\bar{\pi}}(\bar{s}_t,a_t)$ .
3.2 Augmented State Approach
Our objective is to train the agent in a constant delayed environment. In the augmented state approach, a new augmented state is constructed by augmenting the last observed state and previous actions to treat the CDMDP as a regular MDP.
We can apply a modified Bellman operator called the delay Bellman operator  $\bar{T}^{\bar{\pi}}$  to determine the augmented state-based value-function. The operator  $\bar{T}^{\bar{\pi}}$  is given as:
$$
\bar {\mathcal {T}} ^ {\bar {\pi}} \bar {V} (\bar {s} _ {t}) \mapsto \mathbb {E} _ {a _ {t} \sim \bar {\pi} (\cdot | \bar {s} _ {t})} \left[ \mathbb {E} _ {\mathbb {P} (s _ {t} | \bar {s} _ {t})} [ R (s _ {t}, a _ {t}) ] + \gamma \mathbb {E} _ {\bar {s} _ {t + 1} \sim \bar {P} (\cdot | \bar {s} _ {t}, a _ {t})} [ \bar {V} (\bar {s} _ {t + 1}) ] \right], \forall t > d, \tag {7}
$$
where  $d$  denotes the number of delayed timesteps in the environment. The expected reward is used in the delay Bellman operator because we assume that all feedback is delayed in a constant delayed environment, including the reward, which implies that the exact reward  $R(s_{t},a_{t})$  cannot be determined at time  $t$ .
The augmented state-based values can be computed by repeatedly applying the delay Bellman operator. After computing the augmented state-based values, the policy is updated to increase these values for all augmented states in the policy-improvement stage.
Although the augmented state-based values can be computed using the delay Bellman operator, the state-space explosion issue follows; the augmented state space grows exponentially as the number of delayed timesteps  $d$  increases because the augmented state space  $\mathcal{S}$  in the reduced MDP is a Cartesian product of the original state space and  $d$  action space (i.e.,  $\mathcal{S} = \mathcal{X} \times \mathcal{A}^d$ ). This indicates that the larger the state space, the larger the set of samples required to compute the value of the entire set of states. This makes the augmented state approach impractical for a long-delayed environment.
3.3 Alternative Method to Represent the Augmented State-Based Value
Because calculating the value of the augmented state space by applying the delay Bellman operator suffers from the state-space explosion issue, we propose an alternative method for representing the augmented value.
The distribution of the probability of visiting the true current state  $s_t$  (which has not yet been observed) depends on the augmented state  $\bar{s}_t$ . If two successive transition probabilities  $\mathbb{P}(s_t|\bar{s}_t^1)$  and  $\mathbb{P}(s_t|\bar{s}_t^2)$  are similar, we can assume that the augmented states  $\bar{s}_t^1$  and  $\bar{s}_t^2$  have similar representative meanings. Based on this assumption, we define the belief projection of the augmented values as follows.
Definition 3.1. For a policy  $\bar{\pi}$ , state space  $\mathcal{X} = \{s_1, s_2, \dots, s_i\}$  and augmented state space  $\mathcal{S} = \{\bar{s}_1, \bar{s}_2, \dots, \bar{s}_j\}$ , let  $\mathbf{B}$ ,  $\bar{\mathbf{V}}$  and  $\mathbf{D}$  be:
$$
\mathbf {B} = \left( \begin{array}{c c c c} \mathbb {P} (s _ {1} | \bar {s} _ {1}) & \mathbb {P} (s _ {2} | \bar {s} _ {1}) & \dots & \mathbb {P} (s _ {i} | \bar {s} _ {1}) \\ \mathbb {P} (s _ {1} | \bar {s} _ {2}) & \mathbb {P} (s _ {2} | \bar {s} _ {2}) & \dots & \mathbb {P} (s _ {i} | \bar {s} _ {2}) \\ \vdots & \vdots & \ddots & \vdots \\ \mathbb {P} (s _ {1} | \bar {s} _ {j}) & \mathbb {P} (s _ {2} | \bar {s} _ {j}) & \dots & \mathbb {P} (s _ {i} | \bar {s} _ {j}) \end{array} \right), \bar {\mathbf {V}} = \left( \begin{array}{c} \bar {V} ^ {\bar {\pi}} (\bar {s} _ {1}) \\ \vdots \\ \bar {V} ^ {\bar {\pi}} (\bar {s} _ {j}) \end{array} \right), \mathbf {D} = \mathrm {d i a g} (\mathbf {W}),
$$
where  $\mathbf{W} = [\rho^{\bar{\pi}}(\bar{s}_1),\rho^{\bar{\pi}}(\bar{s}_2),\dots,\rho^{\bar{\pi}}(\bar{s}_j)]$  is steady-augmented state probability vector when the agent follows the policy  $\bar{\pi}$ . Then, the projected values  $\mathbf{V}_{\mathrm{proj.}}$  are defined as:
$$
\mathbf {V} _ {\text {p r o j .}} = \Pi_ {\mathbf {w}} \bar {\mathbf {V}}, \tag {8}
$$
where the projection operator  $\Pi_{\mathbf{w}}$  is  $\mathbf{B}(\mathbf{B}^{\top}\mathbf{D}\mathbf{B})^{-1}\mathbf{B}^{\top}\mathbf{D}$
We refer to matrix  $\mathbf{B}$  as the belief matrix. If the matrix  $\mathbf{B}^{\top}\mathbf{DB}$  is not invertible, then the inverse operator can be replaced with the Moore-Penrose pseudoinverse operator. We refer to the outcome
from the projection operator  $\Pi_{\mathbf{W}}$  as the belief projection. The belief projection operator  $\Pi_{\mathbf{W}}$  projects a vector in  $\mathbb{R}^j$  onto the column space of the belief matrix  $\mathbf{B}$  with respect to the weighted Euclidean norm  $||\cdot ||_{\mathbf{W}}$ .
Because the belief projection lies in the column space of the belief matrix, we can use the vector  $\mathbf{V}_{\beta}$  to decompose the projected values as follows:
$$
\mathbf {V} _ {\text {p r o j .}} = \mathbf {B} \mathbf {V} _ {\beta} = \mathbf {B} \left( \begin{array}{c} V _ {\beta} ^ {\bar {\pi}} \left(s _ {1}\right) \\ \vdots \\ V _ {\beta} ^ {\bar {\pi}} \left(s _ {i}\right) \end{array} \right). \tag {9}
$$
We refer to the elements of  $\mathbf{V}_{\beta}$  as the beta values. The vector  $\mathbf{V}_{\beta}$  can also be defined as:
$$
\mathbf {V} _ {\beta} = \underset {\mathbf {V} \in \mathbb {R} ^ {i}} {\arg \min } \| \mathbf {B V} - \bar {\mathbf {V}} \| _ {2, \mathbf {W}}. \tag {10}
$$
From Equation (9), the augmented state-based value can be decomposed into the sum of the expectation of the beta values over the original states and the residual:
$$
\bar {V} ^ {\bar {\pi}} \left(\bar {s} _ {t}\right) = \mathbb {E} _ {\mathbb {P} \left(s _ {t} \mid \bar {s} _ {t}\right)} \left[ V _ {\beta} ^ {\bar {\pi}} (s _ {t}) \right] + \Delta_ {\text {r e s i d u a l}} ^ {\bar {\pi}} (\bar {s} _ {t}), \tag {11}
$$
where  $t > d$ . Similarly, we can extend Equation (9) to the belief projection for the augmented  $Q$ -values (projected  $Q$ -values); the beta  $Q$ -values for a given action  $a$  satisfy the following:
$$
\bar {Q} ^ {\bar {\pi}} \left(\bar {s} _ {t}, a\right) = \mathbb {E} _ {\mathbb {P} \left(s _ {t} \mid \bar {s} _ {t}\right)} \left[ Q _ {\beta} ^ {\bar {\pi}} \left(s _ {t}, a\right) \right] + \delta_ {\text {r e s i d u a l}} ^ {\bar {\pi}} \left(\bar {s} _ {t}, a\right) \tag {12}
$$
3.4 Linear Function Approximation
The augmented state approach is impractical for a long-delayed environment because its state-space size grows exponentially as the number of delayed timesteps increases. In a long-delayed environment, this indicates that directly calculating the augmented state-based  $Q$ -values by applying the delay Bellman operator requires a significantly large set of samples.
To avoid this issue, instead of calculating the augmented state-based values  $\bar{\mathbf{V}}$  directly, we estimate the belief projection  $\Pi_{\mathbf{W}}\bar{\mathbf{V}}$ . Let  $\bar{T}^{\bar{\pi}}$  be a matrix form of the delay Bellman operator  $\bar{T}^{\bar{\pi}}$ , i.e.,  $\bar{T}^{\bar{\pi}}\bar{\mathbf{V}} := \bar{\mathbf{R}} + \gamma \bar{\mathbf{P}}\bar{\mathbf{V}}$ , where  $\bar{\mathbf{R}} = [\mathbb{E}_{\mathbb{P}(s_t|\bar{s}_1),a\sim \bar{\pi} (\cdot |\bar{s}_1)}[R(s,a)],\dots ,\mathbb{E}_{\mathbb{P}(s_t|\bar{s}_j),a\sim \bar{\pi} (\cdot |\bar{s}_j)}[R(s,a)]]^{\top}$  is a vector consisting of expected rewards and  $\bar{\mathbf{P}}$  is the transition matrix. In this setting, focusing on finding a solution  $\mathbf{V}_{\beta}^{*} = \arg \min_{\mathbf{V}\in \mathbb{R}^{i}}||\mathbf{BV} - \Pi_{\mathbf{W}}\bar{T}^{\bar{\pi}}(\mathbf{BV})||_{\mathbf{W}}$  could be achieved very efficiently, especially when  $|\mathcal{S}|\gg |\mathcal{X}|$ .
Note that the values are approximated as linear combinations of the belief matrix and the beta-values. In other words, the belief projection can be seen as a linear function approximator where the feature vector for an augmented state  $\bar{s}_k\in S$  is  $[\mathbb{P}(s_1|\bar{s}_k),\mathbb{P}(s_2|\bar{s}_k),\dots ,\mathbb{P}(s_i|\bar{s}_k)]$  and the corresponding parameters are the beta values  $[V_{\beta}^{\bar{\pi}}(s_1),V_{\beta}^{\bar{\pi}}(s_2),\dots ,V_{\beta}^{\bar{\pi}}(s_i)]^{\top}$ . In that sense, we can guarantee the contraction of the combined operator  $\Pi_{\mathbf{W}}\bar{T}^{\bar{\pi}}$  by using the well-known contraction property of linear function approximation [32, 5].
Proposition 3.2. Let the projection operator onto the column-space of the belief matrix  $\mathbf{B}$  with respect to the weighted Euclidean norm  $||\cdot ||_{\mathbf{W}}$  be  $\Pi_{\mathbf{W}}$ , where  $\mathbf{W} = [\rho^{\bar{\pi}}(\bar{s}_1),\rho^{\bar{\pi}}(\bar{s}_2),\dots,\rho^{\bar{\pi}}(\bar{s}_j)]$  is a steady-augmented state probability vector, and the Markov chain be irreducible  $i,e$ ,  $\rho^{\bar{\pi}}(\bar{s}_k) > 0$  for all  $k\in \{1,2,\ldots ,j\}$ . Then the combined operator  $\Pi_{\mathbf{W}}\bar{T}^{\bar{\pi}}$  is  $\gamma$ -contraction with respect to  $||\cdot ||_{\mathbf{W}}$ .
Proof. See appendix B.
This contraction property of  $\Pi_{\mathbf{W}}\bar{T}^{\bar{\pi}}$  guarantees to find a fixed unique solution  $\hat{\mathbf{V}}_{\beta}^{*} = [\hat{V}_{\beta}^{\bar{\pi}}(s_1),\dots ,\hat{V}_{\beta}^{\bar{\pi}}(s_i)]$  by repeatedly applying the combined operator  $\Pi_{\mathbf{W}}\bar{T}^{\bar{\pi}}$ . This  $\hat{\mathbf{V}}_{\beta}^{*}$  can be used as an estimator for the true beta values  $\mathbf{V}_{\beta}$ , providing a direction for policy update. For example, a new policy  $\bar{\pi}_{\mathrm{new}}$  can be obtained by greedily choosing an action, i.e.,  $\bar{\pi}_{\mathrm{new}}(\bar{s}_t) = \arg \max_{a_t\in \mathcal{A}}\left(\mathbb{E}_{\mathbb{P}(s_t|\bar{s}_t)}[R(s_t,a_t)] + \mathbb{E}_{\bar{P} (\bar{s}_{t + 1}|\bar{s}_t,a_t)}\left[\mathbb{E}_{\mathbb{P}(s_{t + 1}|\bar{s}_{t + 1})}\left[\hat{V}_{\beta}^{\bar{\pi}_{\mathrm{old}}} (s_{t + 1})\right]\right]\right)$ , where the  $\bar{\pi}_{\mathrm{old}}$  is the policy before improved.
4 Actor-Critic Algorithm for Constant Delayed Environment
In the previous section, we presented an iterative method that computes the value function in the smaller state space created by belief projection, rather than in the larger augmented state space. However, despite the favorable convergence property of value function approximation through belief projection, explicitly calculating the belief matrix can be challenging, especially in cases where the augmented space is large or continuous. To address this challenge, we introduce a practical sampling-based reinforcement learning algorithm based on the theoretical insights from the previous section. In this algorithm, the agent learns the beta values without the explicit computation of the belief matrix.
First, we define the delay soft Bellman operator  $\bar{T}_{\mathrm{soft}}^{\bar{\pi}}$ , which is the soft Bellman operator [14] for the delayed environment setting:
$$
\bar {\mathcal {T}} _ {\text {s o f t}} ^ {\bar {\pi}} \bar {Q} ^ {\bar {\pi}} \left(\bar {s} _ {t}, a _ {t}\right) \mapsto \mathbb {E} _ {\mathbb {P} \left(s _ {t} \mid \bar {s} _ {t}\right)} \left[ R \left(s _ {t}, a _ {t}\right) \right] + \gamma \mathbb {E} _ {\bar {s} _ {t + 1} \sim \bar {P}, a _ {t + 1} \sim \bar {\pi}} \left[ \bar {Q} ^ {\bar {\pi}} \left(\bar {s} _ {t + 1}, a _ {t + 1}\right) - \alpha \log \bar {\pi} \left(a _ {t + 1} \mid \bar {s} _ {t + 1}\right) \right], \tag {13}
$$
where  $\alpha$  is the temperature parameter. We can compute the augmented state-based soft  $Q$ -values by repeatedly applying the delay soft Bellman operator.
In the policy improvement stage, the policy is updated towards the exponential of the evaluated augmented state-based soft  $Q$ -function. The improved policy  $\bar{\pi}_{\mathrm{new}}$  can be obtained as [14]:
$$
\bar {\pi} _ {\text {n e w}} = \arg \min  _ {\bar {\pi} ^ {\prime}} \left(\bar {\pi} ^ {\prime} (\cdot | \bar {s} _ {t}) \right\| \frac {\exp \left(\frac {1}{\alpha} \bar {Q} ^ {\bar {\pi} _ {\text {o l d}}} (\bar {s} _ {t} , \cdot)\right)}{\bar {Z} ^ {\bar {\pi} ^ {\text {o l d}}} (\bar {s} _ {t})}, \tag {14}
$$
where  $\bar{Z}^{\bar{\pi}_{\mathrm{old}}}(s_t)$  is a normalizing function. This updated policy  $\bar{\pi}_{\mathrm{new}}$  guarantees  $\bar{Q}^{\bar{\pi}_{\mathrm{new}}}(\bar{s}_t, a_t) \geq \bar{Q}^{\bar{\pi}_{\mathrm{old}}}(\bar{s}_t, a_t)$  for all  $(\bar{s}_t, a_t) \in S \times \mathcal{A}$ .
The beta  $Q$ -values are approximately computed by minimizing the average of weighted squared residual error i.e,  $\delta_{\mathrm{residual}}^{\bar{\pi}}$  in Equation (12) as: