SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Alessio Russo  
Division of Decision and Control Systems  
KTH Royal Institute of Technology  
Stockholm, SE
Alexandre Proutiere  
Division of Decision and Control Systems  
KTH Royal Institute of Technology  
Stockholm, SE
We study the problem of exploration in Reinforcement Learning and present a novel model-free solution. We adopt an information-theoretical viewpoint and start from the instance-specific lower bound of the number of samples that have to be collected to identify a nearly-optimal policy. Deriving this lower bound along with the optimal exploration strategy entails solving an intricate optimization problem and requires a model of the system. In turn, most existing sample optimal exploration algorithms rely on estimating the model. We derive an approximation of the instance-specific lower bound that only involves quantities that can be inferred using model-free approaches. Leveraging this approximation, we devise an ensemble-based model-free exploration strategy applicable to both tabular and continuous Markov decision processes. Numerical results demonstrate that our strategy is able to identify efficient policies faster than state-of-the-art exploration approaches.
Efficient exploration remains a major challenge for reinforcement learning (RL) algorithms. Over the last two decades, several exploration strategies have been proposed in the literature, often designed with the aim of minimizing regret. These include model-based approaches such as Posterior Sampling for RL [36](PSRL) and Upper Confidence Bounds for RL [4, 25, 2](UCRL), along with model-free UCB-like methods [19, 56]. Regret minimization is a relevant objective when one cares about the rewards accumulated during the learning phase. Nevertheless, an often more important objective is to devise strategies that explore the environment so as to learn efficient policies using the fewest number of samples [16]. Such an objective, referred to as Best Policy Identification (BPI), has been investigated in simplistic Multi-Armed Bandit problems [16, 21] and more recently in tabular MDPs [28, 29]. For these problems, tight instance-specific sample complexity lower bounds are known, as well as model-based algorithms approaching these limits. However, model-based approaches may be computationally expensive or infeasible to obtain. In this paper, we investigate whether we can adapt the design of these algorithms so that they become model-free and hence more practical.
Inspired by [28, 29], we adopt an information-theoretical approach, and design our algorithms starting from an instance-specific lower bound on the sample complexity of learning a nearly-optimal policy in a Markov decision process (MDP). This lower bound is the value of an optimization problem, referred to as the lower bound problem, whose solution dictates the optimal exploration strategy in an environment. Algorithms designed on this instance-specific lower bound, rather than minimax bounds, result in truly adaptive methods, capable of tailoring their exploration strategy according to the specific MDP's learning difficulty. Our method estimates the solution to the lower bound problem and employs it as our exploration strategy. However, we face two major challenges: (1) the
Code repository: https://github.com/rssalessio/ModelFreeActiveExplorationRL
lower bound problem is non-convex and often intractable; (2) this lower bound problem depends on the initially unknown MDP. In [29], the authors propose MDP-NAS, a model-based algorithm that explores according to the estimated MDP. They convexify the lower bound problem and explore according to the solution of the resulting simplified problem. However, this latter problem still has a complicated dependency on the MDP. Moreover, extending MDP-NAS to large MDPs is challenging since it requires an estimate of the model, and the capability to perform policy iteration. Additionally, MDP-NAS employs a forced exploration technique to ensure that the parametric uncertainty (the uncertainty about the true underlying MDP) diminishes over time — a method, as we argue later, that we believe not to be efficient in handling this uncertainty.
We propose an alternative way to approximate the lower bound problem, so that its solution can be learnt via a model-free approach. This solution depends only on the  $Q$ -function and the variance of the value function. Both quantities can advantageously be inferred using classical stochastic approximation methods. To handle the parametric uncertainty, we propose an ensemble-based method using a bootstrapping technique. This technique is inspired by posterior sampling and allows us to quantify the uncertainty when estimating the  $Q$ -function and the variance of the value function.
Our contributions are as follows: (1) we shed light on the role of the instance-specific quantities needed to drive exploration in uncertain MDPs; (2) we derive an alternate upper bound of the lower bound problem that in turn can be approximated using quantities that can be learned in a model-free manner. We then evaluate the quality of this approximation on various environments: (i) a random MDP, (ii) the Riverswim environment [47], and (iii) the Forked Riverswim environment (a novel environment with high sample complexity); (3) based on this approximation, we present Model Free Best Policy Identification (MF-BPI), a model-free exploration algorithm for tabular and continuous MDPs. For the tabular MDPs, we test the performance of MF-BPI on the Riverswim and the Forked Riverswim environments, and compare it to that of Q-UCB [19, 56], PSRL[36], and MDP-NAS[29]. For continuous state-spaces, we compare our algorithm to IDS[33] and BSP [39] (Boostrapped DQN with randomized prior value functions) and assess their performance on hard-exploration problems from the DeepMind BSuite [41] (the DeepSea and the Cartpole swingup problems).
The body of work related to exploration methods in RL problems is vast, and we mainly focus on online discounted MDPs (for the generative setting, refer to the analysis presented in [17, 28]). Exploration strategies in RL often draw inspiration from the approaches used in multi-armed bandit problems [26, 49], including  $\epsilon$ -greedy exploration, Boltzmann exploration [57, 49, 26, 1], or more advanced procedures, such as Upper-Confidence Bounds (UCB) methods [2, 3, 26] or Bayesian procedures [52, 58, 14, 44]. We first discuss tabular MDPs, and then extend the discussion to the case of RL with function approximation.
Exploration in tabular MDPs. Numerous algorithms have been proposed with the aim of matching the PAC sample complexity minimax lower bound  $\tilde{\Omega}\left(\frac{|S||A|}{\varepsilon^2(1 - \gamma)^3}\right)$  [25]. In the design of these algorithms, model-free approaches typically rely on a UCB-like exploration [2, 26], whereas model-based methods leverage estimates of the MDP to drive the exploration. Some well-known model-free algorithms are MEDIAN-PAC [42], DELAYED Q-LEARNING [48] and Q-UCB [56, 19]. Some notable model-based algorithms include: DEL [34], an algorithm that achieves asymptotically optimal instance-dependent regret; UCRL [25], an algorithm that uses extended value-iteration to compute an optimistic MDP; PSRL [36], that uses posterior sampling to sample an MDP. Other algorithms include MBIE [47], E3 [22], R-MAX [9, 20], and MORMAX [50]. Most of existing algorithms are designed towards regret minimization. Recently, however, there has been a growing interest towards exploration strategies with minimal sample complexity, see e.g. [60, 28]. In [28, 29], the authors showed that computing an exploration strategy with minimal sample complexity requires to solve a non-convex problem. To overcome this challenge, they derived a tractable approximation of the lower bound problem, whose solution provides an efficient exploration policy under the generative model [28] and the forward model [29]. This policy necessitates an estimate of the model, and includes a forced exploration phase (an  $\epsilon$ -soft policy to guarantee that all state-action pairs are visited infinitely often). In [51], the above procedure is extended to linear MDPs, but there again, computing an optimal exploration strategy remains challenging. On a side note, in [55], the authors provide an
alternative bound in the tabular case for episodic MDPs, and later extend it to linear MDPs [54]. The episodic setting is further explored in [53] for deterministic MDPs.
Exploration in Deep Reinforcement Learning (DRL). Exploration methods in DRL environments face several challenges, related to the fact that the state-action spaces are often continuous, and other issues related to training deep neural architectures [46]. The main issue in these large MDPs is that good exploration becomes extremely hard when either the reward is sparse/delayed or the observations contain distracting features [10, 59]. Numerous heuristics have been proposed to tackle these challenges, such as (1) adding an entropy term to the optimization problem to encourage the policy to be more randomized [31, 18] or (2) injecting noise in the observations/parameters [15, 43]. More generally, exploration techniques generally fall into two categories: uncertainty-based and intrinsic-motivation-based [59, 24]. Uncertainty-based methods decouple the uncertainty into parametric and aleatoric uncertainty. Parametric uncertainty [14, 32, 23, 59] quantifies the uncertainty in the parameters of the state-action value. This uncertainty vanishes as the agent explores and learns. The aleatoric uncertainty accounts for the inherent randomness of the environment and of the policy [32, 23, 59]. Various methods have been proposed to address the parametric uncertainty, including UCB-like mechanisms [11, 59], or TS-like (Thompson Sampling) techniques [38, 36, 5, 35, 37, 40]. However, computing a posterior of the  $Q$ -values is a difficult task. For instance, Bayesian DQN [5] extends Randomized Least-Squares Value Iteration (RLSVI) [38] by considering the features prior to the output layer of the deep- $Q$  network as a fixed feature vector, in order to recast the problem as a linear MDP. Non-parametric posterior sampling methods include Bootstrapped DQN (and Bootstrapped DQN with prior functions) [37, 39, 40], which maintains several independent  $Q$ -value functions and randomly samples one of them to explore the environment. Bootstrapped DQN was extended in various ways by integrating other techniques [6, 27]. For the sake of brevity, we refer the reader to the survey in [59] for an exhaustive list of algorithms. Most of these algorithms do not directly account for aleatoric uncertainty in the value function. This uncertainty is usually estimated using methods like Distributional RL [8, 13, 30]. Well-known exploration methods that account for both aleatoric and epistemic uncertainties include Double Uncertain Value Network (DUVN) [32] and Information Directed Sampling (IDS) [23, 33]. The former uses Bayesian dropout to measure the epistemic uncertainty, and the latter uses distributional RL [8] to estimate the variance of the returns. In addition, IDS uses bootstrapped DQN to estimate the parametric uncertainty in the form of a bound on the estimate of the suboptimality gaps. These uncertainties are then combined to compute an exploration strategy. Similarly, in [12], the authors propose UA-DQN, an approach that uses QR-DQN [13] to learn the parametric and aleatoric uncertainties from the quantile networks. Lastly, we refer the reader to [59, 45, 7] for the class of intrinsic-motivation-based methods.
Markov Decision Process. We consider an infinite-horizon discounted Markov Decision Process (MDP), defined by the tuple  $\phi = (S, A, P, q, \gamma, p_0)$ .  $S$  is the state space,  $A$  is the action space,  $P: S \times A \to \Delta(S)$  is the distribution over the next state given a state-action pair  $(s, a)$ ,  $q: S \times A \to \Delta([0,1])$  is the distribution of the collected reward (with support in  $[0,1]$ ),  $\gamma \in [0,1)$  is the discount factor and  $p_0$  is the distribution over the initial state.
Let  $\pi : S \to \Delta(A)$  be a stationary Markovian policy that maps a state to a distribution over actions, and denote by  $r(s, a) = \mathbb{E}_{r \sim q(|s, a|)}[r]$  the average reward collected when an action  $a$  is chosen in state  $s$ . We denote by  $V^{\pi}(s) = \mathbb{E}_{\phi}^{\pi}[\sum_{t \geq 0} \gamma^{t} r(s_{t}, a_{t}) | s_{0} = s]$  the discounted value of policy  $\pi$ . We denote by  $\pi^{\star}$  an optimal stationary policy: for any  $s \in S$ ,  $\pi^{\star}(s) \in \arg \max_{\pi} V^{\pi}(s)$  and define  $V^{\star}(s) = \max_{\pi} V^{\pi}(s)$ . For the sake of simplicity, we assume that the MDP has a unique optimal policy (we extend our results to more general MDPs in the appendix). We further define  $\Pi_{\varepsilon}^{\star}(\phi) = \{\pi : \|V^{\pi} - V^{\pi^{\star}}\|_{\infty} \leq \varepsilon\}$ , the set of  $\varepsilon$ -optimal policies in  $\phi$  for  $\varepsilon \geq 0$ . Finally, to avoid technicalities, we assume (as in [29]) that the MDP  $\phi$  is communicating (that is, for every pair of states  $(s, s')$ , there exists a deterministic policy  $\pi$  such that state  $s'$  is accessible from state  $s$  using  $\pi$ ).
We denote by  $Q^{\pi}(s,a)\coloneqq r(s,a) + \gamma \mathbb{E}_{s'\sim P(\cdot |s,a)}[V^{\pi}(s')]$  the  $Q$ -function of  $\pi$  in state  $(s,a)$ . We also define the sub-optimality gap of action  $a$  in state  $s$  to be  $\Delta (s,a)\coloneqq Q^{\star}(s,\pi^{\star}(s)) - Q^{\star}(s,a)$ , where  $Q^{\star}$  is the  $Q$ -function of  $\pi^{\star}$ , and let  $\Delta_{\min}:= \min_{s,a\neq \pi^{\star}(s)}\Delta (s,a)$  be the minimum gap in  $\phi$ . For some policy  $\pi$ , we define  $\mathrm{Var}_{sa}[V^{\pi}]\coloneqq \mathrm{Var}_{s'\sim P(\cdot |s,a)}[V^{\pi}(s')]$  to be the variance of the value function  $V^{\pi}$  in the next state after taking action  $a$  in state  $s$ . More generally, we define  $M_{sa}^{k}[V^{\pi}]\coloneqq$
$\mathbb{E}_{s^{\prime}\sim P(\cdot |s,a)}\left[\left(V^{\pi}(s^{\prime}) - \mathbb{E}_{\bar{s}\sim P(\cdot |s,a)}[V^{\pi}(\bar{s})]\right)^{2^{k}}\right]$  to be the  $2^{k}$ -th moment of the value function in the next state after taking action  $a$  in state  $s$ . We also let  $\mathrm{MD}_{sa}[V^{\pi}]\coloneqq \| V^{\pi} - \mathbb{E}_{s^{\prime}\sim P(\cdot |s,a)}[V^{\pi}]\|_{\infty}$  be the span of  $\phi$  under  $\pi$ , i.e., the maximum deviation from the mean of the next state value after taking action  $a$  in state  $s$ .
Best policy identification and sample complexity lower bounds. The MDP  $\phi$  is initially unknown, and we are interested in the scenario where the agent interacts sequentially with  $\phi$ . In each round  $t \in \mathbb{N}$ , the agent selects an action  $a_t$  and observes the next state and the reward  $(s_{t+1}, r_t) \colon s_{t+1} \sim P(\cdot | s_t, a_t)$  and  $r_t \sim q(\cdot | s_t, a_t)$ . The objective of the agent is to learn a policy in  $\Pi_{\varepsilon}^{\star}(\phi)$  (possibly  $\pi^{\star}$ ) as fast as possible. This objective is often formalized in a PAC framework where the learner has to stop interacting with the MDP when she can output an  $\varepsilon$ -optimal policy with probability at least  $1 - \delta$ . In this formalism, the learner strategy consists of (i) a sampling rule or exploration strategy; (ii) a stopping time  $\tau$ ; (iii) an estimated optimal policy  $\hat{\pi}$ . The strategy is called  $(\varepsilon, \delta)$ -PAC if it stops almost surely, and  $\mathbb{P}_{\phi}[\hat{\pi} \in \Pi_{\varepsilon}^{\star}(\phi)] \geq 1 - \delta$ . Interestingly, one may derive instance-specific lower bounds of the sample complexity  $\mathbb{E}_{\phi}[\tau]$  of any  $(\varepsilon, \delta)$ -PAC algorithm [28, 29], which involves computing an optimal allocation vector  $\omega_{\mathrm{opt}} \in \Delta(S \times A)$  (where  $\Delta(S \times A)$  is the set of distributions over  $S \times A$ ) that specifies the proportion of times an agent needs to sample each pair  $(s, a)$  to confidently identify the optimal policy:
$$
\lim  _ {\delta \rightarrow 0} \inf  _ {\mathrm {k l} (\delta , 1 - \delta)} \frac {\mathbb {E} _ {\phi} [ \tau ]}{\mathrm {k l} (\delta , 1 - \delta)} \geq T _ {\varepsilon} \left(\omega_ {\text {o p t}}\right) \text {w h e r e} T _ {\varepsilon} (\omega) ^ {- 1} := \inf  _ {\psi \in \operatorname {A l t} _ {\varepsilon} (\phi)} \mathbb {E} _ {(s, a) \sim \omega} \left[ \mathrm {K L} _ {\phi | \psi} (s, a) \right], \tag {1}
$$
and  $\omega_{\mathrm{opt}} = \arg \inf_{\omega \in \Omega (\phi)}T_{\varepsilon}(\omega)^{-1}$ . Here,  $\mathrm{Alt}_{\varepsilon}(\phi)$  is the set of confusing MDPs  $\psi$  such that the  $\varepsilon$ -optimal policies of  $\phi$  are not  $\varepsilon$ -optimal in  $\psi$ , i.e.,  $\mathrm{Alt}_{\varepsilon}(\phi) := \{\psi : \phi \ll \psi, \Pi_{\varepsilon}^{\star}(\phi) \cap \Pi_{\varepsilon}^{\star}(\psi) = \emptyset\}$ . In this definition, if the next state and reward distributions under  $\psi$  are  $P'(s,a)$  and  $q'(s,a)$ , we write  $\phi \ll \psi$  if for all  $(s,a)$  the distributions of the next state and of the rewards satisfy  $P(s,a) \ll P'(s,a)$  and  $q(s,a) \ll q'(s,a)$ . We further let  $\mathrm{KL}_{\phi|\psi}(s,a) := \mathrm{KL}(P(s,a), P'(s,a)) + \mathrm{KL}(q(s,a), q'(s,a))$ .  $\Omega(\phi)$  is the set of possible allocations; in the generative case it is  $\Delta(S \times A)$ , while with navigation constraints we have  $\Omega(\phi) := \{\omega \in \Delta(S \times A) : \omega(s) = \sum_{s',a'} P(s|s',a')\omega(s',a')\}, \forall s \in S\}$ , with  $\omega(s) := \sum_{a}\omega(s,a)$ . Finally,  $\mathrm{kl}(a,b)$  is the KL-divergence between two Bernoulli distributions of means  $a$  and  $b$ .
4 Towards Efficient Exploration Allocations
We aim to extend previous studies on best policy identification to online model-free exploration. In this section, we derive an approximation to the bound proposed in [28], involving quantities learnable via stochastic approximation, thereby enabling the use of model-free approaches.
The optimization problem (1) leading to instance-specific sample complexity lower bounds has an important interpretation [28, 29]. An allocation  $\omega_{\mathrm{opt}}$  corresponds to an exploration strategy with minimal sample complexity. To devise an efficient exploration strategy, one could then think of estimating the MDP  $\phi$ , and solving (1) for this estimated MDP to get an approximation of  $\omega_{\mathrm{opt}}$ . There are two important challenges towards applying this approach:
(i) Estimating the model can be difficult, especially for MDPs with large state and action spaces, and arguably, a model-free method would be preferable.
(ii) The lower bound problem (1) is, in general, non-convex [28, 29].
A simple way to circumvent issue (ii) involves deriving an upper bound of the value of the sample complexity lower bound problem (1). Specifically, one may derive an upper bound  $U(\omega)$  of  $T_{\varepsilon}(\omega)$  by convexifying the corresponding optimization problem. The exploration strategy can then be the  $\omega^{\star}$  that achieves the infimum of  $U(\omega)$ . This approach ensures that we identify an approximately optimal policy, at the cost of over-exploring at a rate corresponding to the gap  $U(\omega^{\star}) - T_{\varepsilon}(\omega_{\mathrm{opt}})$ . Note that using a lower bound of  $T_{\varepsilon}(\omega)$  would not guarantee the identification of an optimal policy, since we would explore "less" than required. The aforementioned approach was already used in [28] where the authors derive an explicit upper bound  $U_0(\omega)$  of  $T_0(\omega)$ . We also apply it, but derive an upper bound such that implementing the corresponding allocation  $\omega^{\star}$  can be done in a model-free manner (hence solving the first issue (i)).
4.1 Upper bounds on  $T_{\varepsilon}(\omega)$
The next theorem presents the upper bound derived in [28].
Theorem 4.1 ([28]). Consider a communicating MDP  $\phi$  with a unique optimal policy  $\pi^{\star}$ . For all vectors  $\omega \in \Delta(S \times A)$ ,
$$
T _ {0} (\omega) \leq U _ {0} (\omega) := \max  _ {(s, a): a \neq \pi^ {\star} (s)} \frac {H _ {0} (s , a)}{\omega (s , a)} + \max  _ {s} \frac {H _ {0} ^ {\star}}{\omega (s , \pi^ {\star} (s))}, \tag {2}
$$
$$
\left\{ \begin{array}{l l} H _ {0} (s, a) = \frac {2}{\Delta (s , a) ^ {2}} + \max \left(\frac {1 6 \mathrm {V a r} _ {s a} [ V ^ {\star} ]}{\Delta (s , a) ^ {2}}, \frac {6 \mathrm {M D} _ {s a} [ V ^ {\star} ] ^ {4 / 3}}{\Delta (s , a) ^ {\frac {4}{3}}}\right), \\ H _ {0} ^ {\star} = \frac {2}{\Delta_ {\min} ^ {2} (1 - \gamma) ^ {2}} + \min \left(\frac {2 7}{\Delta_ {\min} ^ {2} (1 - \gamma) ^ {3}}, \max \left(\frac {1 6 \max _ {s} \mathrm {V a r} _ {s \pi^ {\star} (s)} [ V ^ {\star} ]}{\Delta_ {\min} ^ {2} (1 - \gamma) ^ {2}}, \frac {6 \max _ {s} \mathrm {M D} _ {s \pi^ {\star} (s)} [ V ^ {\star} ] ^ {4 / 3}}{\Delta_ {\min} ^ {4 / 3} (1 - \gamma) ^ {4 / 3}}\right)\right). \end{array} \right.
$$
In the upper bound presented in this theorem, the following quantities characterize the hardness of learning the optimal policy:  $\Delta(s, a)$  represents the difficulty of learning that in state  $s$  action  $a$  is sub-optimal; the variance  $\mathrm{Var}_{sa}[V^{\star}]$  measures the aleatoric uncertainty in future state values; and the span  $\mathrm{MD}_{sa}[V^{\star}]$  of the optimal value function can be seen as another measure of aleatoric uncertainty, large whenever there is a significant variability in the value for the possible next states.
Estimating the span  $\mathrm{MD}_{sa}[V^{\star}]$ , in an online setting, is a challenging task for large MDPs. Our objective is to derive an alternative upper bound that, in turn, can be approximated using quantities that can be learned in a model-free manner. We observe that the variance of the value function, and more generally its moments  $M_{sa}^{k}[V^{\star}]^{2^{-k}}$  for  $k\geq 1$  (see Appendix C), are smaller than the span. By refining the proof techniques used in [28], we derive the following alternative upper bound.
Theorem 4.2. Let  $\varepsilon \geq 0$  and let  $k(s,a) \coloneqq \arg \sup_{k\in \mathbb{N}}M_{sa}^{k}[V^{\star}]^{2^{-k}}$  (for brevity, we write  $k$  instead of  $k(s,a)$ ). Then,  $\forall \omega \in \Delta(S\times A)$ , we have  $T_{\varepsilon}(\omega)\leq U(\omega)$ , with
$$
U (\omega) := \max  _ {s, a \neq \pi^ {\star} (s)} \left(\frac {2 + 8 \varphi^ {2} M _ {s a} ^ {k} \left[ V ^ {\star} \right] ^ {2 ^ {1 - k}}}{\omega (s , a) \Delta (s , a) ^ {2}} + \max  _ {s ^ {\prime}} \frac {C \left(s ^ {\prime}\right) (1 + \gamma) ^ {2}}{\omega \left(s ^ {\prime} , \pi^ {\star} \left(s ^ {\prime}\right)\right) \Delta (s , a) ^ {2} (1 - \gamma) ^ {2}}\right), \tag {3}
$$
where  $C(s') = \max \left(4, 16\gamma^2 \varphi^2 M_{s', \pi^*(s')}^k [V^\star]^{2^{1-k}}\right)$  and  $\varphi$  is the golden ratio.
We can observe that in the worst case, the upper bound  $U(\omega^{\star})$  of the sample complexity lower bound, with  $\omega^{\star} = \arg \inf_{\omega} U(\omega)$ , scales as  $O\left(\frac{|S||A| \max_s \mathrm{MD}_{s,\pi^*(s)}[V^\star]^2}{\Delta_{\min}^2(1 - \gamma)^2}\right)$ . Since  $\mathrm{MD}_{sa}[V^\star] \leq (1 - \gamma)^{-1}$ , then  $U(\omega^{\star})$  scales at most as  $O\left(\frac{|S||A|}{\Delta_{\min}^2(1 - \gamma)^4}\right)$ . However, the following questions arise: (1) Can we select a single value of  $k$  that provides a good approximation across all states and actions? (2) How much does this bound improve on that of Theorem 4.1? As we illustrate in the example presented in the next subsection, we believe that actually selecting  $k = 1$  for all states and actions leads to sufficiently good results. With this choice, we obtain the following approximation:
$$
U _ {1} (\omega) := \max  _ {s, a \neq \pi^ {\star} (s)} \left(\frac {2 + 8 \varphi^ {2} \operatorname {V a r} _ {s a} [ V ^ {\star} ]}{\omega (s , a) \Delta (s , a) ^ {2}} + \max  _ {s ^ {\prime}} \frac {C ^ {\prime} \left(s ^ {\prime}\right) (1 + \gamma) ^ {2}}{\omega \left(s ^ {\prime} , \pi^ {\star} \left(s ^ {\prime}\right)\right) \Delta (s , a) ^ {2} (1 - \gamma) ^ {2}}\right), \tag {4}
$$