SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jinxin Liu $^{1,2}$  Hongyin Zhang $^{1,2}$  Zifeng Zhuang $^{1,2}$  Yachen Kang $^{1,2}$
Donglin Wang $^{1*}$  Bin Wang $^{3}$
<sup>1</sup>Westlake University <sup>2</sup>Zhejiang University <sup>3</sup>Huawei Noah's Ark Lab
In this work, we decouple the iterative bi-level offline RL (value estimation and policy extraction) from the offline training phase, forming a non-iterative bi-level paradigm and avoiding the iterative error propagation over two levels. Specifically, this non-iterative paradigm allows us to conduct inner-level optimization (value estimation) in training, while performing outer-level optimization (policy extraction) in testing. Naturally, such a paradigm raises three core questions that are not fully answered by prior non-iterative offline RL counterparts like reward-conditioned policy: Q1) What information should we transfer from the inner-level to the outer-level? Q2) What should we pay attention to when exploiting the transferred information for safe/confident outer-level optimization? Q3) What are the benefits of concurrently conducting outer-level optimization during testing? Motivated by model-based optimization (MBO), we propose DROP (Design fROm Policies), which fully answers the above questions. Specifically, in the inner-level, DROP decomposes offline data into multiple subsets and learns an MBO score model (A1). To keep safe exploitation to the score model in the outer-level, we explicitly learn a behavior embedding and introduce a conservative regularization (A2). During testing, we show that DROP permits test-time adaptation, enabling an adaptive inference across states (A3). Empirically, we find that DROP, compared to prior non-iterative offline RL counterparts, gains an average improvement probability of more than  $80\%$ , and achieves comparable or better performance compared to prior iterative baselines.
Offline reinforcement learning (RL) [37, 40] describes a task of learning a policy from static offline data. Due to the overestimation of values at out-of-distribution (OOD) state-actions, recent iterative offline RL methods introduce various policy/value regularization to avoid deviating from the offline data distribution in the training phase. Then, these methods directly deploy the learned policy in an online environment to test its performance. To unfold our following analysis, we term this kind of learning procedure as iterative bi-level offline RL (Figure 1 left), where the inner-level optimization refers to trying to eliminate the OOD issue, the outer-level optimization refers to trying to infer an optimal policy that will be employed at testing. Here, we use the "iterative" term to emphasize that the inner-level and outer-level are iteratively optimized in the training phase. However, due to the iterative error exploitation and propagation [6] over the two levels, performing such an iterative bi-level optimization completely in training often struggles to learn a stable policy/value function.
*Corresponding author: Donglin Wang <wangdonglin@westlake.edu.cn>
Figure 1: Framework of (iterative and non-iterative) bi-level offline RL optimization, where the inner-level optimization refers to regularizing the policy/value function (for OOD issues) and the outer-level refers to updating the policy for reward maximizing. The "?" transferred from inner-level to outer-level depends on the specific choice of algorithm used (see different choices for in Table 1).
In this work, we thus advocate for non-iterative bi-level optimization (Figure 1 right) that decouples the bi-level optimization from the training phase, i.e., performing inner-level optimization (eliminating OOD) in training and performing outer-level optimization (maximizing return) in testing. Intuitively, incorporating the outer-level optimization into the testing phase can eliminate the iterative error propagation over the two levels, i.e., there is no iteratively learned value function that bootstraps off of itself and thus propagates errors further. Then, three core questions are: Q1) What information ("") should we transfer from the inner-level to the outer-level? Q2) What should we pay special attention to when exploiting "?" for safe/confident outer-level optimization? Q3) Notice that the outer-level optimization and the testing rollout form a new loop, what new benefit does this give us?
Intriguingly, prior works under such a non-iterative framework have proposed to transfer (""[in Q1) filtered trajectories [10], a reward-conditioned policy [14, 34], and the Q-value estimation of the behavior policy [6, 24], all of which, however, partially address the above questions (we will discuss these works in Section 3.4). In this work, we thus propose a new alternative method that transfers an embedding-conditioned score model (Q-value) and we will show that this method sufficiently answers the raised questions and benefits most from the non-iterative bi-level framework.
Before introducing our method, we introduce a conceptually similar task (to the non-iterative bi-level optimization) — offline model-based optimization (MBO $^2$  [62]), which aims to discover, from static input-score pairs, a new design input that will lead to the highest score. Typically, offline MBO first learns a score model that maps the input to its score via supervised regression (corresponding to inner-level optimization), and then conducts inference with the learned score model (as "☑"), for instance, by optimizing the input against the learned score model via gradient ascent (corresponding to the outer-level). To enable this MBO implementation in offline RL, we are required to decompose an offline RL task into multiple sub-tasks, each of which thus corresponds to a behavior policy-return (parameters-return) pair. However, there are practical optimization difficulties when using high-dimensional policy's parameter space (as input for the score model) to learn such a score model (inner-level) and conduct inference (outer-level). At inference, directly extrapolating the learned score model ("☑") also tends to drive the high-dimensional candidate policy parameters towards OOD, invalid, and low-scoring parameters [32], as these are falsely and over-optimistically scored by the learned score model in inner-level optimization.
To tackle these problems, we suggest  $^3$  A1) learning low-dimensional embeddings for those sub-tasks decomposed in the MBO implementation, over which we estimate an embedding-conditioned Q-value ("T" in Q1), and A2) introducing a conservative regularization, which pushes down the predicted scores on OOD embeddings. Intuitively, this conservation aims to avoid over-optimistic exploitation and protect against producing unconfident embeddings when conducting outer-level optimization (Q2). Meanwhile, we suggest A3) conducting test-time adaptation, which means we can dynamically adjust the inferred embeddings across testing states (aka deployment adaptation). We name our method DROP (Design fROm Policies). Compared with standard offline MBO for parameter design [62], test-time adaptation in DROP leverages the sequential structure of RL tasks, rather than simply conducting inference at the beginning of testing rollout. Empirically, we demonstrate that DROP can effectively extrapolate a better policy that benefits from the non-iterative framework by answering the raised questions, and achieves better performance compared to state-of-the-art offline RL algorithms.
2Please note that this MBO is different from the typical model-based RL, where the model in MBO denotes a score model while that in the model-based RL denotes a transition dynamics model.
<sup>3</sup>We use A1, A2, and A3 to denote our answers to the raised questions (Q1, Q2, and Q3) respectively.
Reinforcement learning. We model the interaction between agent and environment as a Markov Decision Process (MDP) [61], denoted by the tuple  $(S, \mathcal{A}, P, R, p_0)$ , where  $\mathcal{S}$  is the state space,  $\mathcal{A}$  is the action space,  $P: S \times \mathcal{A} \times \mathcal{S} \to [0, 1]$  is the transition kernel,  $R: S \times \mathcal{A} \to \mathbb{R}$  is the reward function, and  $p_0: S \to [0, 1]$  is the initial state distribution. Let  $\pi \in \Pi := \{\pi: S \times \mathcal{A} \to [0, 1]\}$  denotes a policy. In RL, we aim to find a stationary policy that maximizes the expected discounted return  $J(\pi) := \mathbb{E}_{\tau \sim \pi}[\sum_{t=0}^{\infty} \gamma^t R(\mathbf{s}_t, \mathbf{a}_t)]$  in the environment, where  $\tau = (\mathbf{s}_0, \mathbf{a}_0, r_0, \mathbf{s}_1, \mathbf{a}_1, \dots)$ ,  $r_t = R(\mathbf{s}_t, \mathbf{a}_t)$ , is a sample trajectory and  $\gamma \in (0, 1)$  is the discount factor. We also define the state-action value function  $Q^\pi(\mathbf{s}, \mathbf{a}) := \mathbb{E}_{\tau \sim \pi}[\sum_{t=0}^{\infty} \gamma^t R(\mathbf{s}_t, \mathbf{a}_t)|\mathbf{s}_0 = \mathbf{s}, \mathbf{a}_0 = \mathbf{a}]$ , which describes the expected discounted return starting from state  $\mathbf{s}$  and action  $\mathbf{a}$  and following  $\pi$  afterwards, and the state value function  $V^\pi(\mathbf{s}) = \mathbb{E}_{\mathbf{a} \sim \pi(\mathbf{a}|\mathbf{s})}[Q^\pi(\mathbf{s}, \mathbf{a})]$ . To maximize  $J(\pi)$ , the actor-critic algorithm alternates between policy evaluation and improvement. Given initial  $Q^0$  and  $\pi^0$ , it iterates
$$
Q ^ {k + 1} \leftarrow \arg \min  _ {Q} \mathbb {E} _ {\left(\mathbf {s}, \mathbf {a}, \mathbf {s} ^ {\prime}\right) \sim \mathcal {D} ^ {+}, \mathbf {a} ^ {\prime} \sim \pi^ {k} \left(\mathbf {a} ^ {\prime} \mid \mathbf {s} ^ {\prime}\right)} \left[ \left(R (\mathbf {s}, \mathbf {a}) + \gamma Q ^ {k} \left(\mathbf {s} ^ {\prime}, \mathbf {a} ^ {\prime}\right) - Q (\mathbf {s}, \mathbf {a})\right) ^ {2} \right], \tag {1}
$$
$$
\pi^ {k + 1} \leftarrow \arg \max  _ {\pi} \mathbb {E} _ {\mathbf {s} \sim \mathcal {D} ^ {+}, \mathbf {a} \sim \pi (\mathbf {a} | \mathbf {s})} \left[ Q ^ {k + 1} (\mathbf {s}, \mathbf {a}) \right], \tag {2}
$$
where the value function (critic)  $Q(\mathbf{s}, \mathbf{a})$  is updated by minimizing the mean squared Bellman error with an experience replay  $\mathcal{D}^+$  and, following the deterministic policy gradient theorem [59], the policy (actor)  $\pi(\mathbf{a} | \mathbf{s})$  is updated to maximize the estimated  $Q^{k+1}(\mathbf{s}, \pi(\mathbf{a} | \mathbf{s}))$ .
Offline reinforcement learning. In offline RL [40], the agent is provided with a static data  $\mathcal{D} = \{\tau\}$  which consists of trajectories collected by running some data-generating policies. Note that here we denote static offline data  $\mathcal{D}$ , distinguishing from the experience replay  $\mathcal{D}^+$  in the online setting. Unlike the online RL problem where the experience  $\mathcal{D}^+$  in Equation 1 can be dynamically updated, the agent in offline RL is not allowed to interact with the environment to collect new experience data. As a result, naively performing policy evaluation as in Equation 1 may query the estimated  $Q^{k}(\mathbf{s}^{\prime},\mathbf{a}^{\prime})$  on actions that lie far outside of the static offline data  $\mathcal{D}$ , resulting in pathological value  $Q^{k + 1}(\mathbf{s},\mathbf{a})$  that incurs large error. Further, iterating policy evaluation and improvement will cause the inferred policy  $\pi^{k + 1}(\mathbf{a}|\mathbf{s})$  to be biased towards OOD actions with erroneously overestimated values.
Offline model-based optimization. Model-based optimization (MBO) [63] aims to find an optimal design input  $\mathbf{x}^*$  with a given score function  $f^{*}: \mathcal{X} \to \mathcal{Y} \subset \mathbb{R}$ , i.e.,  $\mathbf{x}^{*} = \arg \max_{\mathbf{x}} f^{*}(\mathbf{x})$ . Typically, we can repeatedly query the oracle score model  $f^{*}$  for new candidate design, until it produces the best design. However, we often do not have the oracle score function  $f^{*}$  but are provided with a static offline dataset  $\{(\mathbf{x}, y)\}$  of labeled input-score pairs. To track such a design from data question, we can fit a parametric model  $f$  to the static offline data  $\{(\mathbf{x}, y)\}$  via the empirical risk minimization (ERM):  $f \gets \arg \min_{f} \mathbb{E}_{(\mathbf{x}, y)}\left[(f(\mathbf{x}) - y)^2\right]$ . Then, starting from the best point in the dataset, we can perform gradient ascent on the design input and set the inferred optimal design  $\mathbf{x}^{*} = \mathbf{x}_{K}^{\circ} := \mathrm{GradAscent}_{f}(\mathbf{x}_{0}^{\circ}, K)$ , where
$$
\mathbf {x} _ {k + 1} ^ {\circ} \leftarrow \mathbf {x} _ {k} ^ {\circ} + \eta \nabla_ {\mathbf {x}} f (\mathbf {x}) | _ {\mathbf {x} = \mathbf {x} _ {k} ^ {\circ}}, \text {f o r} k = 0, \dots , K - 1. \tag {3}
$$
For simplicity, we will omit subscript  $f$  in  $\mathrm{GradAscent}_f$ . Since the aim is to find a better design beyond all the designs in the dataset while directly optimizing score model  $f$  with ERM can not ensure new candidates (OOD designs/inputs) receive confident scores, one crucial requirement of offline MBO is thus to conduct confident extrapolation.
3 DROP: Design from Policies
We present our framework in Figure 2. In Sections 3.1 and 3.2, we will answer questions Q1 and Q2, setting a learned MBO score model as "?" (A1) and introducing a conservative regularization over the score model (A2). In Section 3.3, we will answer Q3, where we show that conducting outer-level optimization during testing can produce an adaptive embedding inference across states (A3).
3.1 Task Decomposition
Our core idea is to explore MBO in the non-iterative bi-level offline RL framework (Figure 1 right), while capturing the sequential characteristics of RL tasks and answering the raised questions (Q1, Q2, and Q3). To begin with, we first decompose the offline data  $\mathcal{D}$  into  $N$  offline subsets
Figure 2: Overview of DROP. Given offline dataset  $\mathcal{D}$ , we decompose the data into  $N$  ( $= 4$  in the diagram) subsets  $\mathcal{D}_{[N]}$ . Over the decomposed sub-tasks, we learn a task embedding  $\phi(\mathbf{z}|n)$  and conduct conservative MBO by learning a score model  $f(\mathbf{s},\mathbf{a},\mathbf{z})$  and  $N$  behavior policies (modeled by a contextual policy  $\beta(\mathbf{a}|\mathbf{s},\mathbf{z})$ ). During test-time, at state  $\mathbf{s}$ , we perform test-time adaptation by adapting the policy (contextual variable) with  $\pi^{*}(\mathbf{a}|\mathbf{s}) = \beta(\mathbf{a}|\mathbf{s},\mathbf{z}^{*})$ , where  $\mathbf{z}^{*} = \arg \max_{\mathbf{z}} f(\mathbf{s},\beta(\mathbf{a}|\mathbf{s},\mathbf{z}),\mathbf{z})$ .
$\mathcal{D}_{[N]} := \{\mathcal{D}_1, \ldots, \mathcal{D}_N\}$ . In other words, we decompose an offline task, learning with  $\mathcal{D}$ , into multiple offline sub-tasks, learning with  $\mathcal{D}_n \in \mathcal{D}_{[N]}$  respectively. Then, for sub-task  $n \in [1, N]$ , we can perform behavior cloning (BC) to fit a parametric behavior policy  $\beta_n: S \times \mathcal{A} \to [0, 1]$  to model the corresponding offline subset  $\mathcal{D}_n$ :
$$
\beta_ {n} \leftarrow \arg \max  _ {\beta_ {n}} \mathbb {E} _ {(\mathbf {s}, \mathbf {a}) \sim \mathcal {D} _ {n}} \left[ \log \beta_ {n} (\mathbf {a} | \mathbf {s}) \right]. \tag {4}
$$
Additionally, such a decomposition also comes with the benefit that it provides an avenue to exploit the hybrid modes in offline data  $\mathcal{D}$ , because that  $\mathcal{D}$  is often collected using hybrid data-generating behavior policies [18], which suggests that fitting a single behavior policy may not be optimal to model the multiple modes of the offline data distribution (see Appendix A for empirical evidence). In real-world tasks, it is also often the case that offline data exhibit hybrid behaviors. Thus, to encourage the emergence of diverse sub-tasks that capture distinct behavior modes, we perform a simple task decomposition according to the returns of trajectories in  $\mathcal{D}$ , heuristically ensuring that trajectories in the same sub-task share similar returns and trajectories from different sub-tasks have distinct returns<sup>4</sup>.
3.2 Task Embedding and Conservative MBO
Naive MBO over behavior policies. Benefiting from the above offline task decomposition, we can conduct MBO over a set of input-score  $(\mathbf{x},y)$  pairs, where we model the (parameters of) behavior policies  $\beta_{n}$  as the design input  $\mathbf{x}$  and the corresponding expected returns at initial states  $J(\beta_{n})$  as the score  $y$ . Note that, ideally, evaluating behavior policy  $\beta_{n}$ , i.e., calculating  $J(\beta_{n}) \equiv \mathbb{E}_{\mathbf{s}_{0}}[V^{\beta_{n}}(\mathbf{s}_{0})]$ , with subset  $\mathcal{D}_n$  will never trigger the overestimation of values in the inner-level optimization. By introducing a score model  $f: \Pi \to \mathbb{R}$  and setting it as the transfer information "in" in Q1, we can then perform outer-level policy inference with
$$
\pi^ {*} (\mathbf {a} | \mathbf {s}) \leftarrow \arg \max  _ {\pi} f (\pi), \text {w h e r e} f = \arg \min  _ {f} \mathbb {E} _ {n} \left[ \left(f \left(\beta_ {n}\right) - J \left(\beta_ {n}\right)\right) ^ {2} \right]. \tag {5}
$$
However, directly performing optimization-based inference (outer-level optimization),  $\max_{\pi} f(\pi)$ , will quickly find an invalid input for which the learned score model  $f$  outputs erroneously large values (Q2). Furthermore, it is particularly severe if we perform the inference directly over the parameters of policy networks, accounting for the fact that the parameters of behavior policies lie on a narrow manifold in a high-dimensional parametric space.
Task embedding. To enable feasible out-level policy inference, we propose to decouple MBO techniques from high-dimensional policy parameters. We achieve this by learning a latent embedding space  $\mathcal{Z}$  with an information bottleneck  $(\dim(\mathcal{Z}) \ll \min(N, \dim(\Pi)))$ . As long as the high-dimensional parameters of behavior policies can be inferred from the embedding  $\mathbf{z} \in \mathcal{Z}$ , we can use  $\mathbf{z}$  to represent the corresponding sub-task (or the behavior policy). Formally, we learn a deterministic task embedding  $^5 \phi: \mathbb{R}^N \to \mathcal{Z}$  and a contextual behavior policy  $\beta: S \times \mathcal{Z} \times \mathcal{A} \to [0,1]$  that replaces
4See more data decomposition rules in Appendix A.2.
<sup>5</sup>We feed the one-hot encoding of the sub-task specification  $(n = 1,\dots ,N)$  into the embedding network  $\phi$ . We write the embedding function as  $\phi (\mathbf{z}|n)$  instead of  $\phi (n)$  to emphasize that the output of  $\phi$  is  $\mathbf{z}$ .
$N$  separate behavior policies in Equation 4:
$$
\beta (\mathbf {a} | \mathbf {s}, \mathbf {z}), \phi (\mathbf {z} | n) \leftarrow \arg \max  _ {\beta , \phi} \mathbb {E} _ {\mathcal {D} _ {n} \sim \mathcal {D} _ {[ N ]}} \mathbb {E} _ {(\mathbf {s}, \mathbf {a}) \sim \mathcal {D} _ {n}} \left[ \log \beta (\mathbf {a} | \mathbf {s}, \phi (\mathbf {z} | n)) \right], \tag {6}
$$
Conservative MBO. In principle, by substituting the learned task embedding  $\phi (\mathbf{z}|n)$  and contextual behavior policy  $\beta (\mathbf{a}|\mathbf{s},\mathbf{z})$  into Equation 5, we can then conduct MBO over the embedding space: first learning  $f:\mathcal{Z}\rightarrow \mathbb{R}$  with  $\min_f\mathbb{E}_{n,\phi (\mathbf{z}|n)}\left[(f(\mathbf{z}) - J(\beta_n))^2\right]$ , and then setting the optimal embedding with  $\mathbf{z}^{*} = \arg \max_{\mathbf{z}}f(\mathbf{z})$  and the corresponding optimal policy with  $\pi^{*}(\mathbf{a}|\mathbf{s}) = \beta (\mathbf{a}|\mathbf{s},\mathbf{z}^{*})$ . However, we must deliberate a new OOD issue in the  $\mathcal{Z}$ -space, stemming from the original distribution shift in the parametric space when directly optimizing Equation 5.
Motivated by the energy model [38] and the conservative regularization [35], we introduce a conservative score model objective, additionally regularizing the scores of OOD embeddings  $\mu (\mathbf{z})$ :
$$
f \leftarrow \arg \min  _ {f} \mathbb {E} _ {n, \phi (\mathbf {z} | n)} \left[ \left(f (\mathbf {z}) - J \left(\beta_ {n}\right)\right) ^ {2} \right], \text {s . t .} \mathbb {E} _ {\mu (\mathbf {z})} [ f (\mathbf {z}) ] - \mathbb {E} _ {n, \phi (\mathbf {z} | n)} [ f (\mathbf {z}) ] \leq \eta , \tag {7}
$$
where we set  $\mu (\mathbf{z})$  to be the uniform distribution over  $\mathcal{Z}$ -space. Intuitively, as long as the scores of OOD embeddings  $\mathbb{E}_{\mu (\mathbf{z})}[f(\mathbf{z})]$  is lower than that of in-distribution embeddings  $\mathbb{E}_{n,\phi (\mathbf{z}|n)}[f(\mathbf{z})]$  (up to a threshold  $\eta$ ), conducting embedding inference with  $\mathbf{z}^{*} = \arg \max_{\mathbf{z}}f(\mathbf{z})$  would produce the best and confident solution, avoiding towards OOD embeddings that are far away from the training set.
Now we have reframed the non-iterative bi-level offline RL problem as one of offline MBO: in the inner-level optimization (Q1), we set the practical choice for "?" as the learned score model  $f$  (A1); in the outer-level optimization (Q2), we introduce task embedding and conservative regularization to avoid over-optimistic exploitation when exploiting  $f$  for policy/embedding inference (A2). In the next section, we will show how to slightly change the form of the score model  $f$ , so as to leverage the sequential characteristic (loop in Figure 1) of RL tasks and answer the left Q3.
3.3 Test-time Adaptation
Recalling that we update  $f(\mathbf{z})$  to regress the value at initial states  $\mathbb{E}_{\mathbf{s}_0}\left[V^{\beta_n}(\mathbf{s}_0)\right]$  in Equation 7, we then conduct outer-level inference with  $\mathbf{z}^{*} = \arg \max_{\mathbf{z}}f(\mathbf{z})$  and rollout the  $z^{*}$ -conditioned policy  $\pi^{*}(\mathbf{a}|\mathbf{s})\coloneqq \beta (\mathbf{a}|\mathbf{s},\mathbf{z}^{*})$  until the end of testing/deployment rollout episode. In essence, such an inference can produce a confident extrapolation over the distribution of behavior policy embeddings. Going beyond the outer-level policy/embedding inference only at the initial states, we propose that we can benefit from performing inference at any rollout state in deployment (A3).
To enable test-time adaptation, we model the score model with  $f: \mathcal{S} \times \mathcal{A} \times \mathcal{Z} \to \mathbb{R}$ , taking a state-action as an extra input. Then, we encourage the score model to regress the values of behavior policies over all state-action pairs,  $\min_{f} \mathbb{E}_{n,\phi(\mathbf{z}|n)} \mathbb{E}_{(\mathbf{s},\mathbf{a}) \sim \mathcal{D}_n} \left[ (f(\mathbf{s},\mathbf{a},\mathbf{z}) - Q^{\beta_n}(\mathbf{s},\mathbf{a}))^2 \right]$ . For simplicity, instead of learning a separate value function  $Q^{\beta_n}$  for each behavior policy, we learn the score model directly with the TD-error used for learning the value function  $Q^{\beta_n}(\mathbf{s},\mathbf{a})$  as in Equation 1, together with the conservative regularization in Equation 7:
$$
f \leftarrow \arg \min  _ {f} \mathbb {E} _ {\mathcal {D} _ {n} \sim \mathcal {D} _ {[ N ]}} \mathbb {E} _ {(\mathbf {s}, \mathbf {a}, \mathbf {s} ^ {\prime}, \mathbf {a} ^ {\prime}) \sim \mathcal {D} _ {n}} \left[ \left(R (\mathbf {s}, \mathbf {a}) + \gamma \bar {f} \left(\mathbf {s} ^ {\prime}, \mathbf {a} ^ {\prime}, \phi (\mathbf {z} | n)\right) - f (\mathbf {s}, \mathbf {a}, \phi (\mathbf {z} | n))\right) ^ {2} \right], \tag {8}
$$
$$
\begin{array}{l} \text {s . t .} \mathbb {E} _ {n, \mu (\mathbf {z})} \mathbb {E} _ {\mathbf {s} \sim \mathcal {D} _ {n}, \mathbf {a} \sim \beta (\mathbf {a} | \mathbf {s}, \mathbf {z})} [ f (\mathbf {s}, \mathbf {a}, \mathbf {z}) ] - \mathbb {E} _ {n, \phi (\mathbf {z} | n)} \mathbb {E} _ {\mathbf {s} \sim \mathcal {D} _ {n}, \mathbf {a} \sim \beta (\mathbf {a} | \mathbf {s}, \mathbf {z})} [ f (\mathbf {s}, \mathbf {a}, \mathbf {z}) ] \leq \eta , \end{array}
$$
where  $\bar{f}$  denotes a target network and we update the target  $\bar{f}$  with soft updates:  $\bar{f} = (1 - v)\bar{f} +vf$