SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Bhavya Sukhija<sup>1</sup> Lenart Treven<sup>1</sup> Cansu Sancaktar<sup>2</sup> Sebastian Blaes<sup>2</sup>  
Stelian Coros<sup>1</sup> Andreas Krause<sup>1</sup>
ETH Zürich<sup>1</sup> MPI for Intelligent Systems<sup>2</sup>
{sukhijab,trevenl,scoros,krausea}@ethz.ch
{cansu.sancaktar,sebastian.blae}@tuebingen.mpg.de
Reinforcement learning algorithms commonly seek to optimize policies for solving one particular task. How should we explore an unknown dynamical system such that the estimated model globally approximates the dynamics and allows us to solve multiple downstream tasks in a zero-shot manner? In this paper, we address this challenge, by developing an algorithm - OPAX- for active exploration. OPAX uses well-calibrated probabilistic models to quantify the epistemic uncertainty about the unknown dynamics. It optimistically—w.r.t. to plausible dynamics—maximizes the information gain between the unknown dynamics and state observations. We show how the resulting optimization problem can be reduced to an optimal control problem that can be solved at each episode using standard approaches. We analyze our algorithm for general models, and, in the case of Gaussian process dynamics, we give a first-of-its-kind sample complexity bound and show that the epistemic uncertainty converges to zero. In our experiments, we compare OPAX with other heuristic active exploration approaches on several environments. Our experiments show that OPAX is not only theoretically sound but also performs well for zero-shot planning on novel downstream tasks.
Most reinforcement learning (RL) algorithms are designed to maximize cumulative rewards for a single task at hand. Particularly, model-based RL algorithms, such as (Chua et al., 2018; Kakade et al., 2020; Curi et al., 2020), excel in efficiently exploring the dynamical system as they direct the exploration in regions with high rewards. However, due to the directional bias, their underlying learned dynamics model fails to generalize in other areas of the state-action space. While this is sufficient if only one control task is considered, it does not scale to the setting where the system is used to perform several tasks, i.e., under the same dynamics optimized for different reward functions. As a result, when presented with a new reward function, they often need to relearn a policy from scratch, requiring many interactions with the system, or employ multi-task (Zhang and Yang, 2021) or transfer learning (Weiss et al., 2016) methods. Traditional control approaches such as trajectory optimization (Biagiotti and Melchiorri, 2008) and model-predictive control (García et al., 1989) assume knowledge of the system's dynamics. They leverage the dynamics model to solve an optimal control problem for each task. Moreover, in the presence of an accurate model, important system properties such as stability and sensitivity can also be studied. Hence, knowing an accurate dynamics model bears many practical benefits. However, in many real-world settings, obtaining a model using just physics' first principles is very challenging. A promising approach is to leverage data for learning the dynamics, i.e., system identification or active learning. To this end, the key question we investigate in this work is: how should we interact with the system to learn its dynamics efficiently?
While active learning for regression and classification tasks is well-studied, active learning in RL is much less understood. In particular, active learning methods that yield strong theoretical and practical results, generally query data points based on information-theoretic criteria (Krause et al., 2008; Settles, 2009; Balcan et al., 2010; Hanneke et al., 2014; Chen et al., 2015). In the context of
dynamical systems, this requires querying arbitrary transitions (Berkenkamp et al., 2017; Mehta et al., 2021). However, in most cases, querying a dynamical system at any state-action pair is unrealistic. Rather, we can only execute policies on the real system and observe the resulting trajectories. Accordingly, an active learning algorithm for RL needs to suggest policies that are "informative" for learning the dynamics. This is challenging since it requires planning with unknown dynamics.
Contributions In this paper, we introduce a new algorithm, Optimistic Active eXploration (OPAX), designed to actively learn nonlinear dynamics within continuous state-action spaces. During each episode, OPAX plans an exploration policy to gather the most information possible about the system. It learns a statistical dynamics model that can quantify its epistemic uncertainty and utilizes this uncertainty for planning. The planned trajectory targets state-action pairs where the model's epistemic uncertainty is high, which naturally encourages exploration. In light of unknown dynamics, OPAX uses an optimistic planner that picks policies that optimistically yield maximal information. We show that this optimism paradigm plays a crucial role in studying the theoretical properties of OPAX. Moreover, we provide a general convergence analysis for OPAX and prove convergence to the true dynamics for Gaussian process (GP) dynamics models. Theoretical guarantees for active learning in RL exist for a limited class of systems (Simchowitz et al., 2018; Wagenmaker and Jamieson, 2020; Mania et al., 2020), but lack for a more general and practical class of dynamics (Chakraborty et al., 2023; Wagenmaker et al., 2023). We are, to the best of our knowledge, the first to give convergence guarantees for a rich class of nonlinear dynamical systems.
We evaluate OPAX on several simulated robotic tasks with state dimensions ranging from two to 58. The empirical results provide validation for our theoretical conclusions, showing that OPAX consistently delivers strong performance across all tested environments. Finally, we provide an efficient implementation<sup>1</sup> of OPAX in JAX (Bradbury et al., 2018).
We study an unknown discrete-time dynamical system  $\pmb{f}^{*}$ , with state  $\pmb{x} \in \mathcal{X} \subset \mathbb{R}^{d_x}$  and control inputs  $\pmb{u} \in \mathcal{U} \subset \mathbb{R}^{d_u}$ .
$$
\boldsymbol {x} _ {k + 1} = \boldsymbol {f} ^ {*} \left(\boldsymbol {x} _ {k}, \boldsymbol {u} _ {k}\right) + \boldsymbol {w} _ {k}. \tag {1}
$$
Here,  $\pmb{w}_k$  represents the stochasticity of the system for which we assume  $\pmb{w}_k \stackrel{i.i.d.}{\sim} \mathcal{N}(\mathbf{0}, \sigma^2 \pmb{I})$  (Assumption 3). Most common approaches in control, such as trajectory optimization and model-predictive control (MPC), assume that the dynamics model  $\pmb{f}^*$  is known and leverages the model to control the system state. Given a cost function  $c: \mathcal{X} \times \mathcal{U} \to \mathbb{R}$ , such approaches formulate and solve an optimal control problem to obtain a sequence of control inputs that drive the system's state
$$
\underset {\boldsymbol {u} _ {0: T - 1}} {\arg \min } \mathbb {E} _ {\boldsymbol {w} _ {0: T - 1}} \left[ \sum_ {t = 0} ^ {T - 1} c (\boldsymbol {x} _ {t}, \boldsymbol {u} _ {t}) \right], \tag {2}
$$
$$
\boldsymbol {x} _ {t + 1} = \boldsymbol {f} ^ {*} (\boldsymbol {x} _ {t}, \boldsymbol {u} _ {t}) + \boldsymbol {w} _ {t} \quad \forall 0 \leq t \leq T.
$$
Moreover, if the dynamics are known, many important characteristics of the system such as stability, and robustness (Khalil, 2015) can be studied. However, in many real-world scenarios, an accurate dynamics model  $f^{*}$  is not available. Accordingly, in this work, we consider the problem of actively learning the dynamics model from data a.k.a. system identification (Åström and Eykhoff, 1971). Specifically, we are interested in devising a cost-agnostic algorithm that focuses solely on learning the dynamics model. Once a good model is learned, it can be used for solving different downstream tasks by varying the cost function in Equation (2).
We study an episodic setting, with episodes  $n = 1,\dots ,N$ . At the beginning of the episode  $n$ , we deploy an exploratory policy  $\pi_{n}$ , chosen from a policy space  $\Pi$  for a horizon of  $T$  on the system. Next, we obtain trajectory  $\pmb{\tau}_{n} = (\pmb{x}_{n,0},\dots ,\pmb{x}_{n,T})$ , which we save to a dataset of transitions  $\mathcal{D}_n = \{(z_{n,i} = (x_{n,i},\pi_n(x_{n,i})),y_{n,i} = x_{n,i + 1})_{0\leq i <   T}\}$ . We use the collected data to learn an estimate  $\pmb{\mu}_{n}$  of  $f^{*}$ . To this end, the goal of this work is to propose an algorithm  $Alg$ , that at each episode  $n$  leverages the data acquired thus far, i.e.,  $\mathcal{D}_{1:n - 1}$  to determine a policy  $\pi_{n}\in \Pi$  for the next step of data collection, that is,  $Alg(\mathcal{D}_{1:n - 1},n)\to \pi_n$ . The proposed algorithm should be consistent, i.e.,  $\pmb{\mu}_n(\pmb {z})\rightarrow \pmb{f}^* (\pmb {z})$  for  $n\to \infty$  for all  $z\in \mathcal{R}$ , where  $\mathcal{R}$  is the reachability set defined as
$$
\mathcal {R} = \left\{\boldsymbol {z} \in \mathcal {Z} \mid \exists (\boldsymbol {\pi} \in \Pi , t \leq T), \text {s . t .}, p \left(\boldsymbol {z} _ {t} = \boldsymbol {z} \mid \boldsymbol {\pi}, \boldsymbol {f} ^ {*}\right) > 0 \right\},
$$
'https://github.com/lasgroup/opax
and efficient w.r.t. rate of convergence of  $\mu_{n}$  to  $f^{*}$ .
To devise such an algorithm, we take inspiration from Bayesian experiment design (Chaloner and Verdinelli, 1995). In the Bayesian setting, given a prior over  $\pmb{f}^{*}$ , a natural objective for active exploration is the mutual information (Lindley, 1956) between  $\pmb{f}^{*}$  and observations  $\pmb{y}_{\mathcal{D}_n}$ .
Definition 1 (Mutual Information, Cover and Thomas (2006)). The mutual information between  $\mathbf{f}^*$  and its noisy measurements  $\mathbf{y}_{\mathcal{D}_n}$  for points in  $\mathcal{D}_n$ , where  $\mathbf{y}_{\mathcal{D}_n}$  is the concatenation of  $(\mathbf{y}_{\mathcal{D}_n,i})_{i < T}$  is defined as,
$$
F \left(\mathcal {D} _ {n}\right) := I \left(\boldsymbol {f} ^ {*}; \boldsymbol {y} _ {\mathcal {D} _ {n}}\right) = H \left(\boldsymbol {y} _ {\mathcal {D} _ {n}}\right) - H \left(\boldsymbol {y} _ {\mathcal {D} _ {n}} \mid \boldsymbol {f} ^ {*}\right), \tag {3}
$$
where  $H$  is the Shannon differential entropy.
The mutual information quantifies the reduction in entropy of  $\pmb{f}^{*}$  conditioned on the observations. Hence, maximizing the mutual information w.r.t. the dataset  $\mathcal{D}_n$  leads to the maximal entropy reduction of our prior. Accordingly, a natural objective for active exploration in RL can be the mutual information between  $\pmb{f}^{*}$  and the collected transitions over a budget of  $N$  episodes, i.e.,  $I(\pmb{f}^{*};\pmb{y}_{\mathcal{D}_{1:N}})$ . This requires maximizing the mutual information over a sequence of policies, which is a challenging planning problem even in settings where the dynamics are known (Mutny et al., 2023). A common approach is to greedily pick a policy that maximizes the information gain conditioned on the previous observations at each episode:
$$
\max  _ {\boldsymbol {\pi} \in \Pi} \mathbb {E} _ {\tau^ {\pi}} \left[ I \left(\boldsymbol {f} _ {\tau^ {\pi}} ^ {*}; \boldsymbol {y} _ {\tau^ {\pi}} \mid \mathcal {D} _ {1: n - 1}\right) \right]. \tag {4}
$$
Here  $\pmb{f}_{\tau^{\pi}}^{*} = (\pmb{f}^{*}(\pmb{z}_{n,0}), \dots, \pmb{f}^{*}(\pmb{z}_{n,T - 1})), \pmb{y}_{\tau^{\pi}} = (\pmb{y}_{n,0}, \dots, \pmb{y}_{n,T - 1}), \tau^{\pi}$  is the trajectory under the policy  $\pmb{\pi}$ , and the expectation is taken w.r.t. the process noise  $\pmb{w}$ .
Interpretation in frequentist setting While information gain is Bayesian in nature (requires a prior over  $\pmb{f}^{*}$ ), it also has a frequentist interpretation. In particular, later in Section 3 we relate it to the epistemic uncertainty of the learned model. Accordingly, while this notion of information gain stems from Bayesian literature, we can use it to motivate our objective in both Bayesian and frequentist settings.
In this work, we learn a probabilistic model of the function  $f^{*}$  from data. Moreover, at each episode  $n$ , we learn the mean estimator  $\pmb{\mu}_n(\pmb{x},\pmb{u})$  and the epistemic uncertainty  $\sigma_{n}(\pmb{x},\pmb{u})$ , which quantifies our uncertainty on the mean prediction. To this end, we use Bayesian models such as Gaussian processes (GPs, Rasmussen and Williams, 2005) or Bayesian neural networks (BNNs, Wang and Yeung, 2020). More generally, we assume our model is well-calibrated:
Definition 2 (All-time calibrated statistical model of  $f^*$ , Rothfuss et al. (2023)). Let,  $z = (x, u)$  and  $\mathcal{Z} := \mathcal{X} \times \mathcal{U}$ . An all-time calibrated statistical model of the function  $f^*$  is a sequence  $(\pmb{\mu}_n, \pmb{\sigma}_n, \beta_n(\delta))_{n \geq 0}$ , such that
$$
\Pr \left(\forall z \in \mathcal {Z}, \forall l \in \{1, \dots , d _ {x} \}, \forall n \in \mathbb {N}: \left| \mu_ {n, l} (z) - f _ {l} (z) \right| \leq \beta_ {n} (\delta) \sigma_ {n, l} (z)\right) \geq 1 - \delta
$$
Here  $\mu_{n,l}$  and  $\sigma_{n,l}$  are the  $l$ -th element in the vector valued functions  $\pmb{\mu}_n$  and  $\pmb{\sigma}_{n}$  respectively. The scalar function,  $\beta_n(\delta)\in \mathbb{R}_{\geq 0}$  quantifies the width of the  $1 - \delta$  confidence intervals. We assume w.l.o.g. that  $\beta_{n}$  monotonically increases with  $n$ , and that  $\sigma_{n,l}(\pmb {z})\leq \sigma_{\mathrm{max}}$  for all  $\pmb {z}\in \mathcal{Z}$ ,  $n\geq 0$ , and  $l\in \{1,\dots ,d_x\}$ .
Assumption 1 (Well calibration assumption). Our learned model is an all-time-calibrated statistical model of  $\mathbf{f}^*$ , i.e., there exists a sequence of  $(\beta_n(\delta))_{n\geq 0}$  such that our model satisfies the well-calibration condition, c.f., Definition 2.
This is a natural assumption on our modeling. It states that we can make a mean prediction and also quantify how far it is off from the true one with high probability. A GP model satisfies this requirement for a very rich class of functions, c.f., Lemma 3. For BNNs, calibration methods (Kuleshov et al., 2018) are often used and perform very well in practice. Next, we make a simple continuity assumption on our function  $f^{*}$ .
Assumption 2 (Lipschitz Continuity). The dynamics model  $\pmb{f}^*$  and our epistemic uncertainty prediction  $\sigma_{n}$  are  $L_{\pmb{f}}$  and  $L_{\sigma}$  Lipschitz continuous, respectively. Moreover, we define  $\Pi$  to be the policy class of  $L_{\pi}$  Lipschitz continuous functions.
The Lipschitz continuity assumption on  $f^{*}$  is quite common in control theory (Khalil, 2015) and learning literature (Curi et al., 2020; Pasztor et al., 2021; Sussex et al., 2023). Furthermore, the Lipschitz continuity of  $\sigma_{n}$  also holds for GPs with common kernels such as the linear or radial basis function (RBF) kernel (Rothfuss et al., 2023).
Finally, we reiterate the assumption of the system's stochasticity.
Assumption 3 (Process noise distribution). The process noise is i.i.d. Gaussian with variance  $\sigma^2$ , i.e.,  $\pmb{w}_k^{i,i,d} \sim \mathcal{N}(\pmb{0}, \sigma^2\pmb{I})$ .
We focus on the setting where  $\boldsymbol{w}$  is homoscedastic for simplicity. However, our framework can also be applied to the more general heteroscedastic and sub-Gaussian case (c.f., Theorem 2).
3 Optimistic Active Exploration
In this section, we propose our optimistic active exploration (OPAX) algorithm. The algorithm consists of two main contributions: (i) First we reformulate the objective in Equation (4) to a simple optimal control problem, which suggests policies that visit states with high epistemic uncertainty. (ii) We leverage the optimistic planner introduced by Curi et al. (2020) to efficiently plan a policy under unknown dynamics. Moreover, we show that the optimistic planner is crucial in giving theoretical guarantees for the algorithm.
3.1 Optimal Exploration Objective
The objective in Equation (4) is still difficult and expensive to solve in general. However, since in this work, we consider Gaussian noise, c.f., Assumption 3, we can simplify this further.
Lemma 1 (Information gain is upper bounded by sum of epistemic uncertainties). Let  $\pmb{y} = \pmb{f}^{*}(\pmb{z}) + \pmb{w}$  with  $\pmb{w} \sim \mathcal{N}(0, \sigma^2\pmb{I})$  and let  $\pmb{\sigma}_{n-1}$  be the epistemic uncertainty after episode  $n-1$ . Then the following holds for all  $n \geq 1$  and dataset  $\mathcal{D}_{1:n-1}$ ,
$$
I \left(\boldsymbol {f} _ {\tau^ {\pi}} ^ {*}; \boldsymbol {y} _ {\tau^ {\pi}} \mid \mathcal {D} _ {1: n - 1}\right) \leq \frac {1}{2} \sum_ {t = 0} ^ {T - 1} \sum_ {j = 1} ^ {d _ {x}} \log \left(1 + \frac {\sigma_ {n - 1 , j} ^ {2} \left(\boldsymbol {z} _ {t}\right)}{\sigma^ {2}}\right). \tag {5}
$$
We prove Lemma 1 in Appendix A. The information gain is non-negative (Cover and Thomas, 2006). Therefore, if the right-hand side of Equation (5) goes to zero, the left-hand side goes to zero as well. Lemma 1 relates the information gain to the model epistemic uncertainty. Therefore, it gives a tractable objective that also has a frequentist interpretation - collect points with the highest epistemic uncertainty. We can use it to plan a trajectory at each episode  $n$ , by solving the following optimal control problem:
$$
\boldsymbol {\pi} _ {n} ^ {*} = \underset {\boldsymbol {\pi} \in \Pi} {\operatorname {a r g m a x}} J _ {n} (\boldsymbol {\pi}) = \underset {\boldsymbol {\pi} \in \Pi} {\operatorname {a r g m a x}} \mathbb {E} _ {\boldsymbol {\tau} ^ {\pi}} \left[ \sum_ {t = 0} ^ {T - 1} \sum_ {j = 1} ^ {d _ {x}} \log \left(1 + \frac {\sigma_ {n - 1 , j} ^ {2} \left(\boldsymbol {x} _ {t} , \boldsymbol {\pi} \left(\boldsymbol {x} _ {t}\right)\right)}{\sigma^ {2}}\right) \right], \tag {6}
$$
$$
\boldsymbol {x} _ {t + 1} = \boldsymbol {f} ^ {*} (\boldsymbol {x} _ {t}, \boldsymbol {\pi} (\boldsymbol {x} _ {t})) + \boldsymbol {w} _ {t}.
$$
The problem in Equation (6) is closely related to previous literature in active exploration for RL. For instance, some works consider different geometries such as the sum of epistemic uncertainties (Pathak et al. (2019); Sekar et al. (2020), c.f., appendix C for more detail).
3.2 Optimistic Planner
The optimal control problem in Equation (6) requires knowledge of the dynamics  $\pmb{f}^{*}$  for planning, however,  $\pmb{f}^{*}$  is unknown. A common choice is to use the mean estimator  $\pmb{\mu}_{n-1}$  in Equation (6) instead of  $\pmb{f}^{*}$  for planning (Buisson-Fenet et al., 2020). However, in general, using the mean estimator is susceptible to model biases (Chua et al., 2018) and is provably optimal only in the case of linear systems (Simchowitz and Foster, 2020). To this end, we propose using an optimistic planner, as suggested in Curi et al. (2020), instead. Accordingly, given the mean estimator  $\pmb{\mu}_{n-1}$  and the epistemic uncertainty  $\pmb{\sigma}_{n-1}$ , we solve the following optimal control problem
$$
\boldsymbol {\pi} _ {n}, \boldsymbol {\eta} _ {n} = \underset {\boldsymbol {\pi} \in \Pi , \boldsymbol {\eta} \in \Xi} {\operatorname {a r g m a x}} J _ {n} (\boldsymbol {\pi}, \boldsymbol {\eta}) = \underset {\boldsymbol {\pi} \in \Pi , \boldsymbol {\eta} \in \Xi} {\operatorname {a r g m a x}} \mathbb {E} _ {\boldsymbol {\tau} ^ {\pi , \eta}} \left[ \sum_ {t = 0} ^ {T - 1} \sum_ {j = 1} ^ {d _ {x}} \log \left(1 + \frac {\sigma_ {n - 1 , j} ^ {2} \left(\hat {\boldsymbol {x}} _ {t} , \boldsymbol {\pi} \left(\hat {\boldsymbol {x}} _ {t}\right)\right)}{\sigma^ {2}}\right) \right], \tag {7}
$$
$$
\hat {\boldsymbol {x}} _ {t + 1} = \boldsymbol {\mu} _ {n - 1} (\hat {\boldsymbol {x}} _ {t}, \boldsymbol {\pi} (\hat {\boldsymbol {x}} _ {t})) + \beta_ {n - 1} (\delta) \boldsymbol {\sigma} _ {n - 1} (\hat {\boldsymbol {x}} _ {t}, \boldsymbol {\pi} (\hat {\boldsymbol {x}} _ {t})) \boldsymbol {\eta} (\hat {\boldsymbol {x}} _ {t}) + \boldsymbol {w} _ {t},
$$
OPAX: OPTIMISTIC ACTIVE EXPLORATION
Init: Aleatoric uncertainty  $\sigma$ , Probability  $\delta$ , Statistical model  $(\pmb{\mu}_0, \pmb{\sigma}_0, \beta_0(\delta))$  for episode  $n = 1, \dots, N$  do
$$
\boldsymbol {\pi} _ {n} = \underset {\boldsymbol {\pi} \in \Pi} {\operatorname {a r g m a x}} \max  _ {\boldsymbol {\eta} \in \Xi} \mathbb {E} \left[ \sum_ {t = 0} ^ {T - 1} \sum_ {j = 1} ^ {d _ {x}} \log \left(1 + \frac {\sigma_ {n - 1 , j} ^ {2} (\boldsymbol {x} _ {t} , \boldsymbol {\pi} (\boldsymbol {x} _ {t}))}{\sigma^ {2}}\right) \right] \quad \succcurlyeq \text {P r e p a r e p o l i c y}
$$
$$
\mathcal {D} _ {n} \leftarrow \operatorname {R O L L O U T} \left(\boldsymbol {\pi} _ {n}\right) \quad > \text {C o l l e c t m e a s u r e m e n t s}
$$
$$
\text {U p d a t e} \left(\boldsymbol {\mu} _ {n}, \boldsymbol {\sigma} _ {n}, \beta_ {n} (\delta)\right) \leftarrow \mathcal {D} _ {1: n} \quad > \text {U p d a t e m o d e l}
$$
where  $\Xi$  is the space of policies  $\eta : \mathcal{X} \to [-1,1]^{d_x}$ . Therefore, we use the policy  $\eta$  to "hallucinate" (pick) transitions that give us the most information. Overall, the resulting formulation corresponds to a simple optimal control problem with a larger action space, i.e., we increase the action space by another  $d_x$  dimension. A natural consequence of Assumption 1 is that  $J_{n}(\pi_{n}^{*}) \leq J_{n}(\pi_{n},\eta_{n})$  with high probability (c.f., Corollary 1 in Appendix A). That is by solving Equation (7), we get an optimistic estimate on Equation (6). Intuitively, the policy  $\pi_n$  that OPAX suggests, behaves optimistically with respect to the information gain at each episode.
We theoretically analyze the convergence properties of OPAX. We first study the regret of planning under unknown dynamics. Specifically, since we cannot evaluate the optimal exploration policy from eq. (6) and use the optimistic one, i.e., eq. (7) instead, we incur a regret. We show that due to the optimism in the face of uncertainty paradigm, we can give sample complexity bounds for the Bayesian and frequentist settings. All the proofs are presented in Appendix A.
Lemma 2 (Regret of optimistic planning under unknown dynamics). Let Assumption 1 hold. Furthermore, define  $J_{n,k}(\pmb{\pi}_n,\pmb{\eta}_n,\pmb{x})$  as
$$
J _ {n, k} \left(\boldsymbol {\pi} _ {n}, \boldsymbol {\eta} _ {n}, \boldsymbol {x}\right) = \mathbb {E} _ {\boldsymbol {\tau} ^ {\boldsymbol {\pi} _ {n}}, \boldsymbol {\eta} _ {n}} \left[ \sum_ {t = k} ^ {T - 1} \sum_ {j = 1} ^ {d _ {x}} \log \left(1 + \frac {\sigma_ {n - 1 , j} ^ {2} \left(\hat {\boldsymbol {x}} _ {t} , \boldsymbol {\pi} _ {n} \left(\hat {\boldsymbol {x}} _ {t}\right)\right)}{\sigma^ {2}}\right) \right],
$$
$$
s. t. \hat {\boldsymbol {x}} _ {t + 1} = \boldsymbol {\mu} _ {n - 1} (\hat {\boldsymbol {x}} _ {t}, \boldsymbol {\pi} _ {n} (\hat {\boldsymbol {x}} _ {t})) + \beta_ {n - 1} (\delta) \boldsymbol {\sigma} _ {n - 1} (\hat {\boldsymbol {x}} _ {t}, \boldsymbol {\pi} _ {n} (\hat {\boldsymbol {x}} _ {t})) \boldsymbol {\eta} _ {n} (\hat {\boldsymbol {x}} _ {t}) + \boldsymbol {w} _ {t}
$$
$$
a n d \hat {\boldsymbol {x}} _ {0} = \boldsymbol {x}.
$$
Then, for all  $n \geq 1$ , with probability at least  $1 - \delta$
$$
J _ {n} \left(\boldsymbol {\pi} _ {n} ^ {*}\right) - J _ {n} \left(\boldsymbol {\pi} _ {n}\right) \leq \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {\boldsymbol {\tau} ^ {\boldsymbol {\pi} _ {n}}} \left[ J _ {n, t + 1} \left(\boldsymbol {\pi} _ {n}, \boldsymbol {\eta} _ {n}, \boldsymbol {x} _ {t + 1} ^ {\prime}\right) - J _ {n, t + 1} \left(\boldsymbol {\pi} _ {n}, \boldsymbol {\eta} _ {n}, \boldsymbol {x} _ {t + 1}\right) \right],
$$
$$
w i t h \boldsymbol {x} _ {t + 1} = \boldsymbol {f} ^ {*} (\boldsymbol {x} _ {t}, \boldsymbol {\pi} _ {n} (\boldsymbol {x} _ {t})) + \boldsymbol {w} _ {t},
$$
$$
a n d \boldsymbol {x} _ {t + 1} ^ {\prime} = \boldsymbol {\mu} _ {n - 1} (\boldsymbol {x} _ {t}, \boldsymbol {\pi} _ {n} (\boldsymbol {x} _ {t})) + \beta_ {n - 1} (\delta) \boldsymbol {\sigma} _ {n - 1} (\boldsymbol {x} _ {t}, \boldsymbol {\pi} _ {n} (\boldsymbol {x} _ {t})) \boldsymbol {\eta} _ {n} (\boldsymbol {x} _ {t}) + \boldsymbol {w} _ {t}.
$$