SlowGuess/NeurIPS_2023_OCR huggingface dataset:
within  $f_{t}$ , e.g.,  $\mathbb{Var}(R_t)$  is replaced with  $\left(\sqrt{\widehat{\mathbb{Var}}_i(R_t)} + \mathbf{C}_{i,t}^\sigma\right)^2$  and  $\mathbb{Cov}(R_t, R_{t'})$  is replaced with  $\widehat{\mathbb{Cov}}_i(R_t, R_{t'}) + \mathbf{C}_{i,t,t'}^c$ . Intuitively, the purpose of the exploration bonus is to consider the uncertainty that arises from replacing exact quantities with their empirical estimation. This introduces in RIDO a source of robustness w.r.t. the noise that is intrinsically present in the underlying estimation process. At this point, concerning the shape of Equation (6), focus for the sake of exposition on  $\mathbf{C}_{i,t}^\sigma$ . First of all, we notice that the hyper-parameter  $\beta$  governs the robustness which is taken into account while replacing  $\mathbb{Var}(R_t)$  with its empirical estimate. Larger values of  $\beta$ , correspond, indeed, to larger  $\mathbf{C}_{i,t}^\sigma$ , and, consequently, a higher level of robustness w.r.t. the uncertainty. Furthermore, as we can notice,  $\mathbf{C}_{i,t}^\sigma$  decreases with the number of samples collected in the previous iterations at step  $t$ , i.e.,  $\sum_{j=1}^{i-1} \hat{n}_{j,t}$ . This quantity coincides with the number of samples that are used to estimate  $\sqrt{\widehat{\mathbb{Var}}_i[R_t]}$ . This formulation captures the following aspect: more data is available to the agent to estimate  $\mathbb{Var}(R_t)$ , the more accurate its estimate will be, and, consequently, its exploration bonus will shrink to 0. As one can expect, with this approach, the quality of the objective function used in RIDO increases with the number of iterations. Consequently, the agent will progressively adapt the mini-batch DCS toward time steps where more data is required to minimize Equation (3), i.e., the variance of the return estimator. We conclude with two remarks. First, we notice that RIDO can be applied with  $\gamma = 1$ , as it does not deeply rely on the property of discounted sums. Secondly, the optimization problem (5) is a complex integer and non-linear optimization problem. Before diving into the statistical analysis of RIDO, we discuss how to solve (5) in the next section.
4.1 Solving the Empirical Optimization Problem
As noticed above, directly solving problem (5) requires significant effort since it is an integer, non-linear optimization problem. In this section, we discuss how to overcome these challenges.
We first perform a continuous relaxation, replacing the integer constraint  $n_t \in \mathbb{N}_+$  with  $n_t \geq 1$ . Once a solution  $\bar{n}^*$  to the relaxed optimization problem is found, it is possible to obtain a proper (i.e., integer) DCS by flooring each  $\bar{n}_t^*$  and allocating the remaining budget uniformly. As we shall see, this approximation introduces constant terms in the theoretical guarantees of RIDO only. At this point, the resulting optimization problem is a non-linear problem that, unfortunately, is generally non-convex. This issue occurs when the following condition is verified for some time step  $t$ :
$$
f _ {t} = \gamma^ {2 t} \left(\sqrt {\hat {\mathbb {V}} \operatorname {a r} _ {i} \left(R _ {t}\right)} + \mathrm {C} _ {i, t} ^ {\sigma}\right) ^ {2} + 2 \sum_ {t ^ {\prime} = t + 1} ^ {T - 1} \gamma^ {t + t ^ {\prime}} \left(\widehat {\mathbb {C}} \operatorname {o v} _ {i} \left(R _ {t}, R _ {t ^ {\prime}}\right) + \mathrm {C} _ {i, t, t ^ {\prime}} ^ {c}\right) <   0. \tag {7}
$$
To solve this challenge and make RIDO computationally efficient, we develop an approach based on a hidden property of the original optimization problem (4). More specifically, we start by noticing that even the continuous relaxation of (4) is non-convex since  $f_{\bar{t}} < 0$  might occur, for some  $\bar{t} \in \{0,\dots ,T - 2\}$ , in the presence of negative covariances with future steps. In this case, however, since  $\sum_{t = \bar{t}}^{T - 1}f_t$  represents a proper variance, which is always non-negative, there always exists  $t^{\prime} > \bar{t}$  such that  $\sum_{t = \bar{t}}^{t^{\prime}}f_{t}\geq 0$ . Furthermore, it is possible to show that the optimal solution of the relaxed optimization problem is uniform in the interval  $\{\bar{t},\ldots ,t'\}$ , namely  $n_{\bar{t}}^{*} = n_{\bar{t} +1}^{*} = \dots = n_{t'}^{*}$  (proof in
Similar comments apply to  $\mathbf{C}_{i,t,t'}^c$  as well. The only difference stands in the fact that to estimate the empirical covariance between two subsequence steps  $t$  and  $t'$ , samples up to time  $t'$  are required. For this reason, the denominator implies the summation of the number of samples gathered at  $t'$  over the previous iterations.
Figure 1: Visualization of the transformation between the optimization problems. The first row shows the objective function of the original optimization problem, while the second one its transformation.
Appendix A). For this reason, it is possible to define a transformation of the optimization problem that preserves the optimal solution, in which the variables  $n_{\bar{t}}, \ldots, n_{t'}$  are replaced with a single variable  $y$ . The objective function is modified accordingly, namely  $\frac{f_{\bar{t}}}{n_{\bar{t}}} + \dots + \frac{f_{t'}}{n_{t'}}$  is replaced with  $\frac{f_{\bar{t}} + \cdots + f_{t'}}{y}$  in the objective function. A visualization of the transformation is proposed in Figure 1. By repeating the procedure for all the negative  $f_{t}$ , we obtain a transformation of the original problem which is now convex. Once the solution to this convex transformed optimization problem is found, one can quickly recover the relaxed DCS in its  $T$ -dimensional form.
Building on these results, we apply in RIDO a similar procedure that transforms the relaxed version of (5) into a new problem where the negative time steps (i.e., steps in which Equation (7) holds) are "grouped" with future time steps as long as the total summation is positive. In this way, (i) the resulting optimization problem is convex and (ii) as our analysis will reveal, this procedure has no impact on the theoretical properties of RIDO (i.e., the result is the same as assuming access to an oracle that can solve non-linear and non-convex problems). As a concluding remark, we refer the reader to Appendix A for a formal description of the above-mentioned procedure.
4.2 Theoretical Analysis
We now present theoretical guarantees on the performance of RIDO. More specifically, we derive high-probability guarantees on the variance of the estimator of Equation (1) when used with the data collected by RIDO. Before diving into the presentation of our results, we highlight some critical challenges behind the result. First, in our analysis, we do not assume access to an oracle that solves (5), but we consider the modifications discussed in Section 4.1 that make the computation tractable. This introduces a first level of challenges in the analysis (e.g., dealing with the roundings that arise from the relaxation and the peculiar strategy that overcomes the non-convexity of the optimization problem). Secondly, we notice that none of the optimization problems (4) and (5), and the ones obtained by relaxing the integer constraints, admit a closed-form solution (further details are provided in Appendix A). This clearly results in an additional challenge in our analysis. At this point, we are ready to state our main theoretical result (proof in Appendix A).
Theorem 4.1. Let  $\pmb{n}^*$  be the optimal solution of problem (4),  $f_t$  as in Equation (3),  $b \geq 2T$  and  $\beta = \frac{6(T + T^2)\Lambda K}{\delta}$ . Consider the DCS  $\hat{\pmb{n}}$  computed by Algorithm 1. Then, with probability at least  $1 - \delta$  it holds that:
$$
\operatorname {V a r} _ {\hat {\boldsymbol {n}}} \left[ \hat {J} _ {\hat {\boldsymbol {n}}} (\pi) \right] \leq 1 9 2 \left(\frac {b}{\Lambda}\right) ^ {\frac {3}{2}} \log (\beta) \left(\sum_ {t = 0} ^ {T - 1} \gamma^ {t}\right) ^ {2} + 4 \operatorname {V a r} _ {\boldsymbol {n} ^ {*}} \left[ \hat {J} _ {\boldsymbol {n} ^ {*}} (\pi) \right] + \frac {2 b}{\Lambda} \sum_ {t: f _ {t} <   0} | f _ {t} |. \tag {8}
$$
Equation (8) comprises three terms which we now discuss in detail. The former is directly responsible for taking into account the cumulative error computed during each phase  $i$ . This term shrinks to zero with rate  $\widetilde{\mathcal{O}}\left(\left(\frac{b}{\Lambda}\right)^{\frac{3}{2}}\log \Lambda\right)$ . We notice that this gets smaller as we decrease  $b$ , thus suggesting to use small batch sizes. This should come as no surprise; indeed, using smaller batch sizes intuitively improves the adaptiveness of the algorithm, since a larger portion of the budget  $\Lambda$  will be allocated following more precise estimates of the quantities of interest. In this sense, there exists a trade-off between theoretical guarantees and computational requirements, since the number of iterations (and, thus, the number of optimization problems to be solved) grows linearly as the batch size decreases. The second term, instead, is the variance of the optimal DCS computed as in (4), and shrinks with a
rate that is at most  $\frac{4}{\Lambda} \sum_{t=0}^{T-1} f_t$ . This term, as we shall show below, represents a particularly desirable property. Finally, the last component of Equation (8) is related to the negative terms possibly present in the objective function, and, among the three terms, it is the one with the worst dependence on  $\Lambda$ . Currently, we are unsure whether this term is an artifact of the analysis, a sub-optimality of the algorithm, or a key challenge of the setting. We leave closing this gap to future work. At this point, we highlight a particular relevant property of Equation (8). Suppose that  $f_t \geq 0$  holds for all time steps  $t$  (so that the last component is not present). In this case, under the mild assumption that  $\sum_{t=0}^{T-1} f_t > 0$  (i.e., the variance is different from 0), for sufficiently large budget of  $\Lambda$ , we have that (formal statement and proof in Appendix A):
$$
\left. \operatorname {V a r} _ {\hat {\boldsymbol {n}}} \left[ \hat {J} _ {\hat {\boldsymbol {n}}} (\pi) \right] \leq 5 \operatorname {V a r} _ {\boldsymbol {n} ^ {*}} \left[ \hat {J} _ {\boldsymbol {n} ^ {*}} (\pi) \right]. \right. \tag {9}
$$
Thus, in this scenario, the variance of the returned DCS computed by RIDO is proportional to the optimal one. Note that this sort of result is not possible for the uniform strategy, nor for the robust one of Poiani et al. [2023]. Further details on this point are provided in Appendix A.
In this section, we propose numerical validations that aim at assessing the empirical performance of RIDO. More specifically, we focus on the comparison between our approach, the classical uniform-in-the-horizon strategy, and the robust DCS by Poiani et al. [2023]. We report the results across multiple domains, values of budget  $\Lambda$ , and discount factor  $\gamma$ . As a performance index, all experiments measure the empirical variance of the estimator in Equation (1) at the end of the data collection process. Before discussing our results in detail, we describe our experimental settings in depth.
Experimental Setting In our experiments, we consider the following four domains. We start with the Inverted Pendulum [Brockman et al., 2016], a classic continuous control benchmark, where the agents' goal is to swing up a suspended body and keep it in the vertical direction. We, then, continue with the Linear Quadratic Gaussian Regulator [LQG, Curtain, 1997], where the agent controls a linear dynamical system with the objective of reducing a total cost that is expressed as a quadratic function. Then, we consider a 2D continuous navigation problem, where an agent starts at the bottom left corner of a room and needs to reach a goal region in the upper right corner. The agent receives reward 0 everywhere except inside the goal area, where the reward is positive and sampled from a Gaussian distribution. Finally, we consider the Ant environment from the MuJoCo [Todorov et al., 2012] suite, where the agent controls a four-legged 3D robot with the goal of moving it forward. Further domain details are provided in Appendix B. Concerning the policy that we evaluate for the Inverted Pendulum and the Ant, we rely on pre-trained deep RL agents made publicly available by Raffin [2020]. For the LQG, instead, we evaluate the optimal policy that is available in closed form by solving the Riccati equations, and, finally, for the 2D navigation task, we roll out a hand-designed policy that minimizes the distance of the agent's position w.r.t. to the center of the goal region. Regarding the performance index, as already anticipated, we report the variance of the empirical policy return at the end of the data collection process. Given a budget and a DCS, for a single run, we estimate this empirical variance using 100 simulations. We then average the results over 100 runs and report the empirical mean together with  $95\%$  confidence intervals. We notice that for each considered value of  $\Lambda$ , the experiment is repeated (i.e., we do not use data collected with smaller  $\Lambda$ 's). To conclude, we refer the reader to Appendix B for further details on the experiments (e.g., ablations, additional results, experiments with  $\gamma = 1$ , hyper-parameters, visualizations of the resulting DCSs).
Results Figure 2 reports the results varying the discount factor and the available budget. The second row is obtained under the same experimental setting as the first one, but with lower values of  $\gamma$ . Let us first focus on the sub-optimality of the non-adaptive DCSs (i.e., the uniform strategy and the robust one of Poiani et al. [2023]). Indeed, as suggested by Theorem 3.1, being computed prior to the interaction with the environment, these algorithms cannot adapt the collection of samples to minimize the variance of the return estimator. This is clear by looking, for instance, at the results of Continuous Navigation and the LQG. Indeed, in the Continuous Navigation domain, the reward is sparse and received close to the end of the estimation horizon  $T$ . In this scenario, the robust DCS blindly truncates trajectories, thus, avoiding the collection of experience in the most relevant timesteps. Conversely, in the LQG experiments, the optimal policy that arises from the Riccati equation pays a
Figure 2: Empirical variance (mean and  $95\%$  confidence intervals over 100 runs) on the considered domains and baselines. The first row considers higher values of  $\gamma$  w.r.t. the second one.
stochastic control cost<sup>6</sup> at the beginning of the estimation horizon to bring the state of the system close to stability, after which the reward will remain almost constant. In this case, the uniform DCS results in a highly sub-optimal behavior as most of the estimation uncertainty is related to the initial interaction steps. RIDO, on the other hand, thanks to its adaptivity, is able to obtain the best results in both domains. Indeed, in the Continuous Navigation problem, it achieves the same performance level as the uniform strategy, while in the LQG it even outperforms the robust DCS of Poiani et al. [2023]. The reason is that Poiani et al. [2023] truncates trajectories solely depending on the value of  $\gamma$ , and, therefore, it might waste a portion of its budget in trajectories of sub-optimal length, while RIDO, since it aims at minimizing the variance of the final estimation, is able to focus the collection of data in the most convenient way. Similar comments to those made for the LQG hold for the Pendulum domain as well. Concerning the Ant environment, instead, we notice that for  $\gamma = 0.999$  there is no significant difference between any of the presented schedules. Interestingly, however, as soon as we decrease  $\gamma$  to 0.99, we can appreciate the sub-optimality of the uniform strategy, which wastes a portion of its budget in gathering samples that are significantly discounted, and, therefore, their weight in the estimator's variance shrinks to 0. On the other hand, the robust strategy and RIDO avoid this pitfall thanks to the exploitation of the discount factor, thus obtaining reduced variance estimates. Finally, we remark that RIDO has achieved the most competitive performance across various domains, values of the discount factor, and budget, thus clearly highlighting the benefits of adaptive strategies w.r.t. pre-determined ones.
We now present a comprehensive analysis and discussion of previous works that are closely connected to our own research. First of all, our work focuses on estimating a policy's performance in a given MDP [Sutton and Barto, 2018]. Considering the significance of this task, reducing the variance, or more generally, the error of the return estimator, is a problem that has received significant attention in the literature. A vast family of approaches that can be used to solve this problem deeply exploits the Markovian properties of the environment by relying on Temporal Difference [TD, see, e.g., Singh and Sutton, 1996, Sutton, 1988, Lee and He, 2019, Riquelme et al., 2019, Qu et al., 2019] learning. On the other hand, our work focuses purely on Monte Carlo simulation, which can be transparently applied to non-Markovian environments. Another relevant line of work deals with optimizing the
<sup>6</sup>The uncertainty, in this case, arises both from the noise of the system together with the stochasticity of the initial state distribution.
agent's policy to collect data within an environment (i.e., behavioral policy) to reduce the variance of an unbiased estimator for the return of a different target policy [Hanna et al., 2017, Zhong et al., 2022, Mukherjee et al., 2022]. These techniques are referred to as off-policy evaluation methods and usually rely on Importance Sampling [e.g., Hesterberg, 1988, Owen, 2013] techniques to guarantee the unbiasedness of the resulting estimate. However, these studies significantly differ from ours in that, instead of aiming for a behavior policy that reduces the estimator variance, our goal is to directly exploit the properties of Monte Carlo data collection to reduce the on-policy estimator variance.
In the context of RL, exploration bonuses are widely adopted in control (where the goal is learning an optimal policy) to tackle the exploration-exploitation dilemma [e.g., Brafman and Tennenholtz, 2002, Auer et al., 2008, Tang et al., 2017, Jin et al., 2018, O'Donoghue et al., 2018, Zanette and Brunskill, 2019]. Initially, when the agent has limited knowledge about the environment, the exploration bonuses drive it to explore widely. As the agent's knowledge improves, the exploration bonuses decrease, and the agent can shift towards exploiting its learned policy more. In our work, instead, we use exploration bonuses to introduce a source of robustness w.r.t. the objective function that we are interested in, i.e., the variance of the return estimator for a given data collection strategy.
Finally, the work that is most related to ours is Poiani et al. [2023], where, the concept of truncating trajectories has been analyzed in the context of Monte Carlo RL. More specifically, the authors derived a non-adaptive schedule of trajectories that provably minimizes confidence intervals around the return estimator. In this work, on the other hand, we have shown the sub-optimality of pre-determined schedules, and we designed an adaptive algorithm that aims at minimizing the variance of the final estimate. The concept of truncating trajectories has also received some attention in other fields of research such as model-based policy optimization [Nguyen et al., 2018, Janner et al., 2019, Bhatia et al., 2022, Zhang et al., 2023], multi-task RL [Farahmand et al., 2016] and imitation learning [Sun et al., 2018]. However, in all these works, the motivation, the method, and the analysis completely differ w.r.t. what has been considered here. Finally, the concept of truncating trajectories in Monte Carlo RL drew inspiration from a recent work in the field of multi-fidelity bandit [Poiani et al., 2022], where the authors considered the idea of cutting trajectories while interacting with the environment to obtain a biased estimate of the return of a policy in planning algorithms such as depth-first search.
7 Conclusions and Future Works
In this work, we studied the problem of allocating a budget  $\Lambda$  of transitions in the context of Monte Carlo policy evaluation to reduce the error of the policy expected return estimate. Leveraging the formalism of Data Collection Strategy (DCS) to model how an agent spends its interaction budget, we started by analyzing, in closed form, the variance of an unbiased return estimator for any possible DCS. Our result reveals that DCSs determined prior to the interaction with the environment (e.g., the usual uniform-in-the-horizon one and the robust one of Poiani et al. [2023]) fail to satisfy the ultimate goal of policy evaluation, i.e., produce a low error estimate. Furthermore, it also suggests that algorithms that spend the available budget  $\Lambda$  iteratively might successfully adapt their strategy to minimize the variance of the return estimator. Inspired by these findings, we propose an adaptive method, RIDO, that, by exploiting information that has already been collected, can dynamically adapt its DCS to allocate a larger portion of transitions in time steps in which more accurate sampling is required to reduce the variance of the final estimate. After conducting a theoretical analysis on the properties of the proposed method, we present empirical studies that confirm its adaptivity across a different number of domains, values of budget  $\Lambda$ , and discount factors  $\gamma$ .
Our study offers exciting possibilities for future research. For example, it would be interesting to extend our ideas to policy search algorithms (e.g., Williams [1992]), with the goal of finding DCSs that minimize the variance of the empirical gradient that is adopted in the update rule. Furthermore, we notice that, since our approach is purely based on MC simulation, it does not fully leverage the Markovian properties of the underlying MDP. Combining TD techniques [Sutton and Barto, 2018] with mechanisms that truncate trajectories is a challenging and open research question that could lead to further improvements in the efficiency of RL algorithms.