Exploiting Minimum-Variance Policy Evaluation for Policy Optimization

1 INTRODUCTION . Policy Optimization methods ( PO , Deisenroth et al. , 2013 ) have been widely exploited in Reinforcement Learning ( RL , Sutton & Barto , 2018 ) with successful results in addressing , to name a few , continuous-control ( e.g. , Peters & Schaal , 2008 ; Lillicrap et al. , 2016 ) , robot manipulation ( e.g. , Gu et al. , 2017 ; Chatzilygeroudis et al. , 2020 ) , and locomotion ( e.g. , Kohl & Stone , 2004 ; Duan et al. , 2016 ) . Most of these algorithms employ the notion of trust region ( Conn et al. , 2000 ) , introduced ante litteram in the RL literature by the safe RL approaches ( Kakade & Langford , 2002 ; Pirotta et al. , 2013 ) , giving rise to a surge of effective algorithms , having TRPO ( Schulman et al. , 2015 ) as the progenitor . The core of any RL algorithm , being value-based or policy-based , lies in the ability to employ the samples collected with the current ( or behavioral ) policy to evaluate the performance of a candidate ( or target ) policy ( Sutton & Barto , 2018 ) . The skeleton rationale behind the usage of a trust region is to control the set of candidate policies whose performance can be accurately evaluated . Intuition suggests that if the candidate policy is “ sufficiently close ” to the current one , this off-policy evaluation problem ( Precup et al. , 2000 ) will provide a good estimate for the performance of the candidate policy . Formally , this idea has been studied in the field of Importance Sampling ( IS , Owen , 2013 ) and the phenomenon is particularly evident looking at the IS estimator variance , which grows exponentially with the Rényi divergence ( Rényi , 1961 ) between the behavioral and the target policy ( Metelli et al. , 2018 ; 2020 ) . In this off-policy learning ( Off-PL ) setting , IS is employed as a what-if analysis tool ( Owen , 2013 ) and its role is passive , as samples have been already collected with the current behavioral policy . In this sense , the trust region is an a-posteriori remedy for the limitations of off-policy evaluation , for controlling the uncertainty injected by the IS procedure . However , IS originated in the Monte Carlo simulation community ( Hesterberg , 1988 ; Hammersley , 2013 ) as an active tool for variance minimization ( Off-VM ) . While in Off-PL , the behavioral policy is fixed and we look for the best target policy , whose performance we aim to estimate , here the roles are reversed . Indeed , in Off-VM , the target policy is fixed and we search for the behavioral policy ( from which to collect samples ) that yields an IS estimate with the minimum possible variance ( Hammersley , 2013 ; Kahn & Marshall , 1953 ) . It might seem surprising , at first , that sampling from a policy , other than the target one , can lead to an estimator with less variance ( even zero in some cases ) w.r.t . the on-policy estimate . In this role , IS has been previously employed in RL , mainly to address rare events ( Frank et al. , 2008 ; Ciosek & Whiteson , 2017 ) which naturally lead to high-variance estimates , when tackled on-policy . The idea of explicitly using IS as a variance reduction technique , with the goal of finding an optimal behavioral policy , was proposed by ( Hanna et al. , 2017 ) for evaluation and subsequently combined with policy gradient learning ( Hanna & Stone , 2018 ; Hanna et al. , 2019 ) . However , in these works , the variance minimization ( Off-VM ) process and the off-policy learning ( Off-PL ) problem are treated separately . The goal of this paper is to investigate the relation between variance minimization ( Off-VM ) and off-policy learning ( Off-PL ) . The core question we address can be summarized as : “ Can Off-VM be employed as a tool for Off-PL , overcoming the need for an explicit trust region ? ” Intuitively , given a target policy , when the reward function is positive , one way to reduce the variance of the IS estimator is to assign larger probability to the trajectories that have a large impact on the mean , i.e. , those with high returns . This provides a first hint about the connection between the minimum-variance sampling policy and the performance improvement , i.e. , between Off-VM and Off-PL . Furthermore , it suggests that we could repeatedly apply the process of identifying the minimum-variance policy as a tool for policy improvement . The interesting aspect of such an approach is that , by minimizing the variance , it implicitly controls the divergence between two consecutive policies . In other words , it allows enforcing a trust region , without the need for divergence constraints or penalizations . Outline of the Contributions In this paper , we provide theoretical , algorithmic , and experimental contributions . After having introduced the background ( Section 2 ) , we present the problem of finding the minimum-variance behavioral distribution ( Section 3 ) . Then , we study the properties of the Off-VM problem in two settings : unconstrained ( Section 4 ) and constrained ( Section 5 ) . First , we assume that there are no restrictions for choosing the behavioral distribution . We show that the minimum-variance behavioral distribution , besides leading to the zero-variance estimator ( Kahn & Marshall , 1953 ) , is guaranteed to yield a performance improvement , requiring the non-negativity of the reward only . Furthermore , we prove that this approach allows controlling the divergence between two consecutive distributions , thus enforcing an implicit trust region . Although this provides a valuable starting point , the minimum-variance distribution might be unrealizable given the environment transition model , i.e. , there might be no policy inducing it . For this reason , we move to the scenario in which the distributions are constrained in a suitable space . In this setting , the zerovariance estimator could not be achievable . Nevertheless , we prove that such a procedure can lead to a performance improvement and preserves the trust region enforcement . Based on these theoretical results , we propose Policy Optimization via Optimal Policy Evaluation ( PO2PE ) , a novel PO algorithm , that we particularize for parametric policy spaces ( Section 6 ) . Finally , we provide numerical simulations on continuous-control benchmarks , in comparison with POIS ( Metelli et al. , 2018 ) and TRPO ( Schulman et al. , 2015 ) , with a particular focus on the robustness of PO2PE to small batch sizes ( Section 7 ) . The proof of the results presented in the main paper are reported in Appendix A . 2 PRELIMINARIES . In this section , we provide the necessary background that will be employed in the paper . Mathematical Notation Let X be a set , and let FX be a σ-algebra over X . We denote with PpX q the space of probability measures over pX , FX q . Let P PPpX q , whenever needed , we assume that P admits a density function p. For a subset YĎR , we denote with BpX , Yq the space of measurable functions f : XÑY . Let P , QPPpX q be two probability measures such that P ! Q , i.e. , P is absolutely continuous w.r.t . Q , for every αPr0,8s , we define the α-Rényi divergence as ( Rényi , 1961 ) : DαpP } Qq “ 1α´1 log ş X ppxq αqpxq1´αdx . In the limit of αÑ1 , the Rényi divergence reduces to the KL-divergence DKLpP } Qq , while for αÑ8 , it corresponds to esssupx „ Q tppxq { qpxqu . Importance Sampling Let P , QPPpX q with P ! Q and let f PBpX , Rq . Importance Sampling ( IS , Owen , 2013 ) allows estimating the expectation of f under a target distribution P , i.e. , Ex „ P rfpxqs having samples txiuiPrns collected with a behavioral distribution Q : pµP { Q “ 1 n ř iPrns ppxiq qpxiqfpxiq . The IS estimator is unbiased ( Owen , 2013 ) , i.e. , Exi „ QrpµP { Qs “ Ex „ P rfpxqs , but it might suffer from large variance , due to the heavy-tailed behavior ( Metelli et al. , 2018 ) . The properties of pµP { Q and its transformations have been extensively studied in the literature ( e.g. , Ionides , 2008 ; Thomas et al. , 2015 ; Papini et al. , 2019 ; Kuzborskij et al. , 2021 ; Metelli et al. , 2020 ) . Policy Optimization A Markov Decision Process ( MDP , Puterman , 1994 ) is a 6-tuple M “ pS , A , P , R , γ , D0q , where S is the state space , A is the action space , P : SˆAÑPpSq is the transition model , R : SˆAÑr0 , Rmaxs is the reward function , γ Pr0,1s is the discount factor , and D0 PPpSq is the initial state distribution . The agent ’ s behavior is modeled by a parametric policy πθ : SÑPpAq belonging to a parametric policy space ΠΘ “ tπθ : θPΘĎRdu . The interaction between an agent and the MDP generates a trajectory τ “ ps0 , a0 , s1 , a1 , . . . , sH´1 , aH´1 , sHq where H PN is the trajectory length and s0 „ D0 , at „ πθp¨|stq , st ` 1 „ P p¨|st , atq for all tP t0 , . . . , H´1u . Given a trajectory τ , the return is the discounted sum of the rewards Rpτq “ řH´1 t “ 0 γ tRpst , atq . For a policy πθ PΠΘ , we denote with pp¨|θq the induced trajectory distribution : ppτ |θq “ D0ps0q śH´1 t “ 0 πθpat|stqP pst ` 1|st , atq . In the action-based ( AB ) setting , an agent aims at finding a parametrization fulfilling : θ˚ PargmaxθPΘ tJpθqu , where Jpθq “ Eτ „ pp¨|θq rRpτqs is the expected return . πθ must be stochastic to ensure exploration . Instead , in the parameter-based ( PB ) setting , we consider a hyperpolicy νρ , belonging to a parametric hyperpolicy space NP “ tνρ : ρPPĎRlu , from which we sample the parameters θ of the policy . In this case , the policy πθ can be deterministic since exploration is managed at the hyperpolicy level and the agent goal becomes to learn a hyperpolicy parametrization maximizing the expected return : ρ˚ PargmaxρPPtJpρqu , where Jpρq “ Eθ „ νρrJpθqs . In the paper , we keep the presentation as general as possible , introducing the results for arbitrary distributions . Then , we will particularize for the parametric PO setting . 3 MINIMUM–VARIANCE BEHAVIORAL DISTRIBUTION . In this section , we revise Off-VM , i.e. , the problem of finding a behavioral distribution QPPpX q that induces an IS estimate pµP { Q with minimum variance , knowing the ( fixed ) target distribution P PPpX q and function f PBpX , r0,8qq.1 Furthermore , we do not enforce any restriction on the possible forms of the behavioral distribution QPPpX q . The problem and the corresponding wellknown minimum-variance behavioral distribution Q˚ are stated in the following ( Kahn , 1950 ) : min QPPpX q '' Varx „ Q „ ppxq qpxqfpxq * ùñ q˚pxq “ ppxqfpxq Ex „ P rfpxqs , @ xPX . ( 1 ) We observe that the IS estimator pµP { Q˚ is non-stochastic , equal to the quantity we aim to estimate , i.e. , pµP { Q˚ “ Ex „ P rfpxqs . This suggests that the construction of Q˚ is infeasible as it requires knowledge of Ex „ P rfpxqs . Since Q˚ generates a non-stochastic estimator , it not only leads to zero-variance but , clearly , simultaneously minimizes the absolute central moments of any order . A second , and most remarkable property , is that Q˚ is a performance improvement w.r.t . P , i.e. , the expectation of f under Q˚ is larger than the expectation of f under the target P ( Owen , 2013 ) : Ex „ Q˚rfpxqs´Ex „ P rfpxqs “ Varx „ P rfpxqs Ex „ P rfpxqs ě0 . ( 2 ) It is worth noting that the magnitude of the improvement is directly related to the reduction in variance Varx „ P rfpxqs . Equation ( 2 ) suggests an appealing connection between the problem of finding the minimum-variance behavioral distribution ( Off-VM ) and the problem of finding a target distribution that maximizes the expectation Ex „ P rfpxqs ( Off-PL ) . In other words , we could employ Off-VM as a performance improvement tool , by repeatedly solving the problem in Equation ( 1 ) . In the following two sections , we will delve into the properties of the repeated construction of the minimum-variance distribution as a performance improvement tool under two assumptions : ( i ) there are no restrictions in the choice of the behavioral distribution QPPpX q ( Section 4 ) ; ( ii ) the behavioral distribution must be chosen within a subsetQPQĎPpX q ( Section 5 ) . In both cases , we will address the following three questions : ( Q1 ) Does this procedure always generate a distribution that is a performance improvement ? ( Q2 ) Does this procedure converge to a ( global or local ) maximum of f ? ( Q3 ) Can we quantify the divergence between two consecutive distributions , i.e. , does this procedure enforce a trust region ? Before proceeding , let us map this general setting to PO . In the action-based ( AB ) setting , x is the trajectory τ , P and Q are trajectory distributions ppτ |θq induced by policies πθ . Instead , in the parameter-based ( PB ) setting , x is the pair pθ , τq , P and Q are joint distributions νρpθqppτ |θq induced by hyperpolicies νρ . In both cases , function f is the trajectory return Rpτq . 1We restrict our attention to non-negative functions . From the RL perspective , this choice is w.l.o.g . since we can always define an equivalent non-negative reward function , by means of a translation of the original one .

Summary: Importance sampling (IS) is at the core of many off-policy reinforcement learning (RL) algorithms.  The common use of IS in RL is to consider a _fixed_ policy behavior, say $q$ and an optimization policy $p$. The samples are acquired with the behavior policy $q$. The objective is to maximize  $$ \max_p \mathbb{E}_{x\sim q}\left[\frac{p(x)}{q(x)}f(x)\right]. $$  This approach suffers from high variance.   This paper highlight that originally, IS was thought to _reduce_ the variance of Monte-Carlo estimates. The main idea of this paper is to use IS as a variance minimization technique. It is interesting to see that the $\min_q Var[p(x)/q(x)f(x)]$ leads to a _policy improvement_.  The paper is structured as follow:  1. The authors show in Equation 1 the _minimum variance_ $q^*$. This analytical result is (as far as I understand) purely theoretical and cannot be used directly in RL. Note that Equation 2 also shows how the minimum variance behavior is a policy improvement. The authors, therefore, suggest that one could simply iterate the minimum variance to consistently improve the policy. At this point, a few questions remain open: will this process converge to a local or global optimum? Can one quantify the divergence between consecutive distributions?  2. To study the convergence properties, the authors introduce an operator $\mathcal{I}$ which takes in input a distribution $p$ and produces the next _minimum variance_ distribution $q$, which will be the next behavior policy. The operator $\mathcal{I}$ has several fixed points: if $p$ is deterministic then $p = \mathcal{I}p$. Theorem 4.2 ensures that the iterated application of $\mathcal{I}$ converges to the optimum in the domain of $p$.  3. Often, trust-region methods are used in RL to ensure stability in the learning process. Theorem 4.3 shows that the application of $\mathcal{I}$ produces a policy with bounded Renyi divergence with the previous policy. Interesting, this result is valid for any order of the divergence. This result is stronger than many other results in RL, where the divergence is only bounded for a unique order (typically 2: KL divergence), leaving the divergence on other orders unbounded.   4. However, the finding $q^*$ is often not possible. In short, if we consider a generic set of distribution, the application of $\mathcal{I}$ to an element of the set, might produce a distribution not contained in the set. The authors need at this point to devise an _approximation_ of $\mathcal{I}$, and check again for the convergence properties.  The first approximation consists in minimizing a $\alpha$-Renyi divergence. The difference between classic divergence minimization (or constraint) approaches is that in this paper, the authors are constraining divergence between "return distribution" (even though modified by the monotonic function $h$) rather than the divergence between parameter distribution or state distribution like in REPS or TRPO (@authors: correct me if I am wrong on this point).  5. The authors analyze the policy improvement of this "approximated" operator. They notice that the policy improvement happens only for some particular choice of $h$    6. The authors prove in Theorem 5.4 that the approximated operator has a tighter trust region (for every single choice of $\alpha$) w.r.t. the unconstrained operator $\mathcal{i}$.  7. The authors provide a practical algorithm in Section 6, and evaluate its efficacy in four classic control benchmarks. They also provide an analysis of the hyperparameters used (monotonic function $h$ and batch-size).   To summarize, the paper proposes a different use of IS, which allows both for variance reduction and policy improvement.  The improvement scheme is new (although it can be related to other approaches) and sheds new light on policy gradient optimization.