Importance Resampling for Off-policy Policy Evaluation

1 INTRODUCTION . An important component of many learning systems is learning value functions for many policies . Some examples of such systems are the Horde architecture composed of General Value Functions ( GVFs ) ( Sutton et al. , 2011 ; Modayil et al. , 2014 ) , systems that use options ( Sutton et al. , 1999 ; Schaul et al. , 2015a ) , predictive representation approaches ( Sutton et al. , 2005 ; Schaul and Ring , 2013 ; Silver et al. , 2017 ) and systems with auxiliary tasks ( Jaderberg et al. , 2017 ) . For each target policy , the value function returns the expected return from a state , and can provide useful information about ( long-term ) outcomes under different behaviors . Off-policy learning is critical for learning many value functions with different policies at scale , because it enables data to be generated from one behavior policy to update the values for each target policy in parallel . The typical strategy for off-policy learning is to use importance sampling ( IS ) . For a given state s , with action a selected according to behaviour µ , the importance sampling ratio is the ratio between the probability of the action under the target policy π and the behaviour : π ( a|s ) µ ( a|s ) . The update is multiplied by this ratio , adjusting the action probabilities so that the expectation of the update is as if the actions were sampled according to the target policy π . Though the IS estimator is unbiased and consistent ( Kahn and Marshall , 1953 ; Rubinstein and Kroese , 2016 ) , it can suffer from high or even infinite variance due to large magnitude IS ratios , in theory ( Andradottir et al. , 1995 ) and in practice ( Precup et al. , 2001 ; Mahmood et al. , 2014 ; 2017 ) . There have been some attempts to modify policy evaluation algorithms to mitigate this variance.1 Weighted IS ( WIS ) algorithms have been introduced ( Precup et al. , 2001 ; Mahmood et al. , 2014 ; Mahmood and Sutton , 2015 ) , which normalize each update by the sample average of the ratios . These algorithms did improve learning over standard IS strategies , but are not straightforward to extend to nonlinear function approximation . In the offline setting , a reweighting scheme , called importance sampling with unequal support ( Thomas and Brunskill , 2017 ) , was introduced to account 1There is substantial literature on variance reduction for another area called off-policy policy evaluation , but which estimates only a single number or value for a policy ( e.g. , see ( Thomas and Brunskill , 2016 ) ) . The resulting algorithms differ substantially , and are not easily applicable for learning the value function . for samples where the ratio is zero , in some cases significantly reducing variance . Another strategy has been to use rescaling or truncation of IS ratios , such as in V-trace ( Espeholt et al. , 2018 ) . Several other methods have introduced bias similarly for algorithms with eligibility traces by truncating or scaling IS ratios to maintain stability of the eligibility trace vector , including Tree-Backup ( Precup et al. , 2000 ) , Retrace ( Munos et al. , 2016 ) and ABQ ( Mahmood et al. , 2017 ) . Truncation of IS-ratios in V-trace can incur significant bias , and this additional truncation parameter does need to be tuned . An alternative to reweighting updates is to instead correct the distribution before updating the estimator using weighted bootstrap sampling : resampling a new set of data from the previously generated samples ( Smith et al. , 1992 ; Arulampalam et al. , 2002 ) . Consider a setting where a buffer of data is stored , generated by a behavior policy . Samples for policy π can be obtained by resampling from this buffer , proportionally to π ( a|s ) µ ( a|s ) for state-action pairs ( s , a ) in the buffer . In the sampling literature , this strategy has been proposed under the name Sampling Importance Resampling ( SIR ) ( Rubin , 1988 ; Smith et al. , 1992 ; Gordon et al. , 1993 ) , and has been particularly successful for Sequential Monte Carlo sampling ( Gordon et al. , 1993 ; Skare et al. , 2003 ) . Such resampling strategies have also been popular in classification , with over-sampling or under-sampling typically being preferred to weighted ( cost-sensitive ) updates ( Lopez et al. , 2013 ) . Such a resampling strategy , however , has yet to be proposed for policy evaluation , though there are several potential benefits . By correcting the distribution before updating , standard on-policy updates can be used , without needing to modify or re-derive with IS ratios . This simplifies application of different optimizers , such as those with momentum terms . Another benefit is that the magnitude of the updates will vary less—because updates are not multiplied by very small or very large importance sampling ratios—potentially reducing variance of stochastic updates and simplifying stepsize selection . Resampling should have larger benefits for learning approaches , as compared to averaging or numerical integration problems , because updates accumulate in the weight vector and change the optimization trajectory of the weights . For example , very large importance sampling ratios could destabilize the weights . Such a problem does not occur for resampling , as instead the same transition will simply be resampled multiple times , spreading out the large magnitude update across multiple updates . On the other extreme , with small ratios , IS will waste updates on transitions with very small IS ratios . Resampling , therefore , should have better sample efficiency . Two important questions , therefore , are if these hypothesized advantages manifest in practice in off-policy learning , and how SIR can be extended for use in policy evaluation . In this work , we investigate the use of SIR for online off-policy policy evaluation . We first introduce Importance Resampling ( IR ) , which uses an SIR strategy to sample transitions from a buffer of ( recent ) transitions . These sampled transitions are then used for on-policy updates . We show that IR , with a sliding-window buffer , is a consistent estimator of the one-step on-policy updates , with the same bias as WIS. We then empirically investigate IR on three toy domains and a racing car simulation learning from images . We find that IR is more sample efficient—learning more quickly in terms of number of updates—and has reduced sensitivity in terms of learning rate parameters , for both fixed parameters and within the RMSProp optimizer . 2 BACKGROUND . We consider the problem of learning General Value Functions ( GVFs ) ( Sutton et al. , 2011 ) . The agent interacts in an environment defined by a set of states S , a set of actions A and Markov transition dynamics , with probability P ( s′|s , a ) of transitions to state s′ when taking action a in state s. A GVF is defined for policy π : S×A→ [ 0 , 1 ] , cumulant c : S×A×S→R and continuation function γ : S×A×S → [ 0 , 1 ] , with ct+1 def = c ( St , At , St+1 ) and γt+1 def = γ ( St , At , St+1 ) , with values V ( s ) def = Eπ [ ct+1 + γt+1ct+2 + γt+1γt+2ct+3 + . . . |St = s ] = Eπ [ ∞∑ i=1 ( i−1∏ j=0 γt+j ) ct+i|St = s ] . The operator Eπ indicates the actions are selected according to policy π for the expectation . GVFs encompass standard definitions of value functions , where the cumulant is a reward and the continuation function is a constant . Otherwise , they specify a broader set of value functions , that enable predictions about discounted sums of others signals into the future , when following a target policy π . These values are typically estimated using parametric function approximation , with parameters θ ∈ Rd defining approximate values Vθ ( s ) . In off-policy learning , transitions are sampled according to behaviour policy , rather than the target policy . To get an unbiased sample of an update to the parameters , the action probabilities need to be adjusted . Consider on-policy temporal difference ( TD ) learning , with update αtδt∇θVθ ( s ) for a given St = s , for stepsize αt ∈ R+ and TD-error δt def = Ct+1 +γt+1Vθ ( St+1 ) −Vθ ( s ) . If actions are instead sampled according to a behaviour policy µ : S × A → [ 0 , 1 ] , then we can use importance sampling ( IS ) to modify the update , giving the off-policy TD update αtρtδt∇θVθ ( s ) for IS ratio ρt def = π ( At|St ) µ ( At|St ) . Given state St = s , if µ ( a|s ) > 0 when π ( a|s ) > 0 , then the expected value of these two updates are equal . To see why , notice that Eµ [ αtρtδt∇θVθ ( s ) |St = s ] = αt∇θVθ ( s ) Eµ [ ρtδt|St = s ] and we have Eµ [ ρtδt|St = s ] = ∑ a∈A µ ( a|s ) E [ ρtδt|St = s , At = a ] = ∑ a∈A µ ( a|s ) π ( a|s ) µ ( a|s ) E [ δt|St = s , At = a ] = ∑ a∈A π ( a|s ) E [ δt|St = s , At = a ] = Eπ [ δt|St = s ] . Other on-policy updates can also be modified with IS ratios to adjust these action probabilities . Though unbiased , IS can be high-variance , and so weighted IS ratios are typically preferred . For a batch consisting of transitions { ( si , ai , si+1 , ci+1 , ρi ) } ni=1 , batch WIS uses a normalized estimate for the update . For example , an offline batch WIS TD algorithm would use update αt ρtδt∇θVθ ( s ) ∑n i=1 ρi . When learning online , an efficient WIS update is less straightforward , and has resulted in algorithms specialized to the tabular setting ( Precup et al. , 2001 ) or linear functions ( Mahmood et al. , 2014 ; Mahmood and Sutton , 2015 ) . We nonetheless use WIS as baseline , in the experiments and theory . 3 RESAMPLING STRATEGIES FOR OFF-POLICY POLICY EVALUATION . In this section , we introduce resampling , as an alternative to importance sampling for off-policy learning . We first introduce the algorithm , Importance Resampling ( IR ) . We then prove consistency and characterize the bias . We conclude with a discussion about its variance properties . Importance Resampling requires access to a buffer of samples , from which we can resample . Replaying experience from a buffer was introduced as a biologically plausible mechanism to reuse old experience ( Lin , 1992 ; 1993 ) , and has since become common for improving sample efficiency , particularly for control ( Mnih et al. , 2015 ; Schaul et al. , 2015b ) . In the simplest case—which we assume here—the buffer is a sliding window of the most recent n samples , { ( si , ai , si+1 , ci+1 , ρi ) } ti=t−n , at time step t > n. These samples are generated by taking actions according to behaviour µ , and so the tuples are generated with probability dµ ( s ) µ ( a|s ) P ( s′|s , a ) , where dµ : S → [ 0 , 1 ] is the stationary distribution for policy µ . The goal is to obtain samples instead according to dµ ( s ) π ( a|s ) P ( s′|s , a ) , as if we had taken actions according to policy π from state s ∼ dµ . This assumption—that states are still sampled from dµ—underlies most off-policy learning algorithms ; very few attempt to use IS to adjust probabilities dµ to dπ ( Precup et al. , 2001 ) . The IR algorithm is simple : resample a mini-batch of size k on each step t from the buffer of size n , proportionally to ρi in the buffer . Standard on-policy updates , such as on-policy TD or on-policy gradient TD , are then used on this resample . The key difference to IS and WIS is that the sampling distribution itself is corrected ( see Theorem C.1 ) , before the update , whereas IS and WIS correct the update itself . This small difference , however , can have larger ramifications practically , particularly with updates that accumulate in the parameters . We consider two variants of IR : with and without bias correction . For point tj sampled from the buffer , let ∆tj be the on-policy update for that transition . For example , for TD , ∆tj = δtj∇θVθ ( stj ) . The first step for either variant is to sample a mini-batch of size k from the buffer , proportionally to ρi , as described above . The standard IR update simply uses a mini-batch , whereas Bias-Corrected IR ( BC-IR ) pre-multiplies with the average ratio in the buffer ρ̄t = 1n ∑n i=1 ρi , IR : αt 1k k∑ j=1 ∆tj BC-IR : αtρ̄t 1k k∑ j=1 ∆tj . BC-IR negates bias introduced by the average ratio in the buffer deviating significantly from the true mean . For reasonably large buffers , ρ̄t will be close to 1 making IR and BC-IR have nearidentical updates . In practice , we find the two variants of IR perform similarly . Nonetheless , they do have different theoretical properties , particularly for small buffer sizes n , so we characterize both . Though BC-IR has better bias properties , IR is simpler—not requiring any modification to the updates—which may be more important than the small amount of bias introduced by the IR without bias correction . For this reason , we advocate for and analyze both variants . Across all results , we make the following assumption . Assumption 1 . Transition tuples Xi = ( Si , Ai , Si+1 ) are sampled i.i.d . according to the distribution p ( x = ( s , a , s′ ) ) = dµ ( s ) µ ( a|s ) P ( s′|s , a ) , for i = 1 , 2 , 3 , . . .. To distinguish expectations under p ( x ) = dµ ( s ) µ ( a|s ) P ( s′|s , a ) and q ( x ) = dµ ( s ) π ( a|s ) P ( s′|s , a ) , we overload the notation from above , using operators Eµ and Eπ respectively . To reduce clutter , we will typically write E to mean Eµ , because most expectations are under the sampling distribution .

Summary: This paper introduces the concept of Sampling Importance Resampling (SIR) and give a simple method to adjust the off-policyness in the TD update rule of (general) value function learning, as an alternative of importance sampling. The authors argue that this resampling technique has several advantages over IS, especially on the stability with respect to step-size if we are doing optimization based the reweighted/resampled samples. In experiment section they show the sensitivity to learning rate of IR TD learning is closer to the on-policy TD learning, comparing with using IS or WIS.