GenDICE: Generalized Offline Estimation of Stationary Values

1 INTRODUCTION . Estimation of quantities defined by the stationary distribution of a Markov chain lies at the heart of many scientific and engineering problems . Famously , the steady-state distribution of a random walk on the World Wide Web provides the foundation of the PageRank algorithm ( Langville & Meyer , 2004 ) . In many areas of machine learning , Markov chain Monte Carlo ( MCMC ) methods are used to conduct approximate Bayesian inference by considering Markov chains whose equilibrium distribution is a desired posterior ( Andrieu et al. , 2002 ) . An example from engineering is queueing theory , where the queue lengths and waiting time under the limiting distribution have been extensively studied ( Gross et al. , 2018 ) . As we will also see below , stationary distribution quantities are of fundamental importance in reinforcement learning ( RL ) ( e.g. , Tsitsiklis & Van Roy , 1997 ) . Classical algorithms for estimating stationary distribution quantities rely on the ability to sample next states from the current state by directly interacting with the environment ( as in on-line RL or MCMC ) , or even require the transition probability distribution to be given explicitly ( as in PageRank ) . Unfortunately , these classical approaches are inapplicable when direct access to the environment is not available , which is often the case in practice . There are many practical scenarios where a collection of sampled trajectories is available , having been collected off-line by an external mechanism that chose states and recorded the subsequent next states . Given such data , we still wish to estimate a stationary quantity . One important example is off-policy policy evaluation in RL , where we wish to estimate the value of a policy different from that used to collect experience . Another example is off-line PageRank ( OPR ) , where we seek to estimate the relative importance of webpages given a sample of the web graph . Motivated by the importance of these off-line scenarios , and by the inapplicability of classical methods , we study the problem of off-line estimation of stationary values via a stationary distribution corrector . Instead of having access to the transition probabilities or a next-state sampler , we assume only access to a fixed sample of state transitions , where states have been sampled from an unknown distribution and next-states are sampled according to the Markov chain ’ s transition operator . The ∗Equal contribution . †Work done while interning at Google . off-line setting is indeed more challenging than its more traditional on-line counterpart , given that one must infer an asymptotic quantity from finite data . Nevertheless , we develop techniques that still allow consistent estimation under general conditions , and provide effective estimates in practice . The main contributions of this work are : • We formalize the problem of off-line estimation of stationary quantities , which captures a wide range of practical applications . • We propose a novel stationary distribution estimator , GenDICE , for this task . The resulting algorithm is based on a new dual embedding formulation for divergence minimization , with a carefully designed mechanism that explicitly eliminates degenerate solutions . • We theoretically establish consistency and other statistical properties of GenDICE , and empirically demonstrate that it achieves significant improvements on several behavior-agnostic offpolicy evaluation benchmarks and an off-line version of PageRank . The methods we develop in this paper fundamentally extend recent work in off-policy policy evaluation ( Liu et al. , 2018 ; Nachum et al. , 2019 ) by introducing a new formulation that leads to a more general , and as we will show , more effective estimation method . 2 BACKGROUND . We first introduce off-line PageRank ( OPR ) and off-policy policy evaluation ( OPE ) as two motivating domains , where the goal is to estimate stationary quantities given only off-line access to a set of sampled transitions from an environment . Off-line PageRank ( OPR ) The celebrated PageRank algorithm ( Page et al. , 1999 ) defines the ranking of a web page in terms of its asymptotic visitation probability under a random walk on the ( augmented ) directed graph specified by the hyperlinks . If we denote the World Wide Web by a directed graph G = ( V , E ) with vertices ( web pages ) v ∈ V and edges ( hyperlinks ) ( v , u ) ∈ E , PageRank considers the random walk defined by the Markov transition operator v → u : P ( u|v ) = ( 1−η ) |v| 1 ( v , u ) ∈E + η |V | , ( 1 ) where |v| denotes the out-degree of vertex v and η ∈ [ 0 , 1 ) is a probability of “ teleporting ” to any page uniformly . Define dt ( v ) : = P ( st = v|s0 ∼ µ0 , ∀i < t , si+1 ∼ P ( ·|si ) ) , where µ0 is the initial distribution over vertices , then the original PageRank algorithm explicitly iterates for the limit d ( v ) : = { limt→∞ dt ( v ) if γ = 1 ( 1− γ ) ∑∞ t=0 γ tdt ( v ) if γ ∈ ( 0 , 1 ) . ( 2 ) The classical version of this problem is solved by tabular methods that simulate Equation 1 . However , we are interested in a more scalable off-line version of the problem where the transition model is not explicitly given . Instead , consider estimating the rank of a particular web page v′ from a large web graph , given only a sampleD = { ( v , u ) i } N i=1 from a random walk on G as specified above . We would still like to estimate d ( v′ ) based on this data . First , note that if one knew the distribution p by which any vertex v appeared in D , the target quantity could be re-expressed by a simple importance ratio d ( v′ ) = Ev∼p [ d ( v ) p ( v ) 1v=v′ ] . Therefore , if one had the correction ratio function τ ( v ) = d ( v ) p ( v ) , an estimate of d ( v′ ) can easily be recovered via d ( v′ ) ≈ p̂ ( v′ ) τ ( v′ ) , where p̂ ( v′ ) is the empirical probability of v′ estimated from D. The main attack on the problem we investigate is to recover a good estimate of the ratio function τ . Policy Evaluation An important generalization of this stationary value estimation problem arises in RL in the form of policy evaluation . Consider a Markov Decision Process ( MDP ) M = 〈S , A , P , R , γ , µ0〉 ( Puterman , 2014 ) , where S is a state space , A is an action space , P ( s′|s , a ) denotes the transition dynamics , R is a reward function , γ ∈ ( 0 , 1 ] is a discounted factor , and µ0 is the initial state distribution . Given a policy , which chooses actions in any state s according to the probability distribution π ( ·|s ) , a trajectory β = ( s0 , a0 , r0 , s1 , a1 , r1 , . . . ) is generated by first sampling the initial state s0 ∼ µ0 , and then for t ≥ 0 , at ∼ π ( ·|st ) , rt ∼ R ( st , at ) , and st+1 ∼ P ( ·|st , at ) . The value of a policy π is the expected per-step reward defined as : Average : R ( π ) : = limT→∞ 1T+1E [ ∑T t=0 rt ] , Discounted : Rγ ( π ) : = ( 1−γ ) E [ ∑∞ t=0 γ trt ] . ( 3 ) In the above , the expectation is taken with respect to the randomness in the state-action pair P ( s′|s , a ) π ( a′|s′ ) and the reward R ( st , at ) . Without loss of generality , we assume the limit exists for the average case , and henceR ( π ) is finite . Behavior-agnostic Off-Policy Evaluation ( OPE ) An important setting of policy evaluation that often arises in practice is to estimateRγ ( π ) orR ( π ) given a fixed datasetD = { ( s , a , r , s′ ) i } N i=1 ∼ P ( s′|s , a ) p ( s , a ) , where p ( s , a ) is an unknown distribution induced by multiple unknown behavior policies . This problem is different from the classical form of OPE , where it is assumed that a known behavior policy πb is used to collect transitions . In the behavior-agnostic scenario , however , typical importance sampling ( IS ) estimators ( e.g. , Precup et al. , 2000 ) do not apply . Even if one can assume D consists of trajectories where the behavior policy can be estimated from data , it is known that that straightforward IS estimators suffer a variance exponential in the trajectory length , known as the “ curse of horizon ” ( Jiang & Li , 2016 ; Liu et al. , 2018 ) . Let dπt ( s , a ) = P ( st = s , at = a|s0 ∼ µ0 , ∀i < t , ai ∼ π ( ·|si ) , si+1 ∼ P ( ·|si , ai ) ) . The stationary distribution can then be defined as µπγ ( s , a ) : = { limT→∞ 1 T+1 ∑T t=0 d π t ( s , a ) = limt→∞ d π t ( s , a ) if γ = 1 ( 1− γ ) ∑∞ t=0 γ tdπt ( s , a ) if γ ∈ ( 0 , 1 ) . ( 4 ) With this definition , R ( π ) andRγ ( π ) can be equivalently re-expressed as Rγ ( π ) : = Eµπγ [ R ( s , a ) ] = Ep [ µπγ ( s , a ) p ( s , a ) R ( s , a ) ] . ( 5 ) Here we see once again that if we had the correction ratio function τ ( s , a ) = µπγ ( s , a ) p ( s , a ) a straightforward estimate of Rγ ( π ) could be recovered via Rγ ( π ) ≈ Ep̂ [ τ ( s , a ) R ( s , a ) ] , where p̂ ( s , a ) is an empirical estimate of p ( s , a ) . In this way , the behavior-agnostic OPE problem can be reduced to estimating the correction ratio function τ , as above . We note that Liu et al . ( 2018 ) and Nachum et al . ( 2019 ) also exploit Equation 5 to reduce OPE to stationary distribution correction , but these prior works are distinct from the current proposal in different ways . First , the inverse propensity score ( IPS ) method of Liu et al . ( 2018 ) assumes the transitions are sampled from a single behavior policy , which must be known beforehand ; hence that approach is not applicable in behavior-agnostic OPE setting . Second , the recent DualDICE algorithm ( Nachum et al. , 2019 ) is also a behavior-agnostic OPE estimator , but its derivation relies on a change-of-variable trick that is only valid for γ < 1 . This previous formulation becomes unstable when γ → 1 , as shown in Section 6 and Appendix E. The behavior-agnostic OPE estimator we derive below in Section 3 is applicable both when γ = 1 and γ ∈ ( 0 , 1 ) . This connection is why we name the new estimator GenDICE , for GENeralized stationary DIstribution Correction Estimation .

Summary: In this paper the authors proposed a framework for off-policy value estimation under the scenario of infinite horizon RL tasks. The new proposed method utilize the variational representation of $f$-divergence, which quantifies the difference between $\mathcal{T}\tau$ and $\tau p$, where $\tau$ is the parametric density ratio between the unknown behavior policy data and the target policy. If only if $\tau$ is the true density ratio, the loss $\mathcal{D}_{f}(\mathcal{T}\tau || \tau p) = 0$.