GenDICE: Generalized Offline Estimation of Stationary Values

1 INTRODUCTION . Estimation of quantities defined by the stationary distribution of a Markov chain lies at the heart of many scientific and engineering problems . Famously , the steady-state distribution of a random walk on the World Wide Web provides the foundation of the PageRank algorithm ( Langville & Meyer , 2004 ) . In many areas of machine learning , Markov chain Monte Carlo ( MCMC ) methods are used to conduct approximate Bayesian inference by considering Markov chains whose equilibrium distribution is a desired posterior ( Andrieu et al. , 2002 ) . An example from engineering is queueing theory , where the queue lengths and waiting time under the limiting distribution have been extensively studied ( Gross et al. , 2018 ) . As we will also see below , stationary distribution quantities are of fundamental importance in reinforcement learning ( RL ) ( e.g. , Tsitsiklis & Van Roy , 1997 ) . Classical algorithms for estimating stationary distribution quantities rely on the ability to sample next states from the current state by directly interacting with the environment ( as in on-line RL or MCMC ) , or even require the transition probability distribution to be given explicitly ( as in PageRank ) . Unfortunately , these classical approaches are inapplicable when direct access to the environment is not available , which is often the case in practice . There are many practical scenarios where a collection of sampled trajectories is available , having been collected off-line by an external mechanism that chose states and recorded the subsequent next states . Given such data , we still wish to estimate a stationary quantity . One important example is off-policy policy evaluation in RL , where we wish to estimate the value of a policy different from that used to collect experience . Another example is off-line PageRank ( OPR ) , where we seek to estimate the relative importance of webpages given a sample of the web graph . Motivated by the importance of these off-line scenarios , and by the inapplicability of classical methods , we study the problem of off-line estimation of stationary values via a stationary distribution corrector . Instead of having access to the transition probabilities or a next-state sampler , we assume only access to a fixed sample of state transitions , where states have been sampled from an unknown distribution and next-states are sampled according to the Markov chain ’ s transition operator . The ∗Equal contribution . †Work done while interning at Google . off-line setting is indeed more challenging than its more traditional on-line counterpart , given that one must infer an asymptotic quantity from finite data . Nevertheless , we develop techniques that still allow consistent estimation under general conditions , and provide effective estimates in practice . The main contributions of this work are : • We formalize the problem of off-line estimation of stationary quantities , which captures a wide range of practical applications . • We propose a novel stationary distribution estimator , GenDICE , for this task . The resulting algorithm is based on a new dual embedding formulation for divergence minimization , with a carefully designed mechanism that explicitly eliminates degenerate solutions . • We theoretically establish consistency and other statistical properties of GenDICE , and empirically demonstrate that it achieves significant improvements on several behavior-agnostic offpolicy evaluation benchmarks and an off-line version of PageRank . The methods we develop in this paper fundamentally extend recent work in off-policy policy evaluation ( Liu et al. , 2018 ; Nachum et al. , 2019 ) by introducing a new formulation that leads to a more general , and as we will show , more effective estimation method . 2 BACKGROUND . We first introduce off-line PageRank ( OPR ) and off-policy policy evaluation ( OPE ) as two motivating domains , where the goal is to estimate stationary quantities given only off-line access to a set of sampled transitions from an environment . Off-line PageRank ( OPR ) The celebrated PageRank algorithm ( Page et al. , 1999 ) defines the ranking of a web page in terms of its asymptotic visitation probability under a random walk on the ( augmented ) directed graph specified by the hyperlinks . If we denote the World Wide Web by a directed graph G = ( V , E ) with vertices ( web pages ) v ∈ V and edges ( hyperlinks ) ( v , u ) ∈ E , PageRank considers the random walk defined by the Markov transition operator v → u : P ( u|v ) = ( 1−η ) |v| 1 ( v , u ) ∈E + η |V | , ( 1 ) where |v| denotes the out-degree of vertex v and η ∈ [ 0 , 1 ) is a probability of “ teleporting ” to any page uniformly . Define dt ( v ) : = P ( st = v|s0 ∼ µ0 , ∀i < t , si+1 ∼ P ( ·|si ) ) , where µ0 is the initial distribution over vertices , then the original PageRank algorithm explicitly iterates for the limit d ( v ) : = { limt→∞ dt ( v ) if γ = 1 ( 1− γ ) ∑∞ t=0 γ tdt ( v ) if γ ∈ ( 0 , 1 ) . ( 2 ) The classical version of this problem is solved by tabular methods that simulate Equation 1 . However , we are interested in a more scalable off-line version of the problem where the transition model is not explicitly given . Instead , consider estimating the rank of a particular web page v′ from a large web graph , given only a sampleD = { ( v , u ) i } N i=1 from a random walk on G as specified above . We would still like to estimate d ( v′ ) based on this data . First , note that if one knew the distribution p by which any vertex v appeared in D , the target quantity could be re-expressed by a simple importance ratio d ( v′ ) = Ev∼p [ d ( v ) p ( v ) 1v=v′ ] . Therefore , if one had the correction ratio function τ ( v ) = d ( v ) p ( v ) , an estimate of d ( v′ ) can easily be recovered via d ( v′ ) ≈ p̂ ( v′ ) τ ( v′ ) , where p̂ ( v′ ) is the empirical probability of v′ estimated from D. The main attack on the problem we investigate is to recover a good estimate of the ratio function τ . Policy Evaluation An important generalization of this stationary value estimation problem arises in RL in the form of policy evaluation . Consider a Markov Decision Process ( MDP ) M = 〈S , A , P , R , γ , µ0〉 ( Puterman , 2014 ) , where S is a state space , A is an action space , P ( s′|s , a ) denotes the transition dynamics , R is a reward function , γ ∈ ( 0 , 1 ] is a discounted factor , and µ0 is the initial state distribution . Given a policy , which chooses actions in any state s according to the probability distribution π ( ·|s ) , a trajectory β = ( s0 , a0 , r0 , s1 , a1 , r1 , . . . ) is generated by first sampling the initial state s0 ∼ µ0 , and then for t ≥ 0 , at ∼ π ( ·|st ) , rt ∼ R ( st , at ) , and st+1 ∼ P ( ·|st , at ) . The value of a policy π is the expected per-step reward defined as : Average : R ( π ) : = limT→∞ 1T+1E [ ∑T t=0 rt ] , Discounted : Rγ ( π ) : = ( 1−γ ) E [ ∑∞ t=0 γ trt ] . ( 3 ) In the above , the expectation is taken with respect to the randomness in the state-action pair P ( s′|s , a ) π ( a′|s′ ) and the reward R ( st , at ) . Without loss of generality , we assume the limit exists for the average case , and henceR ( π ) is finite . Behavior-agnostic Off-Policy Evaluation ( OPE ) An important setting of policy evaluation that often arises in practice is to estimateRγ ( π ) orR ( π ) given a fixed datasetD = { ( s , a , r , s′ ) i } N i=1 ∼ P ( s′|s , a ) p ( s , a ) , where p ( s , a ) is an unknown distribution induced by multiple unknown behavior policies . This problem is different from the classical form of OPE , where it is assumed that a known behavior policy πb is used to collect transitions . In the behavior-agnostic scenario , however , typical importance sampling ( IS ) estimators ( e.g. , Precup et al. , 2000 ) do not apply . Even if one can assume D consists of trajectories where the behavior policy can be estimated from data , it is known that that straightforward IS estimators suffer a variance exponential in the trajectory length , known as the “ curse of horizon ” ( Jiang & Li , 2016 ; Liu et al. , 2018 ) . Let dπt ( s , a ) = P ( st = s , at = a|s0 ∼ µ0 , ∀i < t , ai ∼ π ( ·|si ) , si+1 ∼ P ( ·|si , ai ) ) . The stationary distribution can then be defined as µπγ ( s , a ) : = { limT→∞ 1 T+1 ∑T t=0 d π t ( s , a ) = limt→∞ d π t ( s , a ) if γ = 1 ( 1− γ ) ∑∞ t=0 γ tdπt ( s , a ) if γ ∈ ( 0 , 1 ) . ( 4 ) With this definition , R ( π ) andRγ ( π ) can be equivalently re-expressed as Rγ ( π ) : = Eµπγ [ R ( s , a ) ] = Ep [ µπγ ( s , a ) p ( s , a ) R ( s , a ) ] . ( 5 ) Here we see once again that if we had the correction ratio function τ ( s , a ) = µπγ ( s , a ) p ( s , a ) a straightforward estimate of Rγ ( π ) could be recovered via Rγ ( π ) ≈ Ep̂ [ τ ( s , a ) R ( s , a ) ] , where p̂ ( s , a ) is an empirical estimate of p ( s , a ) . In this way , the behavior-agnostic OPE problem can be reduced to estimating the correction ratio function τ , as above . We note that Liu et al . ( 2018 ) and Nachum et al . ( 2019 ) also exploit Equation 5 to reduce OPE to stationary distribution correction , but these prior works are distinct from the current proposal in different ways . First , the inverse propensity score ( IPS ) method of Liu et al . ( 2018 ) assumes the transitions are sampled from a single behavior policy , which must be known beforehand ; hence that approach is not applicable in behavior-agnostic OPE setting . Second , the recent DualDICE algorithm ( Nachum et al. , 2019 ) is also a behavior-agnostic OPE estimator , but its derivation relies on a change-of-variable trick that is only valid for γ < 1 . This previous formulation becomes unstable when γ → 1 , as shown in Section 6 and Appendix E. The behavior-agnostic OPE estimator we derive below in Section 3 is applicable both when γ = 1 and γ ∈ ( 0 , 1 ) . This connection is why we name the new estimator GenDICE , for GENeralized stationary DIstribution Correction Estimation .

Summary: This paper proposes a new estimator to infer the stationary distribution of a Markov chain, with data from another Markov chain. The method estimates the ratio between stationary distribution of target MC and the empirical data distribution.  It is based on the observation that the ratio is a fixed point solution to certain operators. The proposed method could work in the behavior-agnostic and undiscounted case, which is unsolved by the previous method.