Practical Marginalized Importance Sampling with the Successor Representation

1 INTRODUCTION . Off-policy evaluation ( OPE ) is a reinforcement learning ( RL ) task where the aim is to measure the performance of a target policy from data collected by a separate behavior policy ( Sutton & Barto , 1998 ) . As it can often be difficult or costly to obtain new data , OPE offers an avenue for re-using previously stored data , making it an important challenge for applying RL to real-world domains ( Zhao et al. , 2009 ; Mandel et al. , 2014 ; Swaminathan et al. , 2017 ; Gauci et al. , 2018 ) . Marginalized importance sampling ( MIS ) ( Liu et al. , 2018 ; Xie et al. , 2019 ; Nachum et al. , 2019a ) is a family of OPE methods which re-weight sampled rewards by directly learning the density ratio between the state-action occupancy of the target policy and the sampling distribution . This approach can have significantly lower variance than traditional importance sampling methods ( Precup et al. , 2001 ) , which consider a product of ratios over trajectories , and is amenable to deterministic policies and behavior agnostic settings where the sampling distribution is unknown . However , the body of MIS work is largely theoretical , and as a result , empirical evaluations of MIS have mostly been carried out on simple low-dimensional tasks , such as mountain car ( state dim . of 2 ) or cartpole ( state dim . of 4 ) . In comparison , deep RL algorithms have shown successful behaviors in high-dimensional domains such as Humanoid locomotion ( state dim . of 376 ) and Atari ( image-based ) . In this paper , we present a straightforward approach for MIS that can be computed from the successor representation ( SR ) of the target policy . Our algorithm , the Successor Representation DIstribution Correction Estimation ( SR-DICE ) , is the first method that allows MIS to scale to highdimensional systems , far outperforming previous approaches . In comparison to previous algorithms which rely on minimax optimization or kernel methods ( Liu et al. , 2018 ; Nachum et al. , 2019a ; Uehara & Jiang , 2019 ; Mousavi et al. , 2020 ) , SR-DICE requires only a simple convex loss applied to the linear function determining the reward , after computing the SR . Similar to the deep RL methods which can learn in high-dimensional domains , the SR can be computed easily using behavior-agnostic temporal-difference ( TD ) methods . This makes our algorithm highly amenable to deep learning architectures and applicable to complex tasks . Our derivation of SR-DICE also reveals an interesting connection between MIS methods and value function learning . The key motivation for MIS methods is , unlike traditional importance sampling methods , they can avoid variance with an exponential dependence on horizon , by re-weighting individual transitions rather than accumulating ratios along entire trajectories . We remark that while the MIS ratios only consider individual transitions , the optimization procedure is still subject to the dynamics of the underlying MDP . Subsequently , we use this insight to show a connection between a well-known MIS method , DualDICE ( Nachum et al. , 2019a ) , and Bellman residual minimization ( Bellman , 1957 ; Baird , 1995 ) , which can help explain some of the optimization properties and performance of DualDICE , as well as other related MIS methods . We benchmark the performance of SR-DICE on several high-dimensional domains in MuJoCo ( Todorov et al. , 2012 ) and Atari ( Bellemare et al. , 2013 ) , against several recent MIS methods . Our results demonstrate two key findings regarding high-dimensional tasks . SR-DICE significantly outperforms the benchmark algorithms . We attribute this performance gap to SR-DICE ’ s deep RL components , outperforming the MIS baselines in the same way that deep RL outperforms traditional methods on high-dimensional domains . Unfortunately , part of this performance gap is due to the fact that the baseline MIS methods scale poorly to challenging tasks . In Atari we find that the baseline MIS method exhibit unstable estimates , often reaching errors with many orders of magnitude . MIS underperforms deep RL . Although SR-DICE achieves a high performance , we find its errors are bounded by the quality of the SR. Consequently , we find that SR-DICE and the standard SR achieve a similar performance across all tasks . Worse so , we find that using a deep TD method , comparable to DQN ( Mnih et al. , 2015 ) for policy evaluation outperforms both methods . Although the performance gap is minimal , for OPE there lacks a convincing argument for SR-DICE , or any current MIS method , which introduce unnecessary complexity . However , this does not mean MIS is useless . We remark that the density ratios themselves are an independent objective which have been used for applications such as policy regularization ( Nachum et al. , 2019b ; Touati et al. , 2020 ) , imitation learning ( Kostrikov et al. , 2019 ) , off-policy policy gradients ( Imani et al. , 2018 ; Liu et al. , 2019b ; Zhang et al. , 2019 ) , and non-uniform sampling ( Sinha et al. , 2020 ) . SR-DICE serves as a stable , scalable approach for computing these ratios . We provide extensive experimental details in the supplementary material and our code is made available . 2 BACKGROUND . Reinforcement Learning . RL is a framework for maximizing accumulated reward of an agent interacting with its environment ( Sutton & Barto , 1998 ) . This problem is typically framed as a Markov Decision Process ( MDP ) ( S , A , R , p , d0 , γ ) , with state space S , action space A , reward function R , dynamics model p , initial state distribution d0 and discount factor γ . An agent selects actions according to a policy π : S ×A → [ 0 , 1 ] . In this paper we address the problem of off-policy evaluation ( OPE ) problem where the aim is to measure the normalized expected per-step reward of the policy R ( π ) = ( 1− γ ) Eπ [ ∑∞ t=0 γ tr ( st , at ) ] . An important notion in OPE is the value function Qπ ( s , a ) = Eπ [ ∑∞ t=0 γ tr ( st , at ) |s0 = s , a0 = a ] , which measures the expected sum of discounted rewards when following π , starting from ( s , a ) . We define dπ ( s , a ) as the discounted state-action occupancy , the probability of seeing ( s , a ) under policy π with discount γ : dπ ( s , a ) = ( 1 − γ ) ∑∞ t=0 γ t ∫ s0 d0 ( s0 ) pπ ( s0 → s , t ) π ( a|s ) d ( s0 ) , where pπ ( s0 → s , t ) is the probability of arriving at the state s after t time steps when starting from an initial state s0 . This distribution is important as R ( π ) equals the expected reward r ( s , a ) under dπ : R ( π ) = E ( s , a ) ∼dπ , r [ r ( s , a ) ] . ( 1 ) Successor Representation . The successor representation ( SR ) ( Dayan , 1993 ) of a policy is a measure of occupancy of future states . It can be viewed as a general value function that learns a vector of the expected discounted visitation for each state . The successor representation Ψπ of a given policy π is defined as Ψπ ( s′|s ) = Eπ [ ∑∞ t=0 γ t 1 ( st = s ′ ) |s0 = s ] . Importantly , the value function can be recovered from the SR by summing over the expected reward of each state V π ( s ) = ∑ s′ Ψ π ( s′|s ) Ea′∼π [ r ( s′ , a′ ) ] . For infinite state and action spaces , the SR can instead be generalized to the expected occupancy over features , known as the deep SR ( Kulkarni et al. , 2016 ) or successor features ( Barreto et al. , 2017 ) . For a given encoding function φ : S×A → Rn , the deep SR ψπ : S × A → Rn is defined as the expected discounted sum over features from the encoding function φ when starting from a given state-action pair and following π : ψπ ( s , a ) = Eπ [ ∞∑ t=0 γtφ ( st , at ) ∣∣∣∣s0 = s , a0 = a ] . ( 2 ) If the encoding φ ( s , a ) is learned such that the original reward function is a linear function of the encoding r ( s , a ) = w > φ ( s , a ) , then similar to the original formulation of SR , the value function can be recovered from a linear function of the SR : Qπ ( s , a ) = w > ψπ ( s , a ) . The deep SR network ψπ is trained to minimize the MSE between ψπ ( s , a ) and φ ( s , a ) + γψ′ ( s′ , a′ ) on transitions ( s , a , s′ ) sampled from the data set . A frozen target network ψ′ is used to provide stability ( Mnih et al. , 2015 ; Kulkarni et al. , 2016 ) , and is updated to the current network ψ′ ← ψπ after a fixed number of time steps . The encoding function φ is typically trained by an encoder-decoder network ( Kulkarni et al. , 2016 ; Machado et al. , 2017 ; 2018a ) . Marginalized Importance Sampling . Marginalized importance sampling ( MIS ) is a family of importance sampling approaches for off-policy evaluation in which the performance R ( π ) is evaluated by re-weighting rewards sampled from a data set D = { ( s , a , r , s′ ) } ∼ p ( s′|s , a ) dD ( s , a ) , where dD is an arbitrary distribution , typically but not necessarily , induced by some behavior policy . It follows that R ( π ) can computed with importance sampling weights on the rewards d π ( s , a ) dD ( s , a ) : R ( π ) = E ( s , a ) ∼dD , r [ dπ ( s , a ) dD ( s , a ) r ( s , a ) ] . ( 3 ) The goal of marginalized importance sampling methods is to learn the weights w ( s , a ) ≈ d π ( s , a ) dD ( s , a ) , using data contained in D. The main benefit of MIS is that unlike traditional importance methods , the ratios are applied to individual transitions rather than complete trajectories , which can reduce the variance of long or infinite horizon problems . In other cases , the ratios themselves can be used for a variety of applications which require estimating the occupancy of state-action pairs . DualDICE . Dual stationary DIstribution Correction Estimation ( DualDICE ) ( Nachum et al. , 2019a ) is a well-known MIS method which uses a minimax optimization to learn the density ratios . The underlying objective which DualDICE aims to minimize is the following : min f J ( f ) : = 1 2 E ( s , a ) ∼dD [ ( f ( s , a ) − γEs′ , π [ f ( s′ , a′ ) ] ) 2 ] − ( 1− γ ) Es0 , a0∼π [ f ( s0 , a0 ) ] . ( 4 ) It can be shown that Equation ( 4 ) is uniquely optimized by the MIS density ratio . However , since f ( s , a ) − γEπ [ f ( s′ , a′ ) ] is dependent on transitions ( s , a , s′ ) , there are two practical issues with this underlying objective . First , the objective contains a square within an expectation , giving rise to the double sampling problem ( Baird , 1995 ) , where the gradient will be biased when using only a single sample of ( s , a , s′ ) . Second , computing f ( s , a ) − γEs′ , π [ f ( s′ , a′ ) ] for arbitrary state-action pairs , particularly those not contained in the data set , is non-trivial , as it relies on an expectation over succeeding states , which is generally inaccessible without a model of the environment . To address both concerns , DualDICE uses Fenchel duality ( Rockafellar , 1970 ) to create the following minimax optimization problem : min f max w J ( f , w ) : = E ( s , a ) ∼dD , a′∼π , s′ [ w ( s , a ) ( f ( s , a ) − γf ( s′ , a′ ) ) − 0.5w ( s , a ) 2 ] − ( 1− γ ) Es0 , a0 [ f ( s0 , a0 ) ] . ( 5 ) Similar to the original formulation , Equation ( 4 ) , it can be shown that Equation ( 5 ) is minimized when w ( s , a ) is the desired density ratio .

Summary: The paper proposes an approach to employ successor representation combined with marginalized importance sampling. The basic idea exploited in the paper consists of expressing the occupancies in terms of the successor representation and to model it via a linear combination of some features. This allows handling, although approximately, continuous state-action spaces. After having derived the objective function, an experimental evaluation on both Mujoco and Atari domains is presented, including an ablation study.