C-Learning: Learning to Achieve Goals via Recursive Classification

1 INTRODUCTION . In this paper , we aim to reframe the goal-conditioned reinforcement learning ( RL ) problem as one of predicting and controlling the future state of the world . This reframing is useful not only because it suggests a new algorithm for goal-conditioned RL , but also because it explains a commonly used heuristic in prior methods , and suggests how to automatically choose an important hyperparameter . The problem of predicting the future amounts to learning a probability density function over future states , agnostic of the time that a future state is reached . The future depends on the actions taken by the policy , so our predictions should depend on the agent ’ s policy . While we could simply witness the future , and fit a density model to the observed states , we will be primarily interested in the following prediction question : Given experience collected from one policy , can we predict what states a different policy will visit ? Once we can predict the future states of a different policy , we can control the future by choosing a policy that effects a desired future . While conceptually similar to Q-learning , our perspective is different in that we make no reliance on reward functions . Instead , an agent can solve the prediction problem before being given a reward function , similar to models in model-based RL . Reward functions can require human supervision to construct and evaluate , so a fully autonomous agent can learn to solve this prediction problem before being provided any human supervision , and reuse its predictions to solve many different downstream tasks . Nonetheless , when a reward function is provided , the agent can estimate its expected reward under the predicted future state distribution . This perspective is different from prior approaches . For example , directly fitting a density model to future states only solves the prediction problem in the on-policy setting , precluding us from predicting where a different policy will go . Model-based approaches , which learn an explicit dynamics model , do allow us to predict the future state distribution of different policies , but require a reward function or distance metric to learn goal-reaching policies for controlling the future . Methods based on temporal difference ( TD ) learning ( Sutton , 1988 ) have been used to predict the future state distribution ( Dayan , 1993 ; 1Project website with videos and code : https : //ben-eysenbach.github.io/c_learning/ Szepesvari et al. , 2014 ; Barreto et al. , 2017 ) and to learn goal-reaching policies ( Kaelbling , 1993 ; Schaul et al. , 2015 ) . Section 3 will explain why these approaches do not learn a true Q function in continuous environments with sparse rewards , and it remains unclear what the learned Q function corresponds to . In contrast , our method will estimate a well defined classifier . Since it is unclear how to use Q-learning to estimate such a density , we instead adopt a contrastive approach , learning a classifier to distinguish “ future states ” from random states , akin to Gutmann & Hyvärinen ( 2010 ) . After learning this binary classifier , we apply Bayes ’ rule to obtain a probability density function for the future state distribution , thus solving our prediction problem . While this initial approach requires on-policy data , we then develop a bootstrapping variant for estimating the future state distribution for different policies . This bootstrapping procedure is the core of our goalconditioned RL algorithm . The main contribution of our paper is a reframing of goal-conditioned RL as estimating the probability density over future states . We derive a method for solving this problem , C-learning , which we use to construct a complete algorithm for goal-conditioned RL . Our reframing lends insight into goalconditioned Q-learning , leading to a hypothesis for the optimal ratio for sampling goals , which we demonstrate empirically . Experiments demonstrate that C-learning more accurately estimates the density over future states , while remaining competitive with recent goal-conditioned RL methods across a suite of simulated robotic tasks . 2 RELATED WORK . Common goal-conditioned RL algorithms are based on behavior cloning ( Ghosh et al. , 2019 ; Ding et al. , 2019 ; Gupta et al. , 2019 ; Eysenbach et al. , 2020 ; Lynch et al. , 2020 ; Oh et al. , 2018 ; Sun et al. , 2019 ) , model-based approaches ( Nair et al. , 2020 ; Ebert et al. , 2018 ) , Q-learning ( Kaelbling , 1993 ; Schaul et al. , 2015 ; Pong et al. , 2018 ) , and semi-parametric planning ( Savinov et al. , 2018 ; Eysenbach et al. , 2019 ; Nasiriany et al. , 2019 ; Chaplot et al. , 2020 ) . Most prior work on goalconditioned RL relies on manually-specified reward functions or distance metric , limiting the applicability to high-dimensional tasks . Our method will be most similar to the Q-learning methods , which are applicable to off-policy data . These Q-learning methods often employ hindsight relabeling ( Kaelbling , 1993 ; Andrychowicz et al. , 2017 ) , whereby experience is modified by changing the commanded goal . New goals are often taken to be a future state or a random state , with the precise ratio being a sensitive hyperparameter . We emphasize that our discussion of goal sampling concerns relabeling previously-collected experience , not on the orthogonal problem of sampling goals for exploration ( Pong et al. , 2018 ; Fang et al. , 2019 ; Pitis et al. , 2020 ) . Our work is closely related to prior methods that use TD-learning to predict the future state distribution , such as successor features ( Dayan , 1993 ; Barreto et al. , 2017 ; 2019 ; Szepesvari et al. , 2014 ) and generalized value functions ( Sutton & Tanner , 2005 ; Schaul et al. , 2015 ; Schroecker & Isbell , 2020 ) . Our approach bears a resemblance to these prior TD-learning methods , offering insight into why they work and how hyperparameters such as the goal-sampling ratio should be selected . Our approach differs in that it does not require a reward function or manually designed relabeling strategies , with the corresponding components being derived from first principles . While prior work on off-policy evaluation ( Liu et al. , 2018 ; Nachum et al. , 2019 ) also aims to predict the future state distribution , our work differs is that we describe how to control the future state distribution , leading to goal-conditioned RL algorithm . Our approach is similar to prior work on noise contrastive estimation ( Gutmann & Hyvärinen , 2010 ) , mutual-information based representation learning ( Oord et al. , 2018 ; Nachum et al. , 2018 ) , and variational inference methods ( Bickel et al. , 2007 ; Uehara et al. , 2016 ; Dumoulin et al. , 2016 ; Huszár , 2017 ; Sønderby et al. , 2016 ) . Like prior work on the probabilistic perspective on RL ( Kappen , 2005 ; Todorov , 2008 ; Theodorou et al. , 2010 ; Ziebart , 2010 ; Rawlik et al. , 2013 ; Ortega & Braun , 2013 ; Levine , 2018 ) , we treat control as a density estimation problem , but our main contribution is orthogonal : we propose a method for estimating the future state distribution , which can be used as a subroutine in both standard RL and these probabilistic RL methods . 3 PRELIMINARIES . We start by introducing notation and prior approaches to goal-conditioned RL . We define a controlled Markov process by an initial state distribution p1 ( s1 ) and dynamics function p ( st+1 | st , at ) . We control this process by a Markovian policy πθ ( at | st ) with parameters θ . We use πθ ( at | st , g ) to denote a goal-oriented policy , which is additionally conditioned on a goal g ∈ S . We use st+ to denote the random variable representing a future observation , defined by the following distribution : Definition 1 . The future γ−discounted state density function is pπ+ ( st+ | st , at ) , ( 1− γ ) ∞∑ ∆=1 γ∆pπ∆ ( st+∆ = st+ | st , at ) , where st+∆ denotes the state exactly ∆ in the future , and constant ( 1− γ ) ensures that this density function integrates to 1 . This density reflects the states that an agent would visit if we collected many infinite-length trajectories and weighted states in the near-term future more highly . Equivalently , p ( st+ ) can be seen as the distribution over terminal states we would obtain if we ( hypothetically ) terminated episodes at a random time step , sampled from a geometric distribution . We need not introduce a reward function to define the problems of predicting and controlling the future . In discrete state spaces , we can convert the problem of estimating the future state distribution into a RL problem by defining a reward function rst+ ( st , at ) = 1 ( st = st+ ) , and terminating the episode when the agent arrives at the goal . The Q-function , which typically represents the expected discounted sum of future rewards , can then be interpreted as a ( scaled ) probability mass function : Qπ ( st , at , st+ ) = Eπ [ ∑ t γtrst+ ( st , at ) ] = ∑ t γtPπ ( st = st+ ) = 1 1− γ pπ+ ( st+ | st , at ) . However , in continuous state spaces with some stochasticity in the policy or dynamics , the probability that any state exactly matches the goal state is zero . Remark 1 . In a stochastic , continuous environment , for any policy π the Q-function for the reward function rst+ = 1 ( st = st+ ) is always zero : Q π ( st , at , st+ ) = 0 . This Q-function is not useful for predicting or controlling the future state distribution . Fundamentally , this problem arises because the relationship between the reward function , the Q function , and the future state distribution in prior work remains unclear . Prior work avoids this issue by manually defining reward functions ( Andrychowicz et al. , 2017 ) or distance metrics ( Schaul et al. , 2015 ; Pong et al. , 2018 ; Zhao et al. , 2019 ; Schroecker & Isbell , 2020 ) . An alternative is to use hindsight relabeling , changing the commanded goal to be the goal actually reached . This form of hindsight relabeling does not require a reward function , and indeed learns Q-functions that are not zero ( Lin et al. , 2019 ) . However , taken literally , Q-functions learned in this way must be incorrect : they do not reflect the expected discounted reward . An alternative hypothesis is that these Q-functions reflect probability density functions over future states . However , this also can not be true : Remark 2 . For any MDP with the sparse reward function 1 ( st = st+ ) where the episode terminates upon reaching the goal , Q-learning with hindsight relabeling acquires a Q-function in the range Qπ ( st , at , st+ ) ∈ [ 0 , 1 ] , but the probability density function pπ+ ( st+ | st , at ) has a range [ 0 , ∞ ) . For example , if the state space is S = [ 0 , 12 ] , then there must exist some state st+ such that Qπ ( st , at , st+1 ) ≤ 1 < pπ+ ( st+ = st+ | st , at ) . See Appendix H for two worked examples . Thus , Q-learning with hindsight relabeling also fails to learn the future state distribution . In fact , it is unclear what quantity Q-learning with hindsight relabeling optimizes . In the rest of this paper , we will define goal reaching in continuous state spaces in a way that is consistent and admits well-defined solutions ( Sec . 4 ) , and then present a practical algorithm for finding these solutions ( Sec . 5 ) .

Summary: This paper studies to predict future state density function by using an indirectly method via classification. The main idea is to sample the future state from two sources: 1) from replay buffer, 2) the actual next state in the trajectory (in off policy setting we only need the next state) and then use a classifier to distinguish them. By Bayesian rule we can recalculate the conditional density function by the ratio of the classifier. The paper compare this method with several baseline and find that they can predict the conditional density function very close to reality.