There Is No Turning Back: A Self-Supervised Approach for Reversibility-Aware Reinforcement Learning

1 Introduction . We address the problem of estimating if and how easily actions can be reversed in the Reinforcement Learning ( RL ) context . Irreversible outcomes are often not to be taken lightly when making decisions . As humans , we spend more time evaluating the outcomes of our actions when we know they are irreversible [ 29 ] . As such , irreversibility can be positive ( i.e . takes risk away for good ) or negative ( i.e . leads to later regret ) . Also , decision-makers are more likely to anticipate regret for hard-to-reverse decisions [ 50 ] . All in all , irreversibility seems to be a good prior to exploit for more principled decision-making . In this work , we explore the option of using irreversibility to guide decision-making and confirm the following assertion : by estimating and factoring reversibility in the action selection process , safer behaviors emerge in environments with intrinsic risk factors . In addition to this , we ∗Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . show that exploiting reversibility leads to more efficient exploration in environments with undesirable irreversible behaviors , including the famously difficult Sokoban puzzle game . However , estimating the reversibility of actions is no easy feat . It seemingly requires a combination of planning and causal reasoning in large dimensional spaces . We instead opt for another , simpler approach ( see Fig . 1 ) : we propose to learn in which direction time flows between two observations , directly from the agents ’ experience , and then consider irreversible the transitions that are assigned a temporal direction with high confidence . In fine , we reduce reversibility to a simple classification task that consists in predicting the temporal order of events . Our contributions are the following : 1 ) we formalize the link between reversibility and precedence estimation , and show that reversibility can be approximated via temporal order , 2 ) we propose a practical algorithm to learn temporal order in a self-supervised way , through simple binary classification using sampled pairs of observations from trajectories , 3 ) we propose two novel exploration and control strategies that incorporate reversibility , and study their practical use for directed exploration and safe RL , illustrating their relative merits in synthetic as well as more involved tasks such as Sokoban puzzles . 2 Related Work To the best of our knowledge , this work is the first to explicitly model the reversibility of transitions and actions in the context of RL , using temporal ordering to learn from trajectories in a selfsupervised way , in order to guide exploration and control . Yet , several aspects of the problem we tackle were studied in different contexts , with other motivations ; we review these here . Leveraging reversibility in RL . Kruusmaa et al . [ 26 ] estimate the reversibility of stateaction couples so that robots avoid performing irreversible actions , since they are more likely to damage the robot itself or its environment . A shortcoming of their approach is that they need to collect explicit state-action pairs and their reversal actions , which makes it hard to scale to large environments . Several works [ 40 , 5 , 4 ] use reachability as a curiosity bonus for exploration : if the current state has a large estimated distance to previous states , it means that it is novel and the agent should be rewarded . Reachability and reversibility are related , in the sense that irreversible actions lead to states from which previous states are unreachable . Nevertheless , their motivations and ours diverge , and we learn reversibility through a less involved task than that of learning reachability . Nair et al . [ 33 ] learn to reverse trajectories that start from a goal state so as to generate realistic trajectories that reach similar goals . In contrast , we use reversibility to direct exploration and/or control , not for generating learning data . Closest to our work , Rahaman et al . [ 37 ] propose to learn a potential function of the states that increases with time , which can detect irreversibility to some extent . A drawback of the approach is that the potential function is learned using trajectories sampled from a random policy , which is a problem for many tasks where a random agent might fail to cover interesting parts of the state space . In comparison , our method does not use a potential function and learns jointly with the RL agent , which makes it a viable candidate for more complex tasks . Safe exploration . Safe exploration aims at making sure that the actions of RL agents do not lead to negative or unrecoverable effects that would outweigh the long-term value of exploration [ 2 ] . Notably , previous works developed distinct approaches to avoid irreversible behavior : by incremental updates to safe policies [ 23 , 18 ] , which requires knowing such a policy in advance ; by restricting policy search to ergodic policies [ 32 ] ( i.e . that can always come back to any state visited ) , which is costly ; by active exploration [ 28 ] , where the learner can ask for rollouts instead of exploring potentially unsafe areas of the state space itself ; and by computing regions of attraction [ 9 ] ( the part of the state space where a controller can bring the system back to an equilibrium point ) , which requires prior knowledge of the environment dynamics . Self-supervision from the arrow of time . Self-supervision has become a central component of modern machine learning algorithms , be it for computer vision , natural language or signal processing . In particular , using temporal consistency as a source of self-supervision is now ubiquitous , be it to learn representations for downstream tasks [ 19 , 38 , 12 ] , or to learn to detect temporal inconsistencies [ 47 ] . The closest analogies to our work are methods that specifically estimate some aspects of the arrow of time as self-supervision . Most are to be found in the video processing literature , and self-supervised tasks include predicting which way the time flows [ 35 , 47 ] , verifying the temporal order of a subset of frames [ 30 ] , predicting which video clip has the wrong temporal order among a subset [ 17 ] as well as reordering shuffled frames or clips from the video [ 16 , 14 , 48 ] . Bai et al . [ 6 ] notably propose to combine several of these pretext tasks along with data augmentation for video classification . Using time as a means of supervision was also explored for image sequencing [ 8 ] , audio [ 11 ] or EEG processing [ 39 ] . In RL , self-supervision also gained momentum in recent years [ 22 , 44 , 49 ] , with temporal information being featured [ 1 ] . Notably , several works [ 3 , 13 , 21 , 43 ] leverage temporal consistency to learn useful representations , effectively learning to discriminate between observations that are temporally close and observations that are temporally distant . In comparison to all these works , we estimate the arrow of time through temporal order prediction with the explicit goal of finding irreversible transitions or actions . 3 Reversibility . Degree of Reversibility . We start by introducing formally the notion of reversibility . Intuitively , an action is reversible if it can be undone , meaning that there is a sequence of actions that can bring us back to the original state . Definition 1 . Given a state s , we call degree of reversibility within K steps of an action a φK ( s , a ) : = sup π pπ ( s ∈ τt+1 : t+K+1 | st = s , at = a ) , and the degree of reversibility of an action is defined as φ ( s , a ) : = sup π pπ ( s ∈ τt+1 : ∞ | st = s , at = a ) , with τ = { si } i=1 ... T ∼ π corresponding to a trajectory , and τt : t′ the subset of the trajectory between the timesteps t and t′ ( excluded ) . We omit their dependency on π for the sake of conciseness . Given s ∈ S , the action a is reversible if and only if φ ( s , a ) = 1 , and said irreversible if and only if φ ( s , a ) = 0 . In deterministic environments , an action is either reversible or irreversible : given a state-action couple ( s , a ) and the unique resulting state s′ , φK ( s , a ) is equal to 1 if there is a sequence of less than K actions which brings the agent from s′ to s , and is otherwise equal to zero . In stochastic environments , a given sequence of actions can only reverse a transition up to some probability , hence the need for the notion of degree of reversibility . Policy-Dependent Reversibility . In practice , it is useful to quantify the degree of reversibility of an action as the agent acts according to a fixed policy π , for which we extend the notions introduced above . We simply write : φπ , K ( s , a ) : = pπ ( s ∈ τt+1 : t+K+1 | st = s , at = a ) and φπ ( s , a ) : = pπ ( s ∈ τt+1 : ∞ | st = s , at = a ) . It immediately follows that φK ( s , a ) = supπ φπ , K ( s , a ) and φ ( s , a ) = supπ φπ ( s , a ) . 4 Reversibility Estimation via Classification . Quantifying the exact degree of reversibility of actions is generally hard . In this section , we show that reversibility can be approximated efficiently using simple binary classification . 4.1 Precedence Estimation . Supposing that a trajectory contains the states s and s′ , we want to be able to establish precedence , that is predicting whether s or s′ comes first on average . It is a binary classification problem , which consists in estimating the quantity Est=s , st′=s′ [ 1t′ > t ] . Accordingly , we introduce the precedence estimator which , using a set of trajectories , learns to predict which state of an arbitrary pair is most likely to come first . Definition 2 . Given a fixed policy π , we define the finite-horizon precedence estimator between two states as follows : ψπ , T ( s , s ′ ) = Eτ∼π Est=s , st′=s′ t , t′ < T [ 1t′ > t ] . Conceptually , given two states s and s′ , the precedence estimator gives an approximate probability of s′ being visited after s , given that both s and s′ are observed in a trajectory . The indices are sampled uniformly within the specified horizon T ∈ N , so that this quantity is well-defined even for infinite trajectories . Additional properties of ψ , regarding transitivity for instance , can be found in Appx . A.2 . Remark 1 . The quantity ψπ , T ( s , s′ ) is only defined for pairs of states which can be found in the same trajectory , and is otherwise irrelevant . In what follows , we implicitly impose this condition when considering state pairs . Theorem 1 . For every policy π and s , s′ ∈ S , ψπ , T ( s , s′ ) converges when T goes to infinity . We refer to the limit as the precedence estimator , written ψπ ( s , s′ ) . The proof of this theorem is developed in Appendix A.3 . This result is key to ground theoretically the notion of empirical reversibility φ̄ , which we introduce in the next definition . It simply consists in extending the notion of precedence to a state-action pair . Definition 3 . We finally define the empirical reversibility using the precedence estimator : φ̄π ( s , a ) = Es′∼P ( s , a ) [ ψπ ( s ′ , s ) ] . In a nutshell , given that we start in s and take the action a , the empirical reversibility φ̄π ( s , a ) measures the probability that we go back to s , starting from a state s′ that follows ( s , a ) . We now show that our empirical reversibility is linked with the notion of reversibility defined in the previous section , and can behave as a useful proxy .

Summary: This paper introduces a self-supervised way to measure the reversibility of events. This is done by learning a classifier that takes as input two states from a trajectory and predicts which came first. The paper provides strong theoretical motivation for using this as a measure of reversibility. They then run experiments that use this reversibility metric in two ways: Reversibility-Aware Exploration (RAE), which penalizes the agent with a negative reward when it does an action that is not reversible enough; and Reversibility-Aware Control (RAC), which uses rejection sampling to ensure that actions sampled from the policy are reversible.   In the experiments, it is shown that: (1) reversibility is a strong intrinsic reward, capable of training a Cartpole agent without access to the true rewards; (2) in Sokoban, using IMPALA+RAE results in improvements in sample efficiency; and (3) using a classifier learned using offline data, they show RAC is capable of avoiding irreversible side effects during the whole training process in a custom grid world environment as well as enabling a Cartpole agent with a random policy to perform optimally using only rejection sampling to avoid irreversible actions.