Learning the Arrow of Time for Problems in Reinforcement Learning

1 INTRODUCTION . The asymmetric progression of time has a profound effect on how we , as agents , perceive , process and manipulate our environment . Given a sequence of observations of our familiar surroundings ( e.g . as video frames ) , we possess the innate ability to predict whether the said observations are ordered correctly . We use this ability not just to perceive , but also to act : for instance , we know to be cautious about dropping a vase , guided by the intuition that the act of breaking a vase can not be undone . This profound intuition reflects some fundamental properties of the world in which we dwell , and in this work we ask whether and how these properties can be exploited to learn a representation that functionally mimics our understanding of the asymmetric nature of time . The term Arrow of Time was coined by the British astronomer Eddington ( 1929 ) to denote this inherent asymmetry , which he attributed to the non-decreasing nature of the total thermodynamic entropy of an isolated system , as required by the second law of thermodynamics . Since then , the notion of an arrow of time has been formalized and explored in various contexts , spanning not only physics , but also algorithmic information theory ( Zurek , 1989 ) , causal inference ( Janzing et al. , 2016 ) and time-series analysis ( Janzing , 2010 ; Bauer et al. , 2016 ) . Broadly , an arrow of time can be thought of as a function that monotonously increases as a system evolves in time . Expectedly , the notion of irreversibility plays a central role in the discourse . In statistical physics , it is posited that the arrow of time ( i.e . entropy production ) is driven by irreversible processes ( Prigogine , 1978 ; Seifert , 2012 ) . To understand how a notion of an arrow of time can be useful in the reinforcement learning context , consider the example of a cleaning robot tasked with moving a box across a room ( Amodei et al. , 2016 ) . The optimal way of successfully completing the task might involve the robot doing something disruptive , like knocking a vase over ( Fig 1 ) . Now on the one hand , such disruptions – or side-effects – might be difficult to recover from . In the extreme case , they might be virtually irreversible – say when the vase is broken . On the other hand , irreversibility implies that states with a larger number of broken vases tend to occur in the future , and one should therefore expect an arrow of time ( as a scalar function of the state ) to assign larger values to states with larger number of broken vases . An arrow of time should therefore quantify the amount of disorder in the environment , analogous to the entropy for isolated thermodynamical systems . Now , one possible application could be to detect and preempt such side-effects , for instance by penalizing policies that significantly increment the arrow of time by executing difficult-to-reverse transitions . But the utility of an arrow of time is more general : it serves as a directed measure of reachability . This can be seen by observing that it is more difficult to obtain order from disorder : it is , after all , difficult to reach a state with a vase intact from one with it broken , rather than vice versa . In this sense , we may say that a state is relatively unreachable from another state if an arrow of time assigns a lower value to the former . Further , a directed measure of reachability afforded by an arrow of time can be utilized for deriving an intrinsic reward signal to enable agents to learn complex skills in the absence of external rewards . To see how , consider that an agent tasked with reversing the arrow of time ( by creating order from disorder ) must in general learn complex skills to achieve its goal . Indeed , gluing together a broken vase will require the agent to learn an array of complex planning and motor skills , which is the ultimate goal of such intrinsic rewards . In summary , our contributions are the following . ( a ) We propose a simple objective to learn an arrow of time for a Markov ( Decision ) Process in a self-supervised manner , i.e . entirely from sampled environment trajectories and without external rewards . We call the resulting function ( acting on the state ) the h-potential , and demonstrate its utility and caveats for a selection of discrete and continuous environments . Moreover , we compare the learned h-potential to the free-energy functional of stochastic processes – the latter being a well-known notion of an arrow of time ( Jordan et al. , 1998 ) . While there exist prior work on detecting the arrow of time in videos ( Pickup et al. , 2014 ; Wei et al. , 2018 ) and time-series data ( Peters et al. , 2009 ; Bauer et al. , 2016 ) , we believe our work to be the first towards measuring it in the context of reinforcement learning . ( b ) We critically and transparently discuss the conceptually rich subtleties that arise before an arrow of time can be practically useful in the RL context . ( c ) We expose how the notions of reachability , safety and curiosity can be unified under the common framework afforded by a learned arrow of time . 2 THE h-POTENTIAL Motivated by the preceding discussion , our goal is to learn a function that quantifies the amount of disorder in a given environment state , where we say that irreversible state transitions increase disorder . In this sense , we seek a function ( of the state ) that is constant in expectation along fully reversible state transitions , but increase in expectation along state transitions that are less reversible . To that end , we begin by formally introducing this function , which we call the h-potential , as the solution to a functional optimization problem . Subsequently , we critically discuss a few conceptual roadblocks that must be cleared before such a function can be useful in the RL setting . 2.1 FORMALISM . Consider a Markov Decision Process ( a MDP , i.e . environment ) , and let S and A be its state and action spaces respectively . A policy π is a mapping from the state space to the space of distributions over actions . Given a state s ∈ S sampled from some initial state distribution p0 , we may sample an action a ∈ A from the policy π ( a|s ) , which in turn can be used to sample another state s′ ∈ S from the environment dynamics p ( s′|a , s ) . Iterating N more times for a fixed π , one obtains a sequence of states ( s0 , ... , st , ... , sN ) , which is a realization of the Markov chain ( a trajectory ) with transition probabilities pπ ( st+1|st ) = ∑ a∈A p ( st+1|st , a ) π ( a|st ) . We may now define a function hπ : S → R as the solution to the following functional objective : Jπ [ ĥ ] = Et∼U ( { 0 , ... , N−1 } ) EstEst+1|st [ ĥ ( st+1 ) − ĥ ( st ) |st ] + λT [ ĥ ] ; hπ = arg max ĥ Jπ [ ĥ ] ( 1 ) whereU ( A ) is the uniform distribution over any setA , EtEstEst+1|st is the expectation over all state transitions , λ is a scalar coefficient and T [ ĥ ] is a regularizing term that prevents ĥ from diverging within a finite domain . In words : the first term on the right hand side of the first equation above encourages hπ to increase in expectation along the sampled trajectories , whereas the second term controls this increase ; the two terms are balanced with a coefficient λ . Informally : if a state transition s → s′ is fully reversible , the probability of sampling it equals that of sampling the corresponding reverse transition , s′ → s. For such transitions , the pressure on hπ to increase along the forward transition ( s → s′ ) is compensated by the counter-pressure for it to increase along the reverse transition ( s′ → s ) , or equivalently , decrease along the forward transition . Along such transitions , we should therefore expect hπ to remain constant ( in expectation ) . Accordingly , if the forward transition were to be more likely ( i.e . if the transition is not fully reversible ) , we should expect hπ to increase ( in expectation ) in order to satisfy its objective . The regularizer T must be chosen to suit the problem at hand , and different choices result in solutions that have different characteristics1 . Possible choices for T include ( any combination of ) the negative of L2 norm −‖ĥ‖2 , and/or the following trajectory regularizer : T [ ĥ ] = −Et∼U ( { 0 , ... , N−1 } ) EstEst+1|st [ |ĥ ( st+1 ) − ĥ ( st ) | 2|st ] ( 2 ) Intuitively : while the solution hπ is required to increase in expectation along trajectories , the trajectory regularizer acts as an contrastive term by penalizing hπ for changing at all . With some effort , the problem defined in Eqn 1 can be approached analytically for toy Markov chains ( interested readers may refer to App A for a technical discussion ) . However , such analytical treatment becomes infeasible for more complex and larger-scale environments with unknown transition probabilities . To tackle such environments , we will cast the functional optimization problem in Eqn 1 to an optimization problem over the parameters of a deep neural network and solve it for a variety of discrete and continuous environments . 2.2 SUBTLETIES . In this section , we discuss two conceptually rich subtleties that determine the conditions under which the learned arrow of time ( h-potential ) can be useful in practice . The Role of a Policy . The first subtlety is rooted in the observation that the trajectories ( s0 , ... , sN ) are collected by a given but arbitrary policy . However , there may exist policies for which the resulting arrow of time is unnatural , perhaps even misleading . Consider for instance the actions of a practitioner of Kintsugi , the ancient Japanese art of repairing broken pottery . The corresponding policy2 might cause the environment to transition from a state where the vase is broken to one where it is not . If we learn the h-potential on such trajectories , it might be the case that counter to our intuition , states with a larger number of broken vases are assigned smaller values ( and the vice versa ) . Now , one may choose to resolve this conundrum by defining : J [ h ] = Eπ∼U ( Π ) Jπ [ h ] ( 3 ) where Π is the set of all policies defined on S , and U ( Π ) denotes a uniform distribution over Π . The resulting function h∗ = arg max { J [ h ] + λT [ h ] } would characterize the arrow of time with respect to all possible policies , and one would expect that for a vast majority of such policies , the transition from broken vase to a intact vase is rather unlikely and/or requires highly specialized policies . 1This is not unlike the case for linear regression : for instance , using Lasso instead of ridge-regression will generally yield solutions that have different properties . 2This is analogous to Maxwell ’ s demon in classical thermodynamics . Unfortunately , determining h∗ is not feasible for most interesting applications , given the outer expectation over all possible policies . As a compromise , we use ( uniformly ) random actions to gather trajectories . The simplicity of the corresponding random policy justifies its adoption , since one would expect a policy resembling ( say ) a Kintsugi artist to be rather complex and not implementable with random actions . In this sense , we ensure that the learned arrow of time characterizes the underlying dynamics of the environment , and not the peculiarities of a particular agent3 . The price we pay is the lack of adequate exploration in complex enough environments , although this problem plagues most model-based reinforcement learning approaches4 ( cf . Ha & Schmidhuber ( 2018 ) ) . In the following , we assume π to be uniformly random and use hπ interchangeably with h. Dissipative Environments . The second subtlety concerns what we require of environments in which the arrow of time is informative . To illustrate the matter , consider the class of systems5 , a typical instance of which could be a billiard ball moving on a frictionless arena and bouncing ( elastically ) off the edges ( Bunimovich , 2007 ) . The state space comprises the ball ’ s velocity and its position constrained to a billiard table ( without holes ! ) , where the ball is initialized at a random position on the table . For such a system , it can be seen by time-reversal symmetry that when averaged over a large number of trajectories , the state transition s → s′ is just as likely as the reverse transition s′ → s. In this case , recall that the arrow of time is expected to remain constant . A similar argument can be made for systems that identically follow closed trajectories in their respective state space ( e.g . a frictionless and undriven pendulum ) . It follows that the h-potential must remain constant along the trajectory and that the arrow of time is uninformative . However , for so-called dissipative systems , the notion of an arrow of time is pronounced and well studied ( Willems , 1972 ; Prigogine , 1978 ) . In MDPs , dissipative behaviour may arise in situations where certain transitions are irreversible by design ( e.g . bricks disappearing in Atari Breakout ) , or due to partial observability , e.g . for a damped pendulum , the state space does not track the microscopic processes that give rise to friction6 . Therefore , a central premise underlying the practical utility of learning the arrow of time is that the considered MDP is indeed dissipative , which we shall assume in the following ; in Sec 5 ( Fig 5b ) , we will empirically investigate the case where this assumption is violated .

Summary: This paper proposes that we learn the “arrow of time” for an MDP: that is, a function (called the h-potential) that tends to increase as the MDP steps forward. Such an arrow should automatically capture notions such as irreversibility, and so can be used to define a measure of reachability, which previous work has shown can be used to penalize the agent for causing negative side effects. In addition, it can be used as intrinsic motivation for the agent: in particular, the agent can be rewarded for trajectories that decrease the h-potential (i.e. are “like” going backwards in time, or reducing entropy), which is hard to do and should lead to interesting skills. They propose that we learn the arrow of time by optimizing a function to grow over time along trajectories take from a random policy. Experiments demonstrate that in simple environments the learned function has the properties we would expect it to given results from physics.