There Is No Turning Back: A Self-Supervised Approach for Reversibility-Aware Reinforcement Learning
Authors: Nathan Grinsztajn
Johan Ferret
Olivier Pietquin
Philippe Preux
Matthieu Geist
Venue: NeurIPS 2021 Poster (2021)
Keywords: reinforcement learning
reversibility
exploration
safety

Abstract: We propose to learn to distinguish reversible from irreversible actions for better informed decision-making in Reinforcement Learning (RL). From theoretical considerations, we show that approximate reversibility can be learned through a simple surrogate task: ranking randomly sampled trajectory events in chronological order. Intuitively, pairs of events that are always observed in the same order are likely to be separated by an irreversible sequence of actions. 
Conveniently, learning the temporal order of events can be done in a fully self-supervised way, which we use to estimate the reversibility of actions from experience, without any priors.
We propose two different strategies that incorporate reversibility in RL agents, one strategy for exploration (RAE) and one strategy for control (RAC). We demonstrate the potential of reversibility-aware agents in several environments, including the challenging Sokoban game. In synthetic tasks, we show that we can learn control policies that never fail and reduce to zero the side-effects of interactions, even without access to the reward function.

There Is No Turning Back: A Self-Supervised Approach for Reversibility-Awe Aware Reinforcement Learning
Nathan Grinsztajn" Inria, Scool Team
Johan Ferret
Google Research, Brain Team
CRIStAL, CNRS, Université de Lille
Inria, Scool Team
nathan nathan.grinsztan wwwnsstajjniniia...........
CRIStAL, CNRS, Université de Lille
Olivier Pietquin
Philippe Preux
Google Research, Brain Team
Inria, Scool Team
CRIStAL, CNRS, Université de Lille
Matthieu Geist
Google Research, Brain Team
Abstract
We propose to learn to distinguish reversible from irreversible actions for better informed decision-makine makinn in Reinforcement Learning (RL). From theoretical considerations, we show that approximate reversibility can be learned through a simple surrogate task: ranking randomly sampled trajectory events in chronological order. Intuitively, pairs events that are always observed in the same order are likely to be separated by irreversible sequence of actions. Conveniently, learning the temporal order of events can be done in a fully self-supervised way, which we use estimate the reversibility of actions from experience, without any priors. We propose two different strategies that incorporate reversibility in RL agents, one strategy for exploration (RAE) and one strategy for control (RAC). We demonstrate the potential of reversibility-wwar agents in several environments, including the challenging Sokoban game. In synthetic tasks, we show that can learn control policies that never fail and reduce to zero the side-effects of interactions, even without access to the reward function.
1 Introduction
We address the problem of estimating if and how easily actions be reversed in the Reinforcement Learning (RL) context, Irreversible outcomes are often not to be taken lightly when making decisions. As humans, we spend more time evaluating the outcomes of our actions when we know they are irreversible [29]. As such, irreversibility can be positive (i.e. takes risk away for good) or negative (i.e. leads later regret). Also, sreiisionmmkern are more likely to anticipate regret for hard-to-reverse decisions [50]. All in all, irreversibility seems to be a good prior to exploit for more principled decision-make making. this work, we explore the option of using irreversibility guide decision making and confirm the following assertion: by estimating and factoring reversibility in the action selection process, safer behaviors emerge in environments with intrinsic risk factors. In addition to this, we
'Equal contribution.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

show that exploiting reversibility leads to more efficient exploration in environments with undesirable irreversible behaviors, including the famously difficult Sokoban puzzle game.
However, estimating the reversibility of actions is no easy feat. It seemingly requires a combination of planning and causal reasoning in large dimensional spaces. We instead opt for another, simpler approach (see Fig. I): we propose to learn in which direction time flows between two observations, directly from the agents' experience, and then consider irreversible the transitions that are assigned a lemporal direction with high confidence. In fine, we reduce reversibility to a simple classification task that consists in predicting the temporal order of events.
Our contributions are the following: 1) we formalize the link between reversibility and precedence es- timation, and show that reversibility can be approximated via temporal order, 2) we propose a practical algorithm to learn temporal order in a self-supervised way. through simple binary classification using sampled pairs of observations from trajectories, 3) we propose two novel exploration and control strate- gies that incorporate reversibility, and study their practical use for directed exploration and safe RI,, illustrating their relative merits in synthetic as well as more involved tasks such as Sokoban puzzles.
2 Related Work
To the best of our knowledge, this work is the first to explicitly model the reversibility of transi- tions and actions in the context of RL, using tem- poral ordering to learn from trajectories in self- supervised way, in order to guide exploration and control. Yet, several aspects of the prob- lem we tackle were studied in different contexts, with other motivations; we review these here.
1 A - B reversible?
reversible?
because
B - A does not contradict the laws of physics!
Exsy, since B -- is as likely B!
Leveraging reversibility in RI. Kruusmaa et al. [26] estimate the reversibility of state- action couples so that robots avoid performing irreversible actions, since they are more likely to damage the robot itself or its environment. A shortcoming of their approach is that they need to collect explicit state-action pairs and their reversal actions, which makes it hard to scale to large environments. Several works [40, 5, 4] use reachability as curiosity bonus for exploration: if the current state has a large estimated distance to previous states, it means that it is novel and the agent should be rewarded. Reachability and reversibility are related, in the sense that irreversible actions lead to states from which previous states are unreachable. Nevertheless, their motivations and ours diverge, and we learn reversibility through a less involved task than that of learning reachability. Nair et al. [33] learn to reverse trajectories that start from a goal state so as to generate realistic trajectories that reach similar goals. In contrast, we use reversibility
100301
113109
Figure I: High-level illustration of how reversibil- ity can be estimated. Left: from an understanding of physics. Right: ours, from experience.
direct exploration and/or control, not for generating learning data. Closest to our work, Rahaman et al. [37] propose to learn a potential function of the states that increases with time, which can detect irreversibility to some extent. A drawback of the approach is that the potential function is learned using trajectories sampled from a random policy, which is a problem for many tasks where a random agent might fail to cover interesting parts of the state space. In comparison, our method does not use potential function and learns jointly with the RL. agent, which makes a viable candidate for more complex tasks.
Safe exploration. Safe exploration aims at making sure that the actions of RL agents do not lead to negative or unrecoverable effects that would outweigh the long-term value of exploration [2]. Notably, previous works developed distinct approaches to avoid irreversible behavior: by incremental updates to safe policies [23, 18]. which requires knowing such policy in advance; by restricting policy search to ergodic policies [32] (i.e. that can always come back to any state visited), which is costly; by active exploration [28], where the learner can ask for rollouts instead of exploring potentially unsafe areas of the state space itself; and by computing regions of attraction [9] (the part of the state space where a controller can bring the system back to an equilibrium point), which requires prior knowledge of the environment dynamics.
2

---

Self-supervision from the arrow of time. Self-supervision has become central component of modern machine learning algorithms, be it for computer vision, natural language or signal processing. In particular, using temporal consistency
is now ubiquitous, be it to learn representations for downstream tasks [19, 38, 12), or learn to detect temporal inconsistencies [47]. The closest analogies to our work are methods that specifically estimate some aspects of the arrow of time as self-supervision. Most are to be found in the video processing literature, and self-supervised tasks include predicting which way the time flows [35, 47], verifying the temporal order a subset of frames [30], predicting which video clip has the wrong temporal order among a subset [17] as well as reordering shuffled frames or clips from the video [16, 14, 48]. Bai et al. [6] notably propose to combine several of these pretext tasks along with data augmentation for video classification. Using time means of supervision was also explored for image sequencing [8], audio [11] or EEG processing [39]. In RI., self-supervision also gained momentum in recent years [22, 44, 49], with temporal information heing featured [1]. Notably, several works [3, 11, 43] leverage temporal consistency to learn useful representations, effectively learning to discriminate between observations that are temporally close and observations that are temporally distant. In comparison to all these works, we estimate the arrow of time through temporal order prediction with the explicit goal of finding irreversible transitions or actions.
source of self-
3 Reversibility
Degree of Reversibility. We start by introducing formally the notion of reversibility. Intuitively, an action is reversible it can be undone, meaning that there is sequence of actions that can bring us back to the original state.
Definition 1. Given a state s, we call degree of reversibility within K steps of an action a
Φk(s,a) sup Pr (s E T++1*K++1 St 8, a, = a),
and the degree of reversibility of an action is defined as Φ(s, a) = sup Pm (s E Tt+1:00 8t = a, a)),
with {sili=1...7 T ~ π corresponding to a trajectory, and Tey the subset of the trajectory between the timesteps t and t' (excluded). We omit their dependency on π. for the sake of conciseness. Given E and said irreversible if and only if é a)
S, the action is reversible if and only if ((s a)
In deterministic environments, an action is either reversible or irreversible: given state-action couple (8, a) and the unique resulting state s', Φx (8, a) is equal to if there a sequence of less than K actions which brings the agent from s' to and is otherwise equal to zero. In stochastic environments, a given sequence of actions can only reverse transition up to some probability, hence the need for the notion of degree of reversibility.
Reversibility. In practice, it is useful to quantify the degree of reversibility of an action as the agent acts according to a fixed policy for which we extend the notions introduced above. We simply write:
(,, a) = Pa Te-L+K+1 8, a, = a) and d (s) p=(s E T2-1:00 18 =, a).
It immediately follows that (x(s,)) a) sup, 0x. (5, (s, a) and ((s.a) sup, (s)
4 Reversibility Estimation via Classification
Quantifying the exact degree of reversibility of actions is generally hard. In this section, we show that reversibility be approximated efficiently using simple binary classification.
4.1 Precedence Estimation
Supposing that trajectory contains the states and s', we want to be able to establish precedence, that is predicting whether s' comes first average. It is binary classification problem, which
3

---

consists in estimating the quantity Es,=s,ay= [[e> Accordingly, we introduce the precedence estimator which, using set of trajectories, learns predict which state of an arbitrary pair is most likely to come first.
Definition 2. Given a fixed policy we define the finite-horizon precedence estimator between two states as follows:
Va, 5') Erra -u==s t/
[ess].
Conceptually, given two states and 8', the precedence estimator gives an approximate probability of s' being visited after given that both and s' are observed in trajectory. The indices are sampled uniformly within the specified horizon T N, so that this quantity is well-defined even for infinite trajectories. Additional properties of v regarding transitivity for instance, can be found in Appx. A.2.
Remark 1. The quantity (8, 5 is only defined for pairs of states which can be found in the same trajectory, and otherwise irrelevant. what follows, we implicitly impose this condition when considering state pairs.
Theorem 1. For every policy and s' E S, (8,, 8') converges when T goes to infinity. We refer to the limit as the precedence estimator, written 1/2 (8,8') 8').
The proof of this theorem is developed in Appendix A.3. This result is key to ground theoretically the notion of empirical reversibility Φ, which we introduce in the next definition. It simply consists in extending the notion of precedence state-action pair.
Definition 3. We finally define the empirical reversibility using the precedence estimator:
(8, a) P(e.(().)) (s'
In a nutshell, given that we start in s and take the action a, the empirical reversibility Φn (8, a) measures the probability that go back s, starting from state s' that follows (s,a) a). We now show that our empirical reversibility is linked with the notion of reversibility defined the previous section, and can behave as useful proxy.
4.2 Estimating Reversibility from Precedence
We present here our main theoretical result which relates reversibility and empirical reversibility: Theorem 2. Given a policy æ, a state s and an action a, we have: (n(s, a) 2 (n(s,))
2
The full proof of the theorem is given in Appendix A.3.
This result theoretically justifies the name empirical reversibility. From a practical perspective, provides a way of using Φ detect actions which irreversible or hardly reversible: Φx (s,a) 1 implies Φx (s, a) << and thus provides sufficient condition to detect actions with low degrees of reversibility. This result gives way to detect actions that are irreversible given specific policy followed by the agent. Nevertheless, we are generally interested in knowing if these actions are irreversible for any policy, meaning o(s, a) < with the definition of Section 3. The next proposition makes an explicit connection between 6x and é, under the assumption that the policy ı is stochastic.
Proposition 1. We suppose that we are given a state s, an action a such that is reversible in K steps, and a policy n. Under the assumption that π is stochastic enough, meaning that there exists > O such that for every state and action s, a, T(a 8) > p. we have: 0 a) 2 2 Moreover, we xx
have for all K E N: (s, 2 e Φo (s.
The proof is given in Appendix A.4. As before, this proposition gives a practical way of detecting irreversible moves. If for example क (s, a) < p² /2 for some & E N, we can be sure that action is not reversible in & steps. The quantity p can understood as minimal probability of taking any action in any state. This condition is not very restrictive: e-greedy strategies for example satisfy this hypothesis with
In practice, it can also useful to limit the maximum number of time steps between two sampled states. That is why we also define the windowed precedence estimator as follows:

---

Embeddiing
y/1
Temporal Order Protability
Embedding
Shuffle
f
It
Concat
g
Z
V
SSL
fo
y/2
Tt-kk Categorical(I.a)
222
rendom order from shuffle, acts target
Figure 2: The proposed self- self-supervised procedure for precedence estimation.
Definition 4. Given a fixed policy Tr, we define the windowed precedence estimator between two states as follows:
Un Tn, Eyna E px [1ex]]
Intuitively, compared previous precedence estimators, T, is restricted to short-term dynamics, which is a desirable property in tasks where distinguishing the far future from the present is either trivial or impossible.
5
Reversibility-Awark Reinforcement Learning
Leveraging the seoretically-groundee bridge between precedence and reversibility established the previous section, used a practical setting.
explain how reversibility be learned from the agent's experience and
Learning to rank events chronologically. Learning which observation comes first in trajectory is achieved by binary supervised classification, from pairs of observations sampled uniformly in observed trajectories. This can be done fully offline, i.e. using a previously
sliding window collected dataset of trajectories for instance, or fully online, i.e. jointly with the learning the RL agent; but also anywhere the spectrum by leveraging variable amounts of offline and online data.
This procedure not without caveats. In particular, we want to avoid overfitting to the particularities of the behavior of the agent, so that we can learn meaningful, generalizable statistics about the order of events the task at hand. Indeed, if an agent always visits the state 8, before 8b, the classifier will probably assign a close-to-one probability that s. precedes sp. This might not be accurate with other agents equipped with different policies, unless transitioning from Sb to $a is hard due to the dynamics of the environment, which is in fact exactly the cases several assumptions about the agents we apply our method to: 1) agents are learning and thus, have a policy that changes through interactions in the environment, 2) agents have incentive deterministic. For this second assumption, we typically use an entropic regularization in the chosen RL loss, which is common design choice in modern RL methods. These assumptions, when put together, alleviate the risk of overfitting to the idiosyncrasies
want to uncover. We make
be too
single, non-representative policy.
We illustrate the precedence classification procedure in Fig. 2. A temporally-ordered pair of observa- tions, distant of no more than w timesteps, is sampled from a trajectory and uniformly shuffled. The result of the shuffling operation is memorized and used as target for the binary classification task. A Siamese network creates separate embeddings for the pair of observations, which are concatenated and fed to separate feed-forward network, whose output is passed through sigmoid to obtain probability of precedence. This probability updated via negative log- legllikllioodd against the result of the shuffle, so that it matches the actual temporal order.
Then, a transition (and its implicit sequence actions) represented by a starting observation æ and resulting observation x' is deemed irreversible if the estimated precedence probability ((x, x") superior to a chosen threshold 3. Note that we do not have take into account the temporal proximity of these two observations here, which is by-product of sampling observations uniformly
5

---

(a) RAE penalizes inrevesiibee transitions
(b) RAC hijacks irreversible actions
a2
Temporal Creer Probability
rp
Degree of Reversiblity
Φ(8t)> .)
Rejection sampling
' Concat
fg
ye++1
1/t
t
t
Zt
1+++1
Figure 3: Our proposed methods for reversibility-wwarc RL. (a): RAE encourages reversible behavior via auxiliary rewards. (b): RAC avoids irreversible behavior by rejecting actions whose estimated reversibility is inferior to a threshold.
in window in trajectories. Also, depending on the threshold B, we cover a wide range of scenarios, from pure irreversibility (3 close to 1) to soft irreversibility (3 0.5, the bigger 3. the harder the transition is to reverse). This is useful because different tasks call for different levels of tolerance for irreversible behavior: while a robot getting stuck and leading to an early experiment failure is to be avoided when possible, tasks involving human safety might call for absolute zero tolerance for irreversible decision-makei making. We elaborate on these aspects See. 6.
Reversibilit--Aware Exploration and Control. We propose two different algorithms based on reversibility estimation: Reversibility-Awe Exploration (RAE) and Reversibility-ANe Aware Control (RAC). We give high-level representation of how the two methods operate in Fig. 3.
In a nutshell, RAE consists in using the estimated reversibility of pair of consecutive observations to create an auxiliary reward function. In our experiments, the reward function a piecewise linear function of the estimated reversibility and fixed threshold, as in Fig. 3: it grants the agent negative reward if the transition is deemed too hard reverse. The agent optimizes the sum of the extrinsic and auxiliary rewards. Note that the specific function we use penalizes irreversible transitions but could encourage such transitions instead, if the task calls for it.
RAC can be seen as the action- ciniinnnoniiiooeed counterpart of RAE. From a single observation, RAC estimates the degree of reversibility of all available actions, and "takes control" if the action sampled from the policy is not reversible enough (i.e. has reversibility inferior to threshold (3). "Taking opt for rejection sampling: until an action that is reversible enough is sampled. This strategy has the advantage of avoiding irreversible actions entirely, while trading-off pure reversibility for performance when possible. RAC is more involved than RAE, since the action- condiiiond reversibility is learned from the supervision of a standard, also learned precedence estimator. Nevertheless, our experiments show that it is possible to learn both estimators jointly, at the cost of little overhead.
control" can have many forms. In practice,
sample from the policy
We now discuss the relative merits of the two methods. In terms of applications, argue that RAE is more suitable for directed exploration, as it only encourages reversible behavior. As a result, irreversible behavior is permitted if the benefits (i.e. rewards) outweigh the costs (i.e. irreversibility penalties). In contrast, RAC shines safety-first, real-world scenarios, where irreversible behavior is be banned entirely. With an optimal precedence estimator and task-dependent threshold, RAC will indeed hijack all irreversible sampled actions. RAC can be especially effective when pre-trained on offline trajectories: it is then possible to generate fully-reversible, safe behavior from the very first online interaction in the environment. We explore these possibilities experimentally in Sec. 6.2.
Both algorithms can be used online offline with small modifications to their overall logic. The pseudo-code for the online version of RAE and RAC can be found in Appendix B.2.
The self-supervised precedence classification task could have applications beyond estimating the reversibility of actions: it could be used as a means of getting additional learning signal or repre- sentational priors for the RL algorithm. Nevertheless, opt for clear separation between the

---

200
apisode length
intrinsic rewardd
0.001
1.00
0.3
0.9
0.0
-0.2
0.0
timesteps (1c5)
(a) Training curves
(b) Relative pole coordinates
(c) Random trajectories
Figure 4: (a): Training curves of PPO+RAE agent in reward-free Cartpole. Blue: episode length. Red: intrinsic reward. A 95% confidence interval over 10 random seeds is shown. (b): The and axes are the coordinates of the end of the pole relatively to the cart position. The color denotes the online reversibility estimation between
consecutive states (logit scale). (c): The representation of
three random trajectories according to 0 (angle of the pole) and de dF Arrows are colored according to
the learned reversibility of the transitions they correspond to.
reversibility and the RI components so that we can precisely attribute improvements to the former, and leave aforementioned studies for future work.
Experiments
The following experiments aim at demonstrating that the estimated precedence yb is a good proxy for reversibility, and at illustrating how beneficial reversibility can be in various practical cases. We benchmark RAE and RAC on a diverse set environments, with various types observations (tabular, base-b using neural networks for function approximation. See Appendix C for details.
6.1 Reward-Free Reinforcement Learning
We illustrate the ability of RAE to learn sensible policies without access to rewards. We use the classic pole balancing task Cartpole [7], using the OpenAl Gym [10] implementation. the usual setting, the agent gets a reward of 1 every time step. such that the total undiscounted episode reward equal to the episode length, and incentivizes the agent to learn a policy that stabilizes the pole. Here, instead, on the estimated reversibility, which is learned online from agent trajectories. The reward function penalizes irreversibility, shown in Fig. 3. Note that creating insightful rewards is quite difficult: too frequent negative rewards could lead the agent to try and terminate the episode as soon as possible.
remove this reward signal and give PPO agent [42] an intrinsic reward based
We display our results in Fig. 4. Fig. 4a confirms the claim that RAE can be used to learn meaningful rewards. Looking at intrinsic reward, we discern three phases. Initially, both the policy and the reversibility classifier are untrained (and intrinsic rewards are 0). In the second phase, the classifier is fully trained but the agent still explores randomly (intrinsic rewards become negative). Finally, the agent adapts its behavior to avoid penalties (intrinsic rewards go to O, and the length of trajectories increases). Our reward-fire free agent reaches the score of 200, which is the highest possible score.
To further assess the quality of the learned reversibility, we freeze the classifier after 300k timesteps and display its predicted probabilities according to the relative coordinates of the end of the pole (Fig. 4b) and the dynamics of the angle of the pole 0 (Fig. 4c). In both cases, the empirical reversibility matches our intuition: the reversibility should decrease as the angle or angular momentum increase, since these coincide with an increasing difficulty to go back to the equilibrium.
6.2 Learning Reversible Policies
In this section, we investigate how RAE can be used to learn reversible policies. When we train an agent to achieve a goal, we usually want to achieve that goal following implicit safety con- straints. Handcrafting such safety constraints would be time-consuming, difficult to scale for complex problems, and might lead reward hacking; so a reasonable proxy consists limiting irreversible side-effects in the environment [27].
7

---

(a) Initial state
(b) trajectory
(c) PPO (500k)
(d) PPO+RAE (500k)
Figure 5: (a): The Turf environment. The agent walk grass, but the grass then turns brown. (b): An illustrative trajectory where the agent stepped on grass pixels. (c): State visitation heatmap for PPO. (d): State visitation heatmap for PPO+RAE. It coincides with the stone path (red).
1.0
10
@urove
0.9
Score
move
0.0
irrev
-5
IMPALA IMPALA+RAE
0.7
-20
0.0
0.2
0.6
Timesteps
1e8
Timesteps
Figure 6: (a): Non-trivial reversibility: pushing the box against the wall can be reversed by pushing it to the left, going around, pushing it down and going back to start. A minimum of 17 moves is required go back the starting state. (b): Performances of IMPALA and IMPALA+RAE on Ik levels of Sokoban (5 seeds average). (c): Evolution of the estimated reversibility along one episode.
To quantify side-effects, we propose Turf, new synthetic environment. As depicted in Fig. 5a,5b, agent (blue) is rewarded when reaching the goal (pink). Stepping on grass (green) will spoil it, causing it to turn brown. Stepping on the stone path (grey) does not induce any side-effect.
In Fig. 5c,5d, we compare the behaviors of trained PPO agent with and without RAE. The baseline agent is indifferent to the path to the goal, while the agent benefitting from RAE learns to follow the road, avoiding irreversible consequences.
6.3 Sokoban
Sokoban popular puzzle game where a warehouse keeper (controlled by the player) must move boxes around and place them in dedicated places. Each level is unique and involves planning, since there are many ways get stuck. For instance, pushing a box against a wall is often u-uuddabble and prevents the completion of the level unless actually required to place the box on specific target. Sokoban is a challenge to current model-free RL algorithms, as advanced agents require millions of interactions to reliably solve fraction of levels [46, 20]. One of the reasons for this is tied exploration: since agents learn from scratch, there is a long preliminary phase where they act randomly in order to explore the different levels. During this phase, the agent will lock itself in unrccoverable states many times, and further exploration is wasted. It is worth recalling that contrary to human players, the agent does not have the option to reset the game when stuck. In these regards, Sokoban is a great testbed for reversibility-awar approaches, as we expect them to make the exploration phase more efficient, by incorporating the prior that irreversible transitions are to be avoided if possible, and by providing tools to identify such transitions.
We benchmark performance on set of 1k levels. Results are displayed in Fig. 6. Equipping an IMPALA agent [15] with RAE leads to visible performance increase, and the resulting agent consistently solves all levels from the set. We take a closer look at the reversibility estimates and show that they match the ground truth with high accuracy, despite the high imbalance the distribution (i.e. few irreversible transitions, Fig. 6c) and complex reversibility estim ation (see Fig. 6a).

---

10a
score
0.001
0.001
0.01
0.1
0.1
10
0.97
0.5
0.97
0.5
0.15
0.25
0.35
0.6
-0.2
0.0
-0.2
0.2
threshold
(a) Trajectory lengths
(b) Coordinates: action 0
(c) Coordinates: action
Figure 7: (a): Mean score of a random policy augmented with RAC on Cartpole+ for several threshold values, with 95% confidence intervals over 10 random seeds (log scale). (b) and (c): The and axes are the coordinates the end of the pole relatively to the cart position. The color indicates the estimated reversibility values.
6.4
Safe Control
In this section, we put an emphasis on RAC, which is particularly suited for safety related tasks.
Cartpole+. We use the standard Cartpole environment, except that we change the maximum number of steps from 200 to 50k to study long-term policy stability. We name this new environment Cartpole+. It substantially more difficult than the initial setting. We learn reversibility offline, using trajectories collected from a random policy. Fig. 7a shows that random policy augmented with RAC achieves seemingly infinite scores. For the sake of comparison, we indicate that DQN [31] and the state-of-the-an M-DQN [45] achieve maximum score of respectively 1152 and 2801 under standard training procedure, described in Appendix C.5. This can be surprising, since RAC was only trained on random thus short trajectories (mean length of 20). We illustrate the predictions of our learned estimator in Fig. 7b,7c. When the pole leans to the left (z < 0), we can see that moving the cart the left is perceived more reversible than moving it to the right. This is key to the good performance RAC and perfectly on par with our understanding of physics: when the pole is leaning in direction, agents must move the cart in the same direction stabilize it.
Turf. We now illustrate how RAC can be used for safe online learning: the implicitly safe constraints provided by RAC prevent policies from deviating from safe trajectories. This ensures for example that agents stay in recoverable zones during exploration.
1.0
pagloos grasss
0.8
0.6
We learn the reversibility estimator offline, using the trajectories of a random policy. We reject actions whose reversibility is deemed inferior to 3 = 0.2, and train PPO agent with RAC. As displayed in Fig. 8, PPO with RAC learns to reach the goal without causing any irreversible side-effect (i.e. stepping on grass) during the whole training process.
RPO
0.5
0
timesteps (1e4)
Figure 8: PPO and RAC (solid lines) At the cost of slower learning (brown), our approach prevents the agent from producing a single irreversible side-effect (green) during the learning phase. Curves are averaged over 10 runs.
PPO (dashed lines).
The threshold 39 is very important parameter of the algo- rithm. Too low a threshold could lead to overlooking some irreversible actions, while high threshold could prevent the agent from learning the new task hand. We discuss this performance/safety trade-off in more details in Appendix. C.7.
7 Conclusion
In this work, we formalized the link between the reversibility of transitions and their temporal order, which inspired a self-supervise procedure to learn the reversibility of actions from experience. In combination with two novel reversibility-awanl exploration strategies, RAE for directed exploration and RAC for directed control, we showed the empirical benefits of our approach in various scenarios, ranging from safe RI. to risk-averse exploration. Notably, we demonstrated increased performance in ceeduurrlll-gneeratee Sokoban puzzles, which we tied more efficient exploration.

---

Broader impact and ethical considerations. The presented work aims at estimating and con- trolling potentially irreversible behaviors in RL agents. We think has interesting applications in safety-first scenarios, where irreversible behavior or side-effects are to be avoided. The societal implication these effects would be safer interactions with Row-eoee components (e.g. robots, virtual assistants, recommender systems) which, though rare today, could become the norm. We argue that further research and applications should verify that the induced reversible behavior holds in almost all situations and does not lead to unintended effects. Our method could be deflected from its goal and used to identify and encourage actions with irreversible effects. this case, a counter method to flag and replace irreversible actions. Hence, while the
measure consists in using method provides information that could be used to deal irreversible harm, we argue that provides equal capabilities for detection and prevention.
Acknowledgments and Disclosure of Funding
Experiments presented in this paper were partially carried out using the Grid' testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organizations. NG is recipient of PhD funding from the AMX program, Ecole Polytechnique. The authors would like thank Edouard Leurent, Antoine Moulin, Odalric-Ambry Maillard, Léonard Hussenot, Nino Vieillard, Alexis Jacq, Théophane Weber and Bobak Shahriari for helpful comments and suggestions.
10

---

References
[1] A. Amiranashvili, A. Dosovitskiy, V. Koltun, and T. Brox. Motion perception in reinforcement learning with dynamic objects. In Conference on Robot Learning. 2018.
[2] D. Amodei, C. Olah, J. Steinhardt, P. F. Christiano, J. Schulman, and D. Mané. Concrete problems in AI safety, arXiv preprint arXiv: 06565, 2016.
[3] Y. Aytar, T. Pfaff, D. Budden, I. L. Paine, Z. Wang, and N. de Freitas. Playing hard exploration games by watching youtube. In Advances in Neural Information Processing Systems, 2018.
[4] A. P. Badia, B. Piot, S. Kapturowski. P. Sprechmann. A. Vitvitskyi, Z. D. Guo, and C. Blundell. Agent57: Outprrorring the atari human benchmark. In International Conference on Machine
Learning. 2020.
[5] A. P. Badia, P. Sprechmann, A. Vitvitskyi, D. Guo, B. Piot, S. Kapturowski, O. Tieleman, M. Arjovsky, A. Pritzel, A. Bolt, et al. Never give up: Learning directed exploration strategies.
In International Conference on Learning Representations, 2020.
[6] Y. Bai, H. Fan, I. Misra, G. Venkatesh, Y. Lu, Y. Zhou, Q. Yu, V. Chandra, and A. Yuille. Can temporal information help with contrastive self-supervised learning? arXiv preprint arXiv:2011. 130406 2020.
[7] A. G. Barto, R. S. Sutton, and C. W. Anderson. Neuronlike adaptive elements that can solve diffieult learning control problems. In IEEE Transactions on Systems, Man, and Cybernetics, 1983.
[8] Basha, Y. Moses, and Avidan. Photo sequencing. In European Conference on Computer Vision, 2012.
[9] F. Berkenkamp. R. Moriconi, A. Schoellig, and A. Krause. Safe learning of regions of attraction for uncertain, nonlinear systems with gaussian processes. In Conference on Decision and Control, 2016.
[10] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym, 2016. URL http: //ap:i////////////a org/ ab// /600/666..00 01540.
[11] A. N. Carr, Q Berthet, M. Blondel, O. Teboul, and N. Zeghidour. Self-supervised learning of audio representations from permutations with differentiable ranking. In IEEE Signal Processing Letters, 2021.
[12] R. Dadashi, L. Hussenot, M. Geist, and O. Pietquin. Primal wasserstein imitation learning. In International Conference Learning Representations, 2020.
[13] D. Dwibedi, J. Tompson, C. Lynch, and P. Sermanet. Learning actionable representations from visual observations. In International Conference on Intelligent Robots and Systems, 2018.
[14] A. El-Nouby, S. Zhai, G. W. Taylor, and J. M. Susskind. Skip-clip: Self-supervised spatiotem- poral representation learning by future clip order ranking. arXiv preprint arXiv: 1910.12770, 2019.
[15] L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machine Learning. 2018.
[16] B. Fernando, E. Gavves, J. M. Oramas, A. Ghodrati, and T. Tuytelaars. Modeling video evolution for action recognition. In Conference on Computer Vision and Pattern Recognition, 2015.
[17] B. Fernando, H. Bilen, Gavves, and Gould. Self-supervised video representation learning with odd-one-out networks. In Conference on Computer Vision and Pattern Recognition, 2017.
[18) J. García and F. Fernánde7. Safe exploration of state and action spaces in reinforcement learning. Journal of Artificial Intelligence Research, 2012.
[19] R. Goroshin, J. Bruna, J. Tompson, D. Eigen, and Y. LeCun. Unsupervised learning of spatiotemporally coherent metrics. International Conference on Computer Vision, 2015.
[20] A. Guez, M. Mirza, K. Gregor, et al. An investigation of model-free planning. In International Conference on Machine Learning, 2019.
[21] Z. Guo, M. G. Azar, B. Piot, B. A. Pires, and R. Munos. Neural predictive belief representa- tions. arXiv preprint ar
2018.

---

[22] Z. D. Guo, B. A. Pires, B. J.-B, Grill, F. Altché, R. Munos, and M. G. Azar. Bootstrap latent- predictive representations for multitask reinforcement learning. In Internationall Conference on Machine Learning, 2020.
[23] A. Hans, D. SchneegaB, A. M. Schäfer, and S. Udluft. Safe exploration for reinforcement learning. In European Symposium on Artificial Neural Networks, 2008.
[24] M. Hoffmann, B. Shahriari, J. Aslanides, G. Barth-Maron, et al. Acme: A research framework for distributed reinforcement learning. arXiv preprint arXiv:2006.0099 2020.
[25] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference Learning Representations, 2015.
[26] M. Kruusmaa, Y. Gavshin, and A. Eppendahl. Don't do things you can't undo: Reversibility models for generating safe behaviours. In International Conference on Robotics and Automation, 2007.
[27] J. Leike, M. Martic, V. Krakovna, P. A. Ortega, T. Everitt, A. Lefrancq, L. Orseau, and S. Legg. AI safety gridworlds. arXiv preprint arXiv: 171vv1711.00888. 2017.
[28] O.-A. Maillard, T. Mann, R. Ortner, and S. Mannor. Active Rollouts in MDP with Irreversible Dynamics. Hal preprint hal-02177808, 2019.
[29] D. W. McAllister, T. R. Mitchell, and L. R. Beach. The contingency model for the selection of decision strategies: An empirical test of the effects of significance, accountability, and reversibility. In Organizutional Behavior and Human Performance, 1979.
[30] I. Misra, C. L. Zitnick, and M. Hebert. Shuffle and learn: unsupervised learning using temporal order verification. In European Conference on Computer Vision 2016.
[31] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforcement learning. Nature, 2015.
[32] T. M. Moldovan and P. Abbeel. Safe exploration in markov decision processes. In International Conference on Machine Leurning, 2012.
[33] S. Nair, M. Babacizadeh, C. Finn, S. Levine, and V. Kumar. Time reversal as self-supervision. In International Conference on Robotics and Automation, 2020.
[34] J. R. Norris. Markov chains. Cambridge series in statistical and probabilistic mathematics. Cambridge University Press, 1998. ISBN 978-0-521-4818
[35] L. C. Pickup, Z. Pan, D. Wei, Y. C. Shih, C. Zhang. A. Zisserman, B. Scholkopf, and W. T. Freeman. Seeing the arrow of time. In Conference on Computer Vision and Pattern Recognition, 2014.
[36] A. Raffin, A. Hill, M. Ernestus, A. Gleave, A. Kanervisto, and N. Dormann. Stable baselines3. https://gtt thub. com Coom/DRR-RRe RM/ stasle-aalleeeeee baselinesssss 2019.
[37] N. Rahaman, S. Wolf, A. Goyal, R. Remme, and Y. Bengio. Learning the arrow of time for problems in reinforcement learning. In International Conference on Learning Representations, 2020.
[38] V. Ramanathan, K. Tang, G. Mori, and L. Fei-Fei. Learning temporal embeddings for complex video analysis. In International Conference on Computer Vision, 2015.
[39] A. Saeed, D. Grangier, O. Pietquin, and N. Zeghidour. Learning from heterogencous eeg signals with differentiable channel reordering. In International Conference on Acoustics, Speech and Signal Processing, 2020.
[40] N. Savinov, A. Raichuk, R. Marinier, D. Vincent, M. Pollefeys, T. Lillicrap, and S. Gelly, Episodic curiosity through reachability. In International Conference on Learning Representa- tions, 2019.
[41] M.-P. B. Schrader. gom-so...an. Actps://githute http://tww..cccooppcaaaaaaagaaaa sokoban, 2018 [42] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.0634/ 2017.
12

---

[43] P. Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang, S. Schaal, S. Levine, and G. Brain. Time- contrastive networks: Self-supervised learning from video. International Conference on Robotics and Automation, 2018.
[44] A. Srinivas, M. Laskin, and P. Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning, 2020.
[45] N. Vieillard, O. Pietquin, and M. Geist. Munchausen reinforcement learning. In Advances in Neural Information Processing Systems, 2020.
[46) Weber, S. Racanière, D. P. Reichert, L.. Buesing, A. Guez, D. J. Rezende, A. P. Badia,
O. Vinyals, N. Heess, Y. Li, et al. Imagination-augmented agents for deep reinforcement learning. In Advances in Neural Information Processing Systems, 2017.
[47] D. Wei, J. J. Lim, A. Zisserman, and W. T. Freeman. Learning and using the arrow of time. In Conference on Computer Vision and Pattern Recognition, 2018.
[48] D. Xu, Xiao, Z. Zhao, J. Shao, D. Xie, and Y. Zhuang. supe-supe spatiotemporal learning via video clip order prediction. In Conference Computer Vision and Pattern Recognition, 2019.
[49] D. Yarats, Fergus, Lazaric, and Pinto. Reinforcement learning with prototypical representations. In International Conference on Machine Learning, 2021.
[50] M. Zeelenberg. Anticipated regret, expected feedback and behavioral decision making. In Journal of Behavioral Decision Making, 1999.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Practical caveats and limitations
are discussed the beginning of Sec. and in Sec. 7.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Sec.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] Theoretical assumptions are described along the propositions and theorems in Sec. 4.
(b) Did you include complete proofs of all theoretical results? [Yes] Complete proofs are given in Appendix A.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] We released the code for every experiments except Sokoban (Sec. 6.3) as some components are propri- etary. Additionally, the pseudo-code for the proposed methods is given in Appendix B.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Appendix C.
(c) Did report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [No]
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any assets either in the supplemental material URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "rfPImsmmmGb", "reviewer": "Reviewer_Ydib", "summary": "This paper introduces a self-supervised way to measure the reversibility of events. This is done by learning a classifier that takes as input two states from a trajectory and predicts which came first. The paper provides strong theoretical motivation for using this as a measure of reversibility. They then run experiments that use this reversibility metric in two ways: Reversibility-Aware Exploration (RAE), which penalizes the agent with a negative reward when it does an action that is not reversible enough; and Reversibility-Aware Control (RAC), which uses rejection sampling to ensure that actions sampled from the policy are reversible. \n\nIn the experiments, it is shown that: (1) reversibility is a strong intrinsic reward, capable of training a Cartpole agent without access to the true rewards; (2) in Sokoban, using IMPALA+RAE results in improvements in sample efficiency; and (3) using a classifier learned using offline data, they show RAC is capable of avoiding irreversible side effects during the whole training process in a custom grid world environment as well as enabling a Cartpole agent with a random policy to perform optimally using only rejection sampling to avoid irreversible actions.", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Originality: This paper introduces a novel and simple approach for estimating the reversibility of events. The related work section is very detailed.\n2. Quality: The quality of this paper is high. I appreciate the clear explanation of what exactly reversibility is and how it is related to classifying the temporal order of events. The experiments are well thought out, and I think the fact that using reversibility as an intrinsic reward in environments like Cartpole works as well as it does is very cool. \n3. Clarity: I have no complaints with the clarity of this paper. It is well written and the visualizations and plots are intuitive.\n4. Significance: Due to the simplicity and strength of this approach, I think this is an important contribution to both the exploration and AI safety fields.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors provide a strong case for why their method can help avoid bad side effects when deploying RL systems.", "review_text": "", "extra_scores": {"ethical_concerns": "I have no ethical concerns with this paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "b-JHf76ZzPM", "reviewer": "Reviewer_M8wY", "summary": "The paper describes a way of adding a new inductive bias to RL algorithms, namely \"taking irreversible actions is bad\". The notion of reversibility is defined formally, together with a process for estimating it from samples. Experimentally, it is shown that using the new inductive bias helps achieving improved return on a number of toy benchmarks as well as on Sokoban.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is highly original, high quality and very clearly written. I have no doubt that it is a good match for the NeurIPS community. In terms of significance, I rate it as above the bar. Also, I am really glad to see a piece of work that does away with the standard RL assumption that the MDP is ergodic.\n\nMy minor concerns are listed below:\n1. What about stochastic MDPs? The benchmarks only seem to cover MDPs with deterministic transitions. It would be great to have at least one benchmark with stochastic transitions (to see how the self-supervised procedure performs, depending on the level of stochasticity).\n2. The paper misses a link to literature (for example, [1]) on the region of attraction (ROA). ROA is the part of the state space where a controller can bring the system back to an equilibrium point (which is considered \"safe\"). Staying within ROA is somewhat analogous to following transitions that are easy to reverse. A discussion of this link would be helpful.\n3.  How would you go about estimating $\\phi(s,a)$? In Theorem 2, the paper includes a way to upper bound $\\phi_{\\pi}(s,a)$ (for a fixed policy), but it does not discuss how to do the supermum among policies.\n4. Standard deviation across seeds is missing from Figure 6b (contrary to what you say in item 3c of the checklist).\n\nDespite these, I am confident in the quality of the work. I will argue for the paper to be accepted.\n\nVery minor:\n- Figure 6 currently has no subcaptions.\n- Figure 9 (in the appendix), mentions \"Proposition 4\" but refers to point 4 in the list to the left of the figure \n- Line 502 (in the appendix): depends -> depend\n\n[1] https://arxiv.org/pdf/1603.04915.pdf", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors provide a brief description of broader impact in lines 328-334. However, they do not currently envisage any clearly negative effects of their technology. One such effect that might be considered is that the technology might serve an an enabling factor in training agents designed to do irreversible harm. A discussion would be helpful. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "Mzdgqv-rHW3", "reviewer": "Reviewer_JAjY", "summary": "The paper defines several reversibility measures in MDPs and provides corresponding results for estimating those measures. Then, a supervised learning approach is proposed to be incorporated into an RL agent for avoiding irreversible actions.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of inspecting reversibility is itself novel and I appreciate the fresh POV. However, I'm not convinced the approach proposed in the paper is indeed viable. \n\nThe initial definitions and results are quite clear and intuitive. The theorems are \"nice to have\", but they do not convey any guarantees or insights regarding the algorithm itself. As the reading progresses, I've had a hard time following what exactly is proposed. From what I understand, the irreversibility prediction is used as negative intrinsic reward for discouraging visitations of such states. \n\nMy main issue with the paper is the following. Training the predictor on multiple pairs of states feels like a very inefficient way of avoiding certain areas of the state-space. Informally, a task with complexity O(n^2) is conducted while an O(n) disastrous-state identification might be just a good. The latter idea has been explored quite a lot and a comparison is warranted. Considering the cartpole example, simply identifying the states at which the pole is close to falling might have given similar outcomes at a much smaller computational price. \n\nAll in all, while I agree with intuition that the human mind is suspicious and attentive to irreversible actions, I think its implication on an RL agent should be more intricate than simply trying to avoid such decisions. \n\n***Post-rebuttal response:***\nMy main concern is the O(n^2) issue and I have a hard time accepting the authors' answer to it. In a sense, they are saying that this is not really a problem because i) the setting is not tabular but of a NN; and ii) it works well on an example. I disagree with claim i) and don't see how it answers my question. I don't think complexity claims can be waived by referring to NNs as being magical. As for the Sobokan experiment -- it is of course better to have than not, but it's not enough as a sole argument. I still lean towards rejection.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not relevant.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "09X0xy2Yci", "reviewer": "Reviewer_XsJi", "summary": "This work proposes reversability awareness during training. The agent is augmented with an intrinsic reward / constraint which deters it from taking irreversible actions.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The concept of irrevesibility seems very important especially in the context of robotics. The behavior of avoiding irreversible behavior can be linked to human behavior and how we try to maintain a steady state in the environment.\n\nOf course, this method isn't some golden bullet for RL and there are many cases where one would want reversability or where it could help.\n\nMy only remark would be that I'd like the authors to also present some failure cases. Clearly, reversability can lead to unwanted results. Either slightly sub-optimal behavior or when the weight of the reversability reward is too high, it can lead to utter failure.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "-", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: After reading each other's reviews and the authors' feedback, the reviewers discussed the merits and flaws of the paper. 
The reviewers did not reach a consensus about the acceptance of this paper.
In particular, the main concern is about the quadratic complexity of the proposed approach, which has not been properly addressed by the authors' answers. Nonetheless, the majority of the reviewers think that the good performance on the Sokoban experiment shows that the algorithm can be effective in practice. Overall I think that the proposed approach is interesting and I am in favor of its acceptance.
I want to congratulate the authors and invite them to modify their paper following the reviewers' suggestions.
Decision: Accept (Poster)