Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: This paper introduces a self-supervised way to measure the reversibility of events. This is done by learning a classifier that takes as input two states from a trajectory and predicts which came first. The paper provides strong theoretical motivation for using this as a measure of reversibility. They then run experiments that use this reversibility metric in two ways: Reversibility-Aware Exploration (RAE), which penalizes the agent with a negative reward when it does an action that is not reversible enough; and Reversibility-Aware Control (RAC), which uses rejection sampling to ensure that actions sampled from the policy are reversible. 

In the experiments, it is shown that: (1) reversibility is a strong intrinsic reward, capable of training a Cartpole agent without access to the true rewards; (2) in Sokoban, using IMPALA+RAE results in improvements in sample efficiency; and (3) using a classifier learned using offline data, they show RAC is capable of avoiding irreversible side effects during the whole training process in a custom grid world environment as well as enabling a Cartpole agent with a random policy to perform optimally using only rejection sampling to avoid irreversible actions.
main_review: 1. Originality: This paper introduces a novel and simple approach for estimating the reversibility of events. The related work section is very detailed.
2. Quality: The quality of this paper is high. I appreciate the clear explanation of what exactly reversibility is and how it is related to classifying the temporal order of events. The experiments are well thought out, and I think the fact that using reversibility as an intrinsic reward in environments like Cartpole works as well as it does is very cool. 
3. Clarity: I have no complaints with the clarity of this paper. It is well written and the visualizations and plots are intuitive.
4. Significance: Due to the simplicity and strength of this approach, I think this is an important contribution to both the exploration and AI safety fields.
limitations_and_societal_impact: The authors provide a strong case for why their method can help avoid bad side effects when deploying RL systems.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper defines several reversibility measures in MDPs and provides corresponding results for estimating those measures. Then, a supervised learning approach is proposed to be incorporated into an RL agent for avoiding irreversible actions.
main_review: The idea of inspecting reversibility is itself novel and I appreciate the fresh POV. However, I'm not convinced the approach proposed in the paper is indeed viable. 

The initial definitions and results are quite clear and intuitive. The theorems are "nice to have", but they do not convey any guarantees or insights regarding the algorithm itself. As the reading progresses, I've had a hard time following what exactly is proposed. From what I understand, the irreversibility prediction is used as negative intrinsic reward for discouraging visitations of such states. 

My main issue with the paper is the following. Training the predictor on multiple pairs of states feels like a very inefficient way of avoiding certain areas of the state-space. Informally, a task with complexity O(n^2) is conducted while an O(n) disastrous-state identification might be just a good. The latter idea has been explored quite a lot and a comparison is warranted. Considering the cartpole example, simply identifying the states at which the pole is close to falling might have given similar outcomes at a much smaller computational price. 

All in all, while I agree with intuition that the human mind is suspicious and attentive to irreversible actions, I think its implication on an RL agent should be more intricate than simply trying to avoid such decisions. 

***Post-rebuttal response:***
My main concern is the O(n^2) issue and I have a hard time accepting the authors' answer to it. In a sense, they are saying that this is not really a problem because i) the setting is not tabular but of a NN; and ii) it works well on an example. I disagree with claim i) and don't see how it answers my question. I don't think complexity claims can be waived by referring to NNs as being magical. As for the Sobokan experiment -- it is of course better to have than not, but it's not enough as a sole argument. I still lean towards rejection.
limitations_and_societal_impact: Not relevant.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work proposes reversability awareness during training. The agent is augmented with an intrinsic reward / constraint which deters it from taking irreversible actions.
main_review: The concept of irrevesibility seems very important especially in the context of robotics. The behavior of avoiding irreversible behavior can be linked to human behavior and how we try to maintain a steady state in the environment.

Of course, this method isn't some golden bullet for RL and there are many cases where one would want reversability or where it could help.

My only remark would be that I'd like the authors to also present some failure cases. Clearly, reversability can lead to unwanted results. Either slightly sub-optimal behavior or when the weight of the reversability reward is too high, it can lead to utter failure.
limitations_and_societal_impact: -
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper describes a way of adding a new inductive bias to RL algorithms, namely "taking irreversible actions is bad". The notion of reversibility is defined formally, together with a process for estimating it from samples. Experimentally, it is shown that using the new inductive bias helps achieving improved return on a number of toy benchmarks as well as on Sokoban.
main_review: The paper is highly original, high quality and very clearly written. I have no doubt that it is a good match for the NeurIPS community. In terms of significance, I rate it as above the bar. Also, I am really glad to see a piece of work that does away with the standard RL assumption that the MDP is ergodic.

My minor concerns are listed below:
1. What about stochastic MDPs? The benchmarks only seem to cover MDPs with deterministic transitions. It would be great to have at least one benchmark with stochastic transitions (to see how the self-supervised procedure performs, depending on the level of stochasticity).
2. The paper misses a link to literature (for example, [1]) on the region of attraction (ROA). ROA is the part of the state space where a controller can bring the system back to an equilibrium point (which is considered "safe"). Staying within ROA is somewhat analogous to following transitions that are easy to reverse. A discussion of this link would be helpful.
3.  How would you go about estimating $\phi(s,a)$? In Theorem 2, the paper includes a way to upper bound $\phi_{\pi}(s,a)$ (for a fixed policy), but it does not discuss how to do the supermum among policies.
4. Standard deviation across seeds is missing from Figure 6b (contrary to what you say in item 3c of the checklist).

Despite these, I am confident in the quality of the work. I will argue for the paper to be accepted.

Very minor:
- Figure 6 currently has no subcaptions.
- Figure 9 (in the appendix), mentions "Proposition 4" but refers to point 4 in the list to the left of the figure 
- Line 502 (in the appendix): depends -> depend

[1] https://arxiv.org/pdf/1603.04915.pdf
limitations_and_societal_impact: The authors provide a brief description of broader impact in lines 328-334. However, they do not currently envisage any clearly negative effects of their technology. One such effect that might be considered is that the technology might serve an an enabling factor in training agents designed to do irreversible harm. A discussion would be helpful.
review_title: null

Meta-review: After reading each other's reviews and the authors' feedback, the reviewers discussed the merits and flaws of the paper. 
The reviewers did not reach a consensus about the acceptance of this paper.
In particular, the main concern is about the quadratic complexity of the proposed approach, which has not been properly addressed by the authors' answers. Nonetheless, the majority of the reviewers think that the good performance on the Sokoban experiment shows that the algorithm can be effective in practice. Overall I think that the proposed approach is interesting and I am in favor of its acceptance.
I want to congratulate the authors and invite them to modify their paper following the reviewers' suggestions.
Decision: Accept (Poster)