Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents an approach to reduce the number of user interactions necessary to reset the underlying system in a value function based reinforcement learning task with states that the system can not reset itself. To achieve this the approach addresses 2 main issues, namely i) reducing the likelihood that the system will enter an irreversible state through a change in the reward function, ii) reducing the number of times that a user has to provide information about the state by identifying relevant candidates to find the first irreversible state.
The resulting approach can be used in the context of any value-function
based RL algorithm and the paper presents performance comparisons with other methods (although those are not exactly designed for the same problem).
strengths_and_weaknesses: 
The paper addresses a problem in autonomous learning and human-machine  collaboration that has potential applications in a number of domains. It is well written and clearly introduces the concepts as well as the formalism where it builds on a reward function modification and a binary search-based strategy to guide the learning and the querying process.
The biggest weaknesses of the paper are in the somewhat limited amount of overall impact and in the experimental evaluation where results are compared with algorithms that were not directly designed with the same objectives in mind. It would have been good to see more discussion on the different assumptions these algorithms make.
Another issue is in Figure 7 where for the left two graphs the legend indicates 5 curves but only 4 seem to be visible.
questions: The querying method for label information used in this paper uses a binary search based criterion to elicit label information to build the classifier for irreversible states. Why does the certainty in the prediction not influence whether a label query is made ?

In the performance comparisons, the proposed algorithm seems to end up with lower overall success than some comparison methods although with convergence with lower interaction numbers. How does this trade off play into decisions ?

limitations: The paper discusses limitations of the approach in the last section and touches on the most important aspects. A little more discussions on the approach’s impact on other performance measures would have been useful.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes and uses the observation that states preceding a reversible state are reversible and states succeeding an irreversible state are irreversible. Leveraging this observation, the paper proposes an approach that predicts and tries to avoid irreversible states that makes is more efficient that than traditional safe-RL methods. The proposed approach has been validated on different RL setups and shown to be better than baseline approaches.
strengths_and_weaknesses: Strengths
1. Work is related to a popular subtopic in the field.
2. Videos of experimental results are available online.
3. Decent coverage of ablation studies and limitations of current work.


Growth opportunities
1. Some thresholds, such as R_{p} appear to be arbitrary.
2. The form of the reversibility classifier is unclear. 
questions: 1. Why is the font size changed during the introduction of the PAINT acronym?
2. Why is it sufficient to only rely on the states preceding a reversible state and states following an irreversible state?
3. How do the safe RL approaches perform if they use the observation regarding reversible and irreversible states proposed in the present study?

limitations: Yes, the conclusion describes a couple of the limitations of the proposed method. It might be worth conjecturing how PAINT might function on shared autonomy tasks.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper tackles the problem of training an RL system in an environment where not all actions are reversible, and therefore the environment must be reset before further training can take place. The approach learns a classifier which labels states with a reversibility label R -> [0,1] and only requests an intervention to reset the environment when this passes a threshold. The classifier training exploits the fact that irreversibility is deterministic once an irreversible state has been reached (i.e. all future states are irreversible too). The approach is evaluated on suitably complex domains, and demonstrates that it requests fewer resets than competitor methods.
strengths_and_weaknesses: Overall I think the paper well written. I like that the approach is straightforward and clearly justified. It leverages existing work (MEDAL) and extends it clearly, adding a new capability. The system is evaluated on domains with well-justified irreversibility, and on these domains the results demonstrate that the proposed method reduces the number of resets requested.


On the negative side, I was disappointed to see that there was no exploration of the threshold used to trigger the resets. Setting this to 0.5 feels like a reasonable thing to do, but I imagine your results depend on this parameter in interesting ways. Similarly, the idea that the agent can execute a fixed number of exploration steps after a reset occurs seems to be thrown in rather late in the paper and not sufficiently justified or explored. I could imagine settings (e.g. maze with all trenches connected) where a single failure could allow all mapping of irreversible states - and this could be a bug or a feature. 

I also would have liked to have seen more discussion on the task metrics in the evaluation. My reading of the results is that PAINT always produces better performance with fewer resets, but then plateaus and does not ask for further resets. This seems to hinder task performance. I understand that this is part of the motivation of the method (fewer resets/reversibility labels), but being unable to obtain competitive task performance seems like a major weakness, particularly if the method is terminating early. 

I like the title, but I think it's misleading. In particular, there are two forms of "help": reversibility labelling and environment resetting. I think the title starts the paper by blurring these ideas, and it doesn't recover totally (even though both types of help are delivered at the same time). 
questions: 1. How would system performance change if the threshold of R>=0.5 was varied.
2. Why does task performance not improve to levels of the baselines. 
3. How easy or hard is it to label and the reversibility of states in control tasks like the peg in hole? Is the object observable once it falls?
4. How does uncertainty impact the problem, e.g. if an action has only some probability of leading to an irreversible state.

limitations: Some limitations are mentioned, and there are no obvious negative social implications.
review_title: null

Meta-review: metareview: The paper introduces a new problem statement for RL, i.e., how to identify irreversible states in RL that require help from a human operator. The problem statement and the algorithm are well motivated and the the reviewers also appreciated the reported experiments. The authors addressed the few concerns (ablations, comparisons) well in the rebuttal. I follow the reviewers and recommend acceptance. 
confidence: Certain
Decision: Accept