Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: summary: This paper uses causal influence diagrams (CIDs) to consider abstract agents that plan according to a model of the world that lacks certain features, such as emergency stop buttons.  As the model used for planning doesn't contain the stop button, the agent will have no incentive to ever remove this.
main_review: The basic idea of this paper seems to be that a model based agent isn't going to intentionally act in a way that disables some safety system, such as an emergency stop button, if the agent's planning model doesn't include this system's function.  Yes, that claim seems true to me.  However, if the AGI agent is highly intelligent, how confident can we be that its world model won't contain this system?  Or at least a suspicion that such a system likely exists and thus an incentive to look for it.  I'd go even further and say that an AGI with roughly human level intelligence or higher will almost certainly suspect such a thing and investigate the possibility, given that its existence could be highly relevant to its future expected utility.  As such, I'm not convinced that this paper presents a solution to the issue being studied.

You could even argue that the paper tries to define away the problem.  In short: if the agent doesn't think about AGI safety mechanisms then it won't try to subvert them.  (where "doesn't think about" is shorthand for not having these in its planning model)

For this approach to be more convincing, I'd need a clear explanation of why the agent wouldn't know about, or suspect and try to learn about, such safety mechanisms.

Line 225-229: The idea of having an AGI tripwire that looks for an unexpected increase in the agent's effectiveness or estimated intelligence isn't a new idea.  I've heard it in discussions for over ten years and the idea is probably much older: if you're interested in AGI tripwires and worried about agents self improving, this is an obvious think to do.  So I presume it must be mentioned in a publication somewhere, perhaps in Bostrom's book Superintelligence?

Finally, isn't "counterfactual planning" just planning?  That is, you have an agent that uses a model of its world to explore a range of futures that haven't yet happened (and are thus counterfactual)?

limitations_and_societal_impact: No concerns.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper tackles the problem of designing "robust" safety mechanisms against hypothetical AGI agents that may override such safety mechanisms to complete its goal.
The paper proposes using counterfactual planning, where an agent plans in a world designed to be different from the real world. The key idea is to plan in a counterfactual world where the safety mechanism does not exist, and use such planned action in the real world. This can then reduce the likelihood of the agent learning a direct correlation between the safety mechanism and goal completion. The paper also argues such planning offers a new vantage point for modeling AGI agents due to its compact and readable graphical language. Two examples are presented to showcase the proposed design approach where construction of counterfactual worlds suppresses some unsafe agent incentives.

main_review: The paper proposes an interesting idea of planning in a counterfactual world to minimize the direct incentives for agents to learn to override safety mechanisms. This idea seems novel and technically sound.  The graphical modeling of AGI agents in its "learning world" and "planning world" helps with interpretability. The limitations listed in the paper are well-thought-out, especially on its potential unintended consequences such as, "false security". 

To improve the paper, it'll be especially helpful to extend on Section 4 and 5 with well-thought-out and tangible examples. Currently, it is hard to evaluate the exact contribution of counterfactual planning to tackle practical examples. For more impact, it would be helpful to focus on one use case per section to thoroughly describe to readers how counterfactual planning can exactly counter the "stop button" problem in well-illustrated examples. For example, by describing what L and S should be for the paper clip example in line 259-264. These would help clarify what is the exact contribution of the proposed method to state-of-the-art methods. For example, how does counterfactual planning help users understand what motivates the planning world agent? It will also be very helpful if the paper includes a toy experiment with the paperclip example, to demonstrate exactly how counterfactual planning can discourage unsafe agent incentives.

Secondly, it will be helpful to offer more technical analysis on the impact of the assumptions and how violations of such assumptions can manifest. For example, on the assumption of L~S. It is unclear if this assumption is reasonable and how we can practically gauge this in the real world. As the paper mentions, "when this approximation is not good enough, the agent may end up behaving in an unexpected and potentially unsafe way". Examples can be given on what is such an unexpected manner and how big of an approximation gap can manifest such behaviors. As this paper mainly includes thought experiments, clarifying such assumptions and details can increase impact. 

Thirdly, more details and analysis can be added to the power-based safety interlock mechanism. The paper claims this as a contribution, with such an interlock offering a further line of defense. However, more detail can be given to substantiate such claims with more examples, and also analysis on potential unintended consequences. For example, would imposing such a mechanism slow down learning?

Lastly, it will be helpful for some comments on how this idea can be implemented on real world systems. As part of the limitation, the authors do note that "their practical usefulness might end up being limited". Despite this, it'll be interesting to know what are potential test cases where this method can make the most impact. 

With more thorough examples and analysis, I believe the findings of the paper can be significant to the community. The idea of planning in counterfactual worlds to design safety mechanisms seems sound and can be an interesting thought exercise when addressing AI safety and alignment.

limitations_and_societal_impact: The authors have adequately addressed the limitations and potential negative societal impact of their work.
Some comments on the inclusion of a power-based safety mechanisms can be helpful.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper present counterfactual planning as an approach for creating safety mechanisms for agents.  It is framed in the context of AGI.  The topic is relevant to the NeurIPS community and the paper is interesting and well written.  I do believe there are contributions in this work towards building safer AI systems and the way that it proposes to do so.  Other than those concerns articulated in my main review below, I think the work is thorough and well presented.  The technical correctness and detail seems sufficient.  
main_review: Below I will list my concerns about the paper and the questions for the authors. Unfortunately the first might be something that is difficult to address in a rebuttal, but it would be helpful to have a little clarification or justification if the authors believe I have misunderstood things.

1. The authors reference the 39-page preprint explicitly in several places.  Unfortunately, I don't believe they have correctly followed the anonymization policy for NeurIPS which states " you need to cite one of your own papers, you should do so with adequate anonymization to preserve double-blind reviewing.  For instance, write “In the previous work of Smith et al. [1]…” rather than “In our previous work [1]...”)."
I can only assume that the report is by the same author and it should have been referenced in the manner above.  Related to this I was struggling to distinguish from the references to this paper, how the authors were distinguishing their submission from the pre-print. I'd like to have a clearer articulation of that in the rebuttal. 

2. Perhaps my main concern with the paper is the illustrated by the lack of evaluation of the methodology.  The authors identify that their contribution is "at its core a theoretical one" but that they are primarily presenting a ``new models and methodologies'' which I would agree with.  It would have been nice to see an example of they models/methodologies demonstrated even on a toy example. What was the reasoning for not doing so?  I don't think these need to be extensive or elaborate but I think it would add a lot to the contribution. I wonder if this might help also differentiate this contribution from the 39-page preprint (albeit that I wasn't able to read that preprint in full).  

3. From my perspective the related work sections was a little weak. I would have liked to have seen a little more discussion of the related work in counterfactual reasoning and other related areas. The were desperately short on space in their submission and so it would be nice to see a little more in the way of background.

I will note that my score below was influenced by the lack of anonymization.  I wasn't sure how much weight to give that.  Overall, I think the paper is otherwise generally good. But I was trying to follow the NeurIPS guidelines as closely as possible.
limitations_and_societal_impact: This is a theoretical paper which the authors admit may or may not have implications.  The motivation is a good one and they do discuss the broader impact and societal implications of their work.  I don't have any concerns there.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper presents an approach to AI safety based on graphical models. It argues that it is providing new insights to safety problems which leverage these models. The paper exemplifies this by using the case of a stop button, and demonstrates how alterations to the graphical model (stop button, input terminal) in the learning (i.e real) world but not the planning (i.e. agent training/reasoning) allows safety to be introduced and not circumvented during the learning process modelled in the planning world. 
main_review: The positive of the work is that using a graphical language for two models (learning, planning) for presenting, motivating, and discussing these cases removes the reliance on formal mathematical semantics or informal linguistic semantics when considering hypothetical future AI agents. This can reduce the ambiguity and sleight of hand of these kinds of discussions particularly for researchers that cannot access the formal mathematics of some of the cited papers. This can broaden the field and simplify the discussion, which is a good thing.

Considering the negative, I will break the paper into 2 parts: the modelling of an AI system as a graphical model, and the use of this model to discuss AI safety.

On the modelling side, the approach is near identical to the work by Toussaint et al under the banner planning as probabilistic inference.  
- https://argmin.lis.tu-berlin.de/papers/12-botvinick-TICS.pdf
- https://www.ics.uci.edu/~dechter/courses/ics-295/fall-2013/papers/toussaint-probinf-pomdp.pdf
etc.

This takes the same approach of modelling states, actions, observations etc. as random variables. It does not explicitly deal with causality, but it does use directed arrows for the progression of time. I suspect they would argue the arrows are causal too. I would not want to see this work accepted for publication without a significant effort linking the proposed modelling formalism to Toussaint's work.

On the use of this model to analyse AI safety, whilst the clarity it brings is good, I fear it simply moves the casual response to AI safety ("don't let it access it's own stop button/reward function") from prose to a diagram ("we simply delete the arrows/factors from the planning world"). Since I am not an expert in this area I can't tell how much of an issue this is: the positive take (as above) is that it's clear from the diagrams, the negative take is that there is still lots of room for ungrounded hypothesising without further specification of a model. But since this is philosophy, that's part of the process. 

Minor typos/issues:
- "If he model parameter above"
- Why use S as the transition function, not T as is standard elsewhere?
- I don't get the "digital actuator" part of the mode design. Does this mean A_0 only affects the physical state in S_2? i.e. there is a digital effect in S_1 then a physical effect in S_2?
limitations_and_societal_impact: The limitations section is honest and enjoyable to read
review_title: null

Meta-review: All the reviewers have similar recommendations for the paper. The authors and the reviewers engaged in detailed discussions on several sub-topics. The reviewers raised concerns about novelty, impact and real-world applicability (amongst other things). I commend authors for their constructive approach towards the reviews and the process.

I also recommend authors to follow anonymization guidelines as suggested by the reviewer NWTJ. Although given similar recommendation across the board from all the reviewers, the issue was not a factor in my final recommendation.

Decision: Reject