Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper introduces a new approach to explainable models called CST, or causal self-talk. There are 4 variations presented. The paper also defines several “desiderata” that should guide explainable AI systems such as faithfulness and scalability. Authors claim only self-models can maximize desiderata and CST is an example of that. There are experiments of CST-based models in a virtual environment called DaxDucks. 
strengths_and_weaknesses: Originality:
This paper is original in introducing and testing out the new concept of causal self-talk for explainable AI. It gives a pretty thorough coverage (concepts, methods, and experiments) of the new idea and offers several variations. The paper also defines concepts good to optimize in explainable AI.

Quality:
This paper has mixed quality. Some sections look underdeveloped. Section 2, for instance, seems pretty important considering their claimed contribution but looks not fully fleshed out. I can tell the authors did a lot of work for this paper and perhaps there’s much valuable insights here but it’s lost in the presentation and explanation.

Clarity:
The paper is well written and clear in some sections (lit review and historical overview) but not others (methods and experiment). The writing is somewhat experimental/creative(?) in some parts. Section 4 in particular, is confusing. Maybe Figure 1 can be improved to better relay the idea of time-step inputs? 


Significance:
Their newly introduced concept is interesting and paves a path forward for the XAI community on causal methods. Their limitation of memory states not currently workable for general multi-time steps is a big one. Overall it’s an interesting early idea and implementation that can be built on with future work.

questions: Questions:
1.Pg. 1, line 30 “ We argue that only the AI system itself is capable of supplying a model that fulfills all the key desiderata”: this seems to be a big claim. I’m not convinced this is argued considerably in the paper or really the focus. The paper does propose one such method but doesn’t build a case for it being the only such system. 
2.Section 2: I would elaborate this section more. Especially “faithful” since that seems to be multiple desiderata in and of itself and vague. Also maybe add in-line citations.
3.Does this method only work for single time-step instances?

Typos and Writing suggestions:
1.Pg. 3, line 86, “au naturale”: odd word choice
2.pg . 1 line 58, “to apply it to any network” change to “to apply it to any size network”
3.Pg. 1, line 30 “key desiderata”: I would flesh out these desiderata (which are discussed in section 2. 
4.Pg. 3, line 117 “feed the model’s outputs as inputs”: this seems to be the crux of the whole paper. I would introduce this way earlier.
5.Pg.2 line 59 “accurate, factual, and truthful”: seems to all mean the same thing?

limitations: I can tell this paper was a lot of work to write. There’s lots of parts (good) but they're all kind of underdeveloped (bad). The main limitation is that it’s very hard to follow. The methods and experiments sections can be clarified. If the authors can do that successfully, this could be a huge improvement. Also, the paper tries to do a lot of things in just 9 pages. There’s the desiderata definitions and start off sounding like it’s a conceptual/theoretical paper. Then it introduces and experiments with four different variations of CST.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper focuses on the problem of improving the faithfulness and explainability of AI system. The authors proposed a architecture based on POMDP and add a decoder to the internal representation to generate semantically meaningful information that could be further leveraged for internal state update and future policies. The authors showed that by intervening on the semantically grounded information, models could learn from the causal self-talk by incorporating prior internal representations with new semantic information. The experimental results show improvement of the models causal faithfulness (i.e., when passed in new semantic information, how often will the model act following the guidance) in a synthetic environment.
strengths_and_weaknesses: [+] In this paper the authors listed five desiderata for explainable AI systems and designed a corresponding model architecture that contains semantic outputs at each time-step. The proposed architecture is general enough to be a common backbone of explanatory models with different design choices of intermediate representation used and decoder architectures.

[+] The proposed causal self-talk asks the model to condition on the semantic output $z_t$ of itself as input and do counterfactual reasoning given a state $m_t'$ that is not aligned to the $z_t$. With this intuitive design, the model can learn to ground its internal representation and policy on the semantically meaningful representation. This makes the model follows from the explanations $z_t$ it provided and could be acting faithfully with new semantic guidance.

[-] One concern of this paper is on the causal faithfulness. The current definition provided by the authors is to what degree would the model act consistently with the intervention of semantic input $z_t$. This definition to me seems directly related to some current studies in language-conditioned policies learning or grounded policy learning. As the overall goal is to have model that not only acts with its own semantic output but also act consistently given new semantic output, there should be other offline learning methods that solves similar issue. This makes potentially makes the self-talk process less significant.
questions: As stated previously, my questions are as follows:
1. How significant is the self-talk process compared to grounded representation learning that requires $z_t$ and $m_t$ to be aligned?
2. The proposed 3 causal-self talk models have similar performance under current experimental settings. I was wondering if there will be specific scenarios where some of them would be best (e.g., is CST-MR always subordinate compared to the other two or could it be useful in some specific scenario). It will also be interesting if the authors could discuss if the composition of the three methods is feasible and will perform well (since each of them works toward the same goal)

limitations: The authors have stated their limitations thoroughly in the paper.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors delineate a set of desiderata for interpretabiliy of RL systems and then build a causally inspired interpretable solution in which the agent generate a textual output to describe its internal state and utilizes it as a form of memory to update its internal state during the course of a trajectory rollout. The causal aspect stems from interventions on the internal state of the agent being set to its initialization value. The authors evaluate their method on their own benchmark.
strengths_and_weaknesses: Strengths:
1. Well Written paper: The authors describe not only the intuition behind their work but also how the intuition came into being. This is useful when learning to do research in a new field. 
2. Important problem: The authors have selected an important problem. Explainability of RL agents is key to making RL agents safer and interpretable. 
3. New evaluation environment. The authors have built their own RL environment which (if open-sourced) could yield future results. 
4. Detailed Desiderata: The authors define their own set of desiderata for interpretable RL systems. 

Weaknesses:
1. No pictorial representation of architecture. The authors should provide a figure depicting a rollout of the RL agent in the environment. I can conceive of this by representing an MDP along with a rollout of the policy for a few steps. 
2. Biggest Gains are in the most simple settings: Providing the textual input as a one-hot encoded vector generates most gains. The authors could try to pretrained embeddings for text to improve this.
3. Highly simplistic evaluation setting: There exist multiple environments with textual annotations, e.g., https://openreview.net/forum?id=eCPCn25gat which could allow for similar experiments. Experiments on them would be useful. My worry is that the approach will not scale.
4. Not enough explanation delineating the relationship between experiments and desiderata: While defining desiderata is helpful, a callback to the desiderata is necessary in the experiments sections describing how they were achieved. This is missing. 
5. Missing citing work: https://arxiv.org/abs/2010.03110 and https://proceedings.neurips.cc/paper/2021/file/c1722a7941d61aad6e651a35b65a9c3e-Paper.pdf and https://ieeexplore.ieee.org/document/9561439 


questions: 1. Do you think the intervention could hamper policy learning in the complex environments? It appears that “resetting” the internal state constantly to the initialization value could potentially be harmful. 
2. Do you have any plans for releasing code + environment?
3. Have you considered how the method might be used for language-conditioned imitation learning? One could feed in instructions instead of z from the previous time step. This could work like a conversational RL agent where it takes an instruction and performs some actions and then returns a string of text for conversation. 

limitations: I believe the paper in its current form needs significant improvement. There seems to be a disconnect between the evaluation and the desiderata which can be bridged through thorough experiments. Additional work is necessary before the work is ready for publication. I wish the authors all the best for this.
review_title: null

final_score: rating: 9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors define 5 desiderata for AI explainability: grounding, flexibility, minimal-interference, scalability, and faithfulness. They show how these desiderata are satisfied by a system that learns an internal model via "causal self-talk" (a data augmentation strategy in which externally grounded information is decoded and then rerouted back into the internal state). Intuitively, the decoded information functions as an explanation for why the system behaved in a particular way. To enforce "causal faithfulness" (the decoded information exerts a causal influence on the system's output), the authors train the decoder to communicate information to earlier versions of the same agent. This creates the information asymmetry necessary for the agent to use the decoder output. The authors explore 3 different objective functions for training the decoder (based on reinforcement learning, memory reconstruction, and policy distillation). They find that the reinforcement learning variant performed best in terms of causal faithfulness and "semantic control" (using decoded information to obtain reward).
strengths_and_weaknesses: Strengths:
- The work is original, as far as I can tell (I am not an expert in this area).
- The paper is clearly written and logically argued.
- The ideas are interesting.
- The experiments demonstrate the efficacy of the proposed method.
- Given that explainability is an important real-world problem, the results are potentially significant.

Weaknesses:
- I thought the experiments were useful for demonstrating the efficacy, but they also felt rather contrived relative to real-world problems where explainability is needed.
- Only variations of the proposed system were studied. This has the advantage of provided well-controlled comparisons, but it doesn't establish to what extent the system outperforms previously proposed systems. 
- This is perhaps less important, but I would have thought that human judgment of explanatory adequacy is important for evaluating real-world explanation systems. This matters at the end of the day, right?
questions: If the authors can address the two main weaknesses raised above (somewhat contrived experiments and lack of comparison to other approaches) then I will be happy to increase my rating.

UPDATE: the authors have addressed these concerns. Although nothing really substantive has changed in the paper, I understand their reasoning as well as the page constraints.
limitations: I don't see any major negative societal impact of this work. I thought the authors did a good drop addressing limitations and future directions.
review_title: null

Meta-review: metareview: This paper proposes causal self-talk (CST) as a means to obtain more explainable AI systems. The work lists a set of desiderata for explainable AI and argues that CST satisfies this set. The paper is well written and the experimental results, although in a toy setting called "DaxDucks" are reasonably convincing. The reviewers lean towards acceptance, with Reviewer g2gH in particular strongly advocating for the work.

I like that this approach is generally applicable in principle, and I strongly dislike that it is only showcased on one (non-open source, toy) task. The other qualm I have is that the introduction reads way too much as if the authors came up with this all on their own, and I strongly encourage them to pay proper respect to prior work in this field, rather than stashing all that work away as a long enumeration in the related work section, which most people will gloss over.
confidence: Less certain
Decision: Accept