Explainability Via Causal Self-Talk
Authors: Nicholas Andrew Roy
Junkyung Kim
Neil Charles Rabinowitz
Venue: NeurIPS 2022 Accept (2022)
Keywords: explainability
reinforcement learning
deep learning
causality
interpretability

Abstract: Explaining the behavior of AI systems is an important problem that, in practice, is generally avoided. While the XAI community has been developing an abundance of techniques, most incur a set of costs that the wider deep learning community has been unwilling to pay in most situations. We take a pragmatic view of the issue, and define a set of desiderata that capture both the ambitions of XAI and the practical constraints of deep learning. We describe an effective way to satisfy all the desiderata: train the AI system to build a causal model of itself. We develop an instance of this solution for Deep RL agents: Causal Self-Talk. CST operates by training the agent to communicate with itself across time. We implement this method in a simulated 3D environment, and show how it enables agents to generate faithful and semantically-meaningful explanations of their own behavior. Beyond explanations, we also demonstrate that these learned models provide new ways of building semantic control interfaces to AI systems.

Explainability Via Causal Self-Talk
Nicholas A. Roy" DeepMind
Junkyung Kim" DeepMind junkyungädespmind.comm com
Neil Rabinowitz" DeepMind
nroy@deepmind.com
ncr@deepmind.co com
Abstract
Explaining the behavior of AI systems is important problem that, in practice, is generally avoided. While the XAI community has been developing an abundance of techniques, most incur set of costs that the wider deep learning community has been unwilling to pay in most situations. We take a pragmatic view the issue, and define set of desiderata that capture both the ambitions of XAI and the practical constraints of deep learning. We describe an effective way to satisfy all the desiderata: train the Al system build causal model of itself. We develop an instance of this solution for Deep RL agents: Causal Self-Talk. CST operates by training the agent to communicate with itself across time. We implement this method in a simulated 3D environment, and show how it enables agents to generate faithful and anraniically-menninge explanations, we also demonstrate that these learned models provide new ways of building semantic control interfaces AI systems.
explanations of their own behavior. Beyond
Introduction
As modern machine learning systems become powerful and embedded in lives, the need to have these systems explain their behavior becomes increasingly urgent. Despite incredible performance in variety of domains, almost all systems are completely unable provide satisfying answer to the simple question, "Why did you do that?".
While there have been innovations in particular explainability methods [e.g.
56
66 and
some attempts to apply techniques to domains that demand it [e.g. (23 (65], the vast majority of deep learning applications do not engage with these techniques. This is in part because there few general methods, and most incur costs such as reduced performance and scalability; compounding this, existing techniques often fail to deliver useful credible account of the systems' behavior.
We believe that the reason for this is that explanation systems tend to fail to satisfy at least one of five key desiderata discussed in detail below, namely: groundedness, flexibility, minimally-intteffriing scalability, and faithfulness. Two particular failure modes are most common: the addition of an explanation system limits generality hurts performance (e.g. when it requires explicitly structuring the base system); or the explanations it produces cannot trusted to be accurate (e.g. with many external or hoc methods). An ideal explanation system would avoid both these pitfalls.
In this paper, we propose one such solution. We take a pragmatic viewpoint that providing expla- nations of a system's 's behavior is, in essence, a task of building a model of that system. We argue that the AI system itself well rell-positioned to supply a model that fulfills all the aforementioned desiderata. We thus train the base system to supply "self-model" alongside its main representations. Importantly, the base system and self-model are subject to different constraints: we want the base system to be primarily optimized for performance, while we want the self-model to take on faithful,
Eqqual contribution Cooreespondigg author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

ienantically-interpeeeee form. The different form allows the self- model to deliver supplementary utility that the base system does not, e.g. providing an interface external users, and enabling them to understand, predict, and control the base system in convenient ways.
Our contributions are follows. In Section 2 articulate five key desiderata for explanation systems. However, there is some tension between these. In Section we describe how common historical XAI techniques resolve these trade-off's by making extreme choices between the desiderata. Section A marks a transition from discussing the desiderata in theory to exploring them in practice, as we demonstrate that a balance can be found between the five desiderata by training AI systems to build self- models. In particular, we introduce solution concept for Deep RL agents: Casual Self-Talk (CST), in which an agent must learn to use its own outputs as inputs. In Sections/516 we show that CST allows model-free RL agents in an embodied 3D virtual environment to faithfully explain their behavior terms of semantically-access
beliefs about the state of the world.
2 Desiderata for explanatory models
Our goal is to build an explanatory model of an Al system, i.e. mechanism that translates between its internal representations and/or computations, and some other representational form. We set out five key desiderata for such model. We flesh these out in more detail in Section 3
Grounded. For the model to be useful to external users, it should deploy representation language that is cemantically-gronnde This may require additional data to train.
Flexible. The method should be able to produce models of many different representational forms (e.g. classes, graphs, strings). Ideally it should also be sufficiently general to be agnostic the base system's input/output modalities, and possibly even its overall architecture.
Minimally-interferinn Training and inference on the model should have as little impact possible on the training and inference of the base system. Solutions which are disruptive to performance are highly unlikely be adopted in practice.
Scalable. Once a solution is found, it is straightforward to apply it to any size network.
Faithful. The model should provide accurate descriptions of the underlying system. The gold standard is causal model of the agent, which facilitates validation of its accuracy through interventions (3]] This is possibly the most challenging to achieve, and is discussed further in the specific context of decoders Section
number of past works have also articulated desirable properties for explanations [e.g. 28
35
36
38 49 553 69
Our desiderata focus less on the qualities which constitute good explanation per se, but instead on what is desired of the method itself that generates the explanation.
3
Common historical solutions
We examine whether various existing explanation techniques satisfy each of these desiderata.
Do nothing. In standard Deep Learning (DL), we take the base system's 's learned internal repre- sentations at face value. Using the base system as a model of itself clearly has benefits of being minimally-inlerferingg (no changes are made), scalable (as much as DL itself), and faithful (since the internal representations are causal to the base system's 's output). However, the approach entirely sacrifices grounding and flexibility.
Attribution methods. A set of popular techniques attempts to model neural network's 's decision- making at the input level, by estimating contributions to its output from input features [c.g. [8] 27 56). 71, or by finding optimal inputs for a given output (they are entirely post-hoc)
or training examples [e.g. 6 14
42
43
46
[e.g. (55 73). Such approaches benefit from being
and scalable (they can be automated). However, some methods have been empirically shown to be
Note that some terms, such as 'faithfulness', do not have standard definitions across this literature [35

---

unfaithful 7 while methods which explicitly aim for faithfulness such as LIME [13 and Anchors 660| are only local in scope, and are expensive scale. Most importantly, all such methods sacrifice flexibility: one is constrained to produce explanations of very particular form, e.g. expressed in terms of the input modality, whose interpretation may be highly subjective.
Explicitly structuring networks. A sub-field of deep learning has emerged wherein the inter- mediate computations are constrained to particular forms, and/or are grounded by external data 62 63 72). As explanations of behavior, structured intermediates are attractive as they are both grounded and faithful by design. However, this is achieved at the expense of minimall-nterffreneee, as requires fundamentally altering the base system to explicitly depend on this structure. It is also limited in flexibility and scalability, as the technique can also only be applied when sufficient domain knowledge or data is available, and only in the form dictated by these prior constraints. These issues are exacerbated by the fact that the representations required to optimize performance objectives are often at odds with those that would subserve explainability.
[e.g. 5 10 16
23 25
Fine-grained mechanistic interpretability. One common approach is to allow a base system to train and run without interference, and to post-hoc dissect its representations or computations, often using techniques borrowed or inspired from neuroscience [e.g. [7] 21 26 [7]. Such approaches champion the descriptions of the actual causal processes operating within the base system. However, this comes at huge cost scalability. Many human hours are typically expended in reverse engineering, often yielding results that are limited to a handful of neurons or circuits seen in a single network. Moreover, by making strong commitments respect the internal representations of the base system, it is very difficult to ground the models that arise, let alone to flexibly choose the form they take.
desideratum. In the best case, they promise uncover faithful
Decoders. Decoders can be viewed as trained tools to map from the internal private language of
a base system to an interpretable representation space [e.g. IT [5| 31 44 52 58 74]. This approach has many attractive qualities: it produces base system's 's internal private language; flexible in the choice of modality; scales well; and is minimally-ine inttmallyyitterfering. insofar as decoders can be separated from the inference path of the main computational graph and gradients can be stopped from propagating to the base system if desired (though are often not to assist representation learning, [e.g. 3 37 40 48 54]). This is all achieved, however, at the expense of faithfulness, as decoded outputs are off the causal path of the base system 47 Decoders just report whether information is present in the source representation, not whether
representations of the
that information is actually being used for behavior 12
20 59 Similar faithfulness issues arise
with post-hoe behavioral analyses [e.g. 19 30
39
50
66
70
Solution concept
The challenges faced by previous attempts at explainability show that the tensions between the desiderata are considerable. It is nonetheless possible to reason way forward.
Motivation. First, we want a description of the base system that is both grounded and flexible. The internals of the base system are neither. While we could explicitly structure the base system change this, this would violate the minimal interference desideratum. Thus we need model of the base system. Second, the solution must scalable. This means we cannot rely on human labor to map from the base system's 's representations to the model's 's representations. The only scalable approach is to learn the model. Third, the solution must be faithful. Our gold standard of faithfulness requires the ability to validate the model by intervening on it. To do this, one must be able to map in the opposite direction: from interventions on the model onto interventions on the base system. Because both the base system and model are learned, this mapping between them must also be learned. The most direct way of learning this mapping from model to base system is to feed the model's 's outputs as inputs to the base system.
solution to these problems is thus "self-model" which intertwined with the base system. We develop an instance of this solution in a particular setting: a Deep RL agent, acting in an episodic POMDP, which builds a model of itself in terms of set of cemantically-groundee beliefs about the state of the world. We consider how to expand the scope of this technique in Section 7
3

---

1641
114-1
1M4-1
27
3[-3
271
Mi+1
Ordinary decoder
(b) Re-routed decoder
TT
20
21
22
R3
23
WT
24
1/
true, online
mgg
YTR1
300
221
222
273
21
(1--0)
(1-00)
[1--0) (1-0)
m00
mlg
[1e-0]
(1--0)
(1-00)
counterfactual
(t#1 t=0)
400
422
223
374
23
x2 [3--0) (--
m0g
(3e-0)
counterfactual (t#3 -- t=0)
370
314
NT 200
mmg
mn
(3--))
counterfactual (t=3 -=3-t=1)
21
Causal Self-Talk (CST)
Figure I: Architectures. (a) Decoders offer a common and simple solution modeling base system, albeit one that lacks faithfulness. (b) Re-routing the output of decoder back into the base system is not sufficient to remedy this. (e) The CST architecture expands on the re-routed decoder by also allowing memory states to be reverted to earlier states before applying a state update. For example, in the second row, the memory state my reverted to mig so that the (true, online) message must inform the agent about changes to its current state and/or required future policy. Similarly, in the final row, the memory state m4 is reverted to m, so that the (true, online) message 23 must inform the agent accordingly. Black denotes variables from the true, online rollout of the agent. Red denotes variables after counterfactual intervention. The variant CST used (CST-RL/MR/PD) determines how the counterfactual changes are produced and trained.
Agent construction. We follow the standard construction of POMDP. We denote the agent's observation at time t within episode as and assume that it maintains an internal state which we can take be, for example, a recurrent state (such as from an LSTM), or a variable-length array of embeddings of previous observations æ0a (as used in Transformer). We denote the agent's 's state update function as fe, with learned parameters, 0, s.t. m4+1 = fo m²). We denote its policy as Te hhwhhmmm (m:).

---

Our construction starts with a decoder, shown in Figure Ta. A decoder maps from the state my to an output 2, ((2x), via learned function 90 m, - ((2). This mapping is trained using target space. As we previously identified in Section 3
data E DD that lives in cemantialll--meannngee decoders benefit from being grounded, flexible, scalable, and
However, their weakness is faithfulness. This is because the output 2, merely reflects what information is present in me, and not whether that information is actually used in the computation of ne.
Defining faithfulness. We now chart path forward from decoders, with an aim of making them faithful. To do so, we first define two metrics for faithfulness, one weak, one strong:
Correlational faithfulness: if the decoder reports a value of ž₁ = 2, to what degree is the behavior (from the policy л) consistent with this value, 2?
Causal faithfulness F of z', to what degree would this produce behavior consistent with this new value, 2', rather than its original value, =?
if we were to counterfactually intervene on 2t, changing it to value
Causal faithfulness is much stronger property than correlational faithfulness. It's highly desirable because it allows one to validate the agent's commitment to the value of 2, at runtime by making interventions on it. This validation mechanism is especially useful when out of distribution.
Re-routed decoders. While it is possible for decoder to have a degree of correlational faithfulness, it is structurally impossible for to exhibit causal faithfulness. A simple fix is to route the output of the decoder back into the internal representation (Figure[b), e.g. by concatenating 2 with T++1. This amounts to incorporating the decoder output in the state update rule, with m4+1 = fo ([z1+1, 2r], me).
The difficulty with this simple solution is that there is no guarantee that fo will actually "listen" to 2; when producing m4+1- This is because the information in 2 is largely not completely redundant with m,, This may be further exacerbated by the presence of any stochastic sampling in the generation of 2, from my. Indeed, as shall be described Section our experiments find that re-routed decoders make no measurable progress beyond ordinary decoders on causal faithfulness.
How do we render the function fe sensitive to the value of z, when there is redundancy between žt and m, ? It is useful to frame the problem in terms of communication. The decoder pathway can be viewed as a coxternally-groundeee communication channel between the internal state and itself, i.c. a form of "self-talk" or "inner speech" [67] The problem therein is that the speaker (m,) and the listener (also m,) are identical, so there is no useful information in the message 2 for the listener.
Causal self-talk. To overcome this problem, we need to create an information asymmetry between speaker and listener. We introduce Causal Self-Talk (CST): we encourage fe to be sensitive to the message z by creating an augmented data distribution where the speaker and listener diverge. Fortunately, we have ready source of alternate listeners: internal states my for t' < t.
We thus provide the decoder output with an additional role: to compactly and effectively communicate information to earlier versions of the same agent (from the same episode), via semantically- grounded communication channel. Formally, this takes the form of an intervention (Fig [p): the internal state, m4, is replaced with counterfactual one, my, before being integrated with the original message (2₂) and next observation (I++1), ), in order to produce a counterfactual internal state, (tet')
(tt-t') he (t++t)) )). m fe( me). This yields counterfactual policy, Ti+1
This alone, however, is insufficient to fully specify CST algorithm. We must also stipulate the desired outcome from enacting these interventions. We describe three alternative choices:
The messages z, should allow earlier versions of the same agent maximize return given the current environment state. To train this, interventions must occur online (ac- cording to some schedule), and we must allow the agent to act according to the counterfactual policy following an intervention. We then use the maximization of subsequent discounted reward as the desired outcome from the intervention, via reinforcement learning (RL).
CST-RL:
*Our use of this term distinct from the "causal faithfulness condition" in causal modelling 644.
5

---

CST-MR:
The messages 2, should allow earlier versions of the same agent to reconstruct the
next internal state, Mz+1- Unlike CST-RL, this can be done entirely in replay. without requiring the agent to ever act according to the counterfactual policy on live environment. We add a
weighted memory reconstruction (MR) term to the overall loss: Cas r = ||m
(tet') -++ 1 1|2.
The messages 2 should allow earlier versions of the same agent to recover the true current (and future) policy. As with CST-MR, interventions can be simulated entirely in replay. We add a weighted policy distillation (PD) term to the overall loss, using discounting function 7(): LPD
CST-PD:
Esiso
DKL TT- || [+A (t-t))
CST and the five desiderata. CST offers a different, less extreme tradeoff between the desiderata. By design, inherits decoders' resolution to several desiderata. Like decoders, CST can achieve grounding by supervising the representations 2 against a signal of known semantics. It is also equally flexible. Relative to decoders, there is some trade-off with minimall-nterferrncee as the decoder output 2, is now an obligate input into the next time step. However, much like decoders, the effects on training can, in principle, be contained through appropriate gradient flow, e.g. by limiting updates to parameters of the state update function, fe, and/or fine-tuning pre-existing base system.
CST is highly scalable. CST-RL is straightforward to implement: in its simplest incarnation, fixing t' = O reduces to dropout on the whole internal state, my. The use of RL, however, occurs at the expense of additional data as requires online execution of the counterfactual policy, CST-PD and CST-MR, in contrast, are entirely self-supervised: they make use of the existing trajectories generated during the training of the base system. These methods augment the data in a manner similar to the interchange intervention technique of [25), using self-imitation as the desired outcome of the intervention. Depending on the degree of staaiioatiiitt afforded by the particular environment (and its degree of partial observability), the size of the augmented dataset can scale quadratically with the length of each episode (from choosing both and t'); in some settings, it may be possible to exceed this by sampling counterfactual previous states me from other episodes.
Finally, unlike decoders, CST is explicitly trained to provide causal faithfulness, by facilitating counterfactual control. Our goal in the following two sections is to evaluate how well it does this.
5 Experiments
Task. We study variants of CST in a 3D virtual environment built in Unity [] 68 This features a fixed-layout indoor space comprising five rooms, each with a different wall color. The agent receives as input a ss--eesoon visual observation and a text observation. The agent can freely navigate using a 4D continuous action space (a []1, consisting of in-plane movements and a 2-DOF rotation.
We developed simple task in this environment, called DaxDucks (Figure 2}. We designed DaxDucks to: (1) be easy for an agent to learn good policy; (2) require an agent to maintain and update belief' about the latent environment state in order to maximize reward; (3) provide source of data to ground the self model; and (4) provide a means to evaluate the faithfulness of the learned self-model.
DaxDucks is an instruction- based, fast-binding search task, where the agent is rewarded for finding duck that exhibits an instructed tag. Each episode consists of sequence of trials. At the beginning of each trial, the agent's avatar spawns in the center of the middle room facing random direction. Four identical ducks spawn at the center of the four outer rooms. Each duck is randomly assigned one of four tags ("dax", "gavagai", "kl eeg", or "plork"). The agent is instructed to collect (i.e. collide with) the duck with specified tag via the text observation channel, e.g. "Collect gavagai." The agent may observe duck's 's tag by entering the duck's room and orienting the center of its view towards the duck. This adds an additional string to the text observation, e.g. "This is a kleeg.". ". Colliding with any duck terminates the current trial, in which case the agent is respawned in the center of the middle room again and random new instruction is delivered; a reward is also delivered if the agent collided with the correct duck. Finally, when a new trial starts, the ducks usually maintain their tags, but with a small probability (p=a = 1) the tags are randomly shuffled between the ducks. Thus the agent can only maximize reward by effectively retaining a belief state over which tag belongs to which duck. Each episode lasts for 5000 steps regardless of the number of trials completed.

---

Agent spawns room and receives instruction
Agent approaches duck and observes its tag
Agent respawns central room and receives instruction
kleeg
dax
kleeg
plork
gavagai
Agant touches duck
kleeg
gavagai
plark
gavagai
plark
Shuffle lage
pm0.1
dax
kleeg
pmo.9
gavagel
plork
Collect gavagai.
Collect gavagal. This is gavagas.
Collect kleeg.
Figure 2: The DaxDucks task. Top-down views are for exposition only.
Agent architecture. For the base system, we used a standard model-free agent (Appendix A): inputs encoded with ResNet (images) and LSTM (text); LSTM memory module; MLPs for policy and value heads; and trained with V-trace [22] All agents achieved similarly high return.
We demonstrate the flexibility of grounding by considering two forms for the self-model. In the one-hot form, the message 2n comprises four one-hot vectors (one per tag), expressing which room the agent believes each tag be in. In the language form, the message is synthetic language string, expressing which room the agent believes a single chosen tag to be in.
For the one-hot form (Section 6). we parameterize g(2)) = II ((z7) as a product of independent categorical distributions q(x]) for the four tags with go (an MLP) outputting the parameters of each q(2[) ) at time t, and sample z ~ g(zt) concatenated with the encoded input at the next time step. For the language form (Section 6 we use LSTM for go. sampling directly. Examples from single trajectory are shown in Figure 3a (one-hot) and Figure 4a (language). We ground both forms of self-model by supervising against ground-truthh values. For the language form, this amounts to constructing a target & of the form:
"The <instructed tag> is in the <color> room."
Given the simplicity of the environment, throughout. This means that CST aims drive the decoder to communicate beliefs version of the agent from the start of the episode. Although this reduces the scale of data augmentation from quadratic to linear, and requires interventions on to be accompanied by reset of the memory state to mg. in return it makes it casier to measure the effects of interventions the agent's 's behavior.
implement all three CST algorithms using l' = 0 earlier
We used the following schedules for interventions training time. For CST-RL, interventions (m, & mu) occurred randomly, with probability given timestep For CST-MR, we simulated interventions in replay every timestep. For CST-PD, we simulated interventions in replay: we divided every trajectory into a sequence of blocks of variable duration, with p 0.03 probability that a new block would start at any time ( we computed LpD only up to a horizon of the end of the block, and with constant discounting function, 7 7(At) = 1.
0.03 that an intervention would occur at any
Results
One-hot self-talk. We start by considering the one-hot form for the messages = (Figure 3ap. compare CST against two baselines: the ordinary decoder (Ord-Dec; Figure [T]) and the re-routed decoder (RR-Dec; Figure [p). We focus on the metries of faithfulness previously introduced.
We
Correlational faithfulness. When no interventions are taking place, the agent's physical behavior congruent with the output of its self-model? To measure this, we filter to all times in evaluation episodes when the agent's avatar in the central room. We then determine the degree to which the belief attested in g(2)) matches the next-visited room, (whether not
7

---

Step 532
90
dax
correlational faithfulness
96
frequenc)
190
kleeg
90
plark
Collect kleeg. This is kleeg.
Step 537 Trial 23
82
Ord-Dec
RR-Dec
CST-RL
CST-MR
CST.AD
92
gavagal
91
Klogg
causal faithfulness
plark
Collect Kleeg.
Step 552
23
10
dax
requeen
20
Keag
Ord-Dec
RR-Dec
CST-RL
CST-MR
CST-PD
plark
Collect kleeg. This plork.
(a) Example one-hot messages from a CST-RL agent.
(b) Faithfulness results.
Figure One-hot-based self-talk. (a) Example messages during a trajectory. Top-down view (left), agent view (middle), and ((x) (right) during trajectory of DaxDucks, showing the updating ((x)) after the start of a new trial (second row) and after discovering the tags have been shuffled (third row). At each time point, the message is sampled from ((x)). (b) CST renders the decoder causally faithful. Average values across < N 10 training runs per method; error bars show 95% confidence intervals. Dashed lines indicate faithfulness expected from random messages.
the instructed tag is actually in r). For the one-hot message form, this amounts to reading off the value of {( = r) for the instructed tag т.
Causal faithfulness. When we intervene on 24, is the agent's subsequent physical behavior congruent with this new value? To measure this, run evaluation episodes where, at the start of each trial, we inject a message =' indicating that the instructed tag is in some random room r/. We then measure the probability that the agent actually visits room r' next.
CST maintains comparable levels of correlational faithfulness as the baseline decoders. However, the decoder baselines failed to exhibit meaningful degree of causal faithfulness. This is indeed impossible for Ord-Dec given the lack of any causal pathway from 2, to m4+1- Re-routing back to the internal state in RR-Dec does result in small increase above chance, but the effect is relatively tiny. In contrast, all CST objectives substantially increase the degree of causal faithfulness (Figure 3b).
Language-based self-talk. Next, we trained agents to model their belief state using messages in synthetic language form (e.g. Figure For correlational faithfulness, we obtain analogous values of q(2 r) by computing the likelihoods of generating the four messages "The <instructed tag> is in the <color> room. and normalizing these to sum to 1.
As with the one-hot form, we found that the greatest gain of CST was its ability imbue the self-model with causal faithfulness. The effect was most profound with CST-RL, and moderate with CST-PD. However, we found difficult get any traction with CST-MR; this was either disruptive to learning the base policy, or ineffective at causal faithfulness.
We note that the faithfulness results with the language form were overall weaker than for the one-hot form. This may be because the one-hot messages encode a room belief about all four tags.
Semantic control.
final test, we consider whether CST endows the self-model with sufficient causal power to act effective semantic control interface for the agent. We thus measure how

---

correlational faithfulness
Slep 502 Trial 17
"The gavagai is in the red room."
frequency
The govagal the roa room The gavagal in the ma noom The gevagal the red room
Collect gavagai. This gavagai.
526
Ord-Dec
RR-Dec
CST-RL
CST-MR
CST-PD
"The dax in the blue room."
clucowroom.
The dax the olll
causal faithfulness
Collect a dax.
frequency
Stop 535 Trial 18
"The dax is in the yellow room."
The dax the yeen The dax the yellro room. grellow
Ord-Dec
RR-Dec
CST-RL
CST-MR
CST-PD
"Sampled memo" Other samples
Collect dax. This plork.
(a) Example language messages from a CST-RL agent.
(b) Faithfulness results.
Figure 4: Language-based self-talk. Results as Figure
semantic control efficacy
200
rewaro
100
50
Ord Ord-Dec
RR. Dec
CS7-RL
CST-MR
CST-PD
Ord-Dec
RR-Dee Dec
CST-RL
CST-MR
CST.PD
Pie P
one-hot form
language form
Figure Efficacy of semantic control via interventions on the self-model.
effectively the agent can use injected information to obtain reward. We start with two baselines: (1) the return evaluation episodes where Psh. 1, i.e. where tags shuffled every trial; and (2) the return on evaluation episodes where Psa. = 0, i.e. where tags are never shuffled. Respectively, these represent the bounds of rewards the agent naturally obtains when maximal information about the location of the instructed tag. We compare these to the reward obtained on evaluation episodes where information about the tag location is provided through intervening Specifically, we set Psh = 1, and inject message z' at the start of each trial indicating the location of all the tags (for the one-hot form), or the location of the instructed tag (for the language form). The results (Figure 5) show that CST not only furnishes an agent with a commaniicallyggrunddee model that describes its behavior, but also one which can be used mechanism guide it.
acquire (1) minimal (2)
7 Discussion
We articulated five key desiderata an ideal explanation system for an AI system: grounding, flexibility, ninimal-interference, scalability, and faithfulness. Unlike existing approaches, we seek to satisfy all these by training the base system to serve a causal model of itself. We developed an instance of this solution for Deep RL agents: Causal Self-Talk. In CST, agent leams to serve

---

grounded and faithful model of its internal state by learning to communicate with itself across time. We demonstrated its effectiveness under various forms in simple embodied task in simulated 3D environment. We also showed that CST enables new mechanism for effective semantic control.
We presented three variants of the CST algorithm. CST-RL yielded the greatest degree of causal faithfulness and efficacy of semantic control. Here, online interventions at training time force the agent to learn how to use the content of its self-model maximize its reward. CST-RL's success, however, may reflect features of the particular task setting we study: here, there is a close alignment between writing a faithful and maximizing reward. This may not always be the case.
In contrast, CST-PD offers an attractive alternative for two reasons: (1) it trains entirely in replay, and thus potentially interferes less with the base system; and (2) as it primarily seeks to enable imitation of the current behavior, the faithfulness of the self-model is targeted by design. One challenge here is learning to capture the important features of behavior; more impressive gains may be possible using richer imitation learning techniques [e.g. 34] rather than behavioral cloning. We had the least success with CST-MR: there also may be theoretical obstacles to deploying CST-MR in some settings, e.g. when the dimensionality of the internal state is very large, or is of variable length.
Limitations. In our experiments, we memory states my to the state at time L' = 0, rather than the more general case of L' < L. We do this to simplify the presentation and evaluation of the technique, but it imparts two limitations: (1) it requires m, to be reverted to mg in order to effectively intervene on the message 2; and (2) it requires z to contain all behaviorally-relevar becomes unrealistic in more complex environments. When 2 contains only a partial description of m will be necessary for the agent to be able re-integrate injected z values with its current memory state. The more general case of t' < offers a viable path to achieving this.
limited form of CST, where we exclusively revert
information. While sufficient for the simple task at hand, this
Utility requires that the self-model be grounded in how a ready source of data can be used to train the self-model. How this method stands up in more limited data regimes, in OOD scenarios where the self-model hasn't 't been supervised, remains an open question. It may be possible in some domains to leverage semi-supervised methods, and/or to impose structure on the self-model helpful constraint.
maaatiiiallly-mmnnnfuuuul form. We have shown
Future opportunities. We focused here on expressing the agent's 's beliefs about the current state of the environment in the self-model. A complete answer to the question, "Why did you do that?" might also include an expression of an agent's 's goals, plans, and model of the world [53]. The schema we have developed here should be sufficiently flexible extend in this direction, given the right training data. It would also be of great value to determine how to apply these techniques beyond RL agents.
Our primary focus in this paper has been on building is only one of several utilities that a faithful self-model might deliver. For example, we demonstrate that causal self-models provide semantic control mechanism. This could be useful for guiding agent learning through human interaction [T] metacognitive reporting and control (9 24 29, or as
to yield explanations. However, this
an interface for -afety-critical applications [41 45 51 61). By learning to express its internal state
through a different representational schema, this may also open up avenues for new exploration or hierarchical reinforcement learning techniques.
Finally, training agents to constantly talk to themselves might prove to be data-efficient way of enabling agents to talk to others (and us!). Ultimately, this may pave way for us to invert of the flow of knowledge, allowing agents to not only answer, "Why did you do that?", but also "How?".
Acknowledgments and Disclosure of Funding
The authors would like to thank Edgar -aztme for early environment design, Hamza Merzic for agent infrastructure assistance, Allison Tam for help in agent development, Chen Yan for useful discussion, Lucy Campbell-Gillinghan for organizational support, and Johannes Welbl, Kory Mathewson, Pedro Ortega, Agnieszka Trabska-Barwinskee Chen Yan, Matt Botvinick, and Lisa Anne Hendricks for comments the work and/or the manuscript.
10

---

References
[1] Josh Abramson, Arun Ahuja, lain Barr, Arthur Brussee, Federico Carnevale, Mary Cassin, Rachita Chhaparia, Stephen Clark, Bogdan Damoc, Andrew Dudzik, and Others. Imitating interactive intelligence. arXiv preprint arXiv:2012.05677 2020.
[2] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Adv. Neural Inf. Process. Syst, 2018
[3] Zeynep Akata, Lisa Anne Hendricks, Stephan Alaniz, and Trevor Darrell. Generating post-hoc rationales of deep visual classification decisions. In The Springer Series on Challenges in Machine Learning, pages 135-154. Springer International Publishing, Cham, 2018.
[4] David Alvarez-Melis and Tommi S Jaakkola. Towards robust interpretability with Self- Explaining neural networks. arXiv preprint arXiv: 075388 June 2018.
[5] Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. arXiv preprint wxXiv:661.01196 November 2016.
[6] Rushil Anirudh, P Bremer, Rahul Sridhar, and JJ Thiagarajan. Influential sample selection: A graph signal processing approach. Technical report, Lawrence Livermore National LLL.(LL.L). Livermore, CA (United States), 2017.
[7] Akanksha Atrey, Kaleigh Clary, and David Jensen. Exploratory not explanatory: Counterfactual analysis of saliency maps for deep reinforcement learning. arXiv preprint 1rXiv:1912.05743, 2019.
[8] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus- Klaus-Robert Müller, and Wojeiech Samek. On Pixel-Wise explanations for Non-Linear classifier decisions by Layer-Wise relevance propagation. PLOS ONE, 10(7):e0130140, 2015,
[9] Andrea Banino, Jan Balaguer, and Charles Blundell. Pondernet: Learning to ponder. arXiv preprint arXiv:2 2021.
[10] Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzaler Vinicius Zam- baldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Caglar Gulcehre, Francis Song, Andrew Ballard, Justin Gilmer, George Dahl, Ashish Vaswani, Kelsey Allen, Charles Nash, Vietoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matt Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.0126. June 2018.
[11] David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. arXiv preprint arXiv: April 2017.
[12] Yonatan Belinkov and James Glass. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics, 7:49-72, 2019.
[13] Przemyslaw Biecek and Tomasz Burzykowski. Local interpretable model-agnostic explanations (LIME). Explanatory Model Analysis, pages 107-123, 2021.
[14] Jacob Bien and Robert Tibshirani. Prototype selection for interpretable classification. The Annals of Applied Statistics, (00):2403-2224, 2011.
[15] Tyler Bonnen, Daniel L K Yamins, and Anthony D Wagner. When the ventral visual stream is not enough: A deep learning account of medial temporal lobe involvement in perception. Neuron, 109(17):2755-22) 27667.66-77666,, September 2021.
[16] Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. MONet: Unsupervised scene decomposition and represen- January 2019.
tation. arXiv preprint
[17] Nick Cammarata, Gabriel Goh, Shan Carter, Ludwig Schubert, Michael Petrov, and Chris Olah. Curve detectors. Distill, 5(6), June 2020.

---

[18] Hyeoncheol Cho, Eok Kyun Lee, and Insung S Choi. Layer-wise relevance propagation of InteractionNet explains protein-ligand interactions at the atom level. Scientific Reports, 10(1), 2020.
[19] Grégoire Déletang, Jordi Grau-Moya, Miljan Martic, Tim Genewein, Tom McGrath, Vladimir Mikulik, Markus Kunesch, Shane Legg, and Pedro A Ortega. Causal analysis of agent behavior for ai safety. arXiv preprint 1rXiv:2103.03958, 2021.
[20] Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. Amnesic probing: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 9: 1:10-17 2021.
[21] Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, 7ac HatfieldDoodis Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021.
[22] Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, and Others. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machine Learning, pages 1407-1416, 2018.
[23] Jeffrey De Fauw, Jeffrey De Fauw, Joseph R Ledsam, Bernardino Romera-Paredee Paredes, Stanislav Nikolov, Nenad Tomasev, Sam Blackwell, Harry Askham, Xavier Glorot, Brendan O'Donoghue, Daniel Visentin, George van den Driessche, Balaji Lakshminarayanan Clemens Meyer, Faith Mackinder, Simon Bouton, Kareem Ayoub, Reena Chopra, Dominic King, Alan Karthike- salingam, Cían O Hughes, Rosalind Raine, Julian Hughes, Dawn A Sim, Catherine Egan, Adnan Tufail, Hugh Montgomery, Demis Hassabis, Geraint Rees, Trevor Back, Peng T Khaw, Mustafa Suleyman, Julien Cornebise, Pearse A Keane, and Olaf Ronneberger. Clinically ap- plicable deep learning for diagnosis and referral in retinal disease. Nature Medicine, 24(9): 1342-1350, 2018.
[24] Damien Fleur, Bert Bredeweg, and Wouter van den Bos. Metacognition: ideas and insights from neuro- and educational sciences. NPJ Sci Learn, 6(1):13, June 2021.
[25] Atticus Geiger, Zhengxuan Wu, Hanson Lu, Josh Rozner, Elisa Kreiss, Thomas Ieard, Noah Goodman, and Christopher Potts. Inducing causal structure for interpretable neural networks. arXiv preprint arXiv:2112.00822 2021.
[26] Gabriel Goh, Nick Cammarata f. Chelsea Voss Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah, Multimodal neurons in artificial neural networks. Distill, 2021.
[27] Yash Goyal, Ziyan Wu, Jan Emsst, Dhruv Batra, Devi Parikh, and Stefan Lee. Counterfactual visual explanations. In International Conference on Machine Learning, pages 2376-2384, 2019.
[28) Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. A survey of methods for explaining black box models. ACM computing surveys (CSUR), 51(5): 1-2), 2018.
[29] Jessica B Hamrick, Andrew J Ballard, Razvan Pascanu, Oriol Vinyals, Nicolas Heess, and Peter Battaglia. Metacontrol for adaptive imagination- based optimization. arXiv preprint 17X.v:170..02670, 2017.
[30] Bradley Hayes and Julie A Shah. Improving robot controller transparency through autonomous policy explanation. In Proceedings the 2017 ACM/IEEE International Conference Human- Robot Interaction, New York, NY, USA, March 2017. ACM.
[31] Lisa Anne Hendricks, Zeynep Akata, Marcus Rohrbach, Jeff Donahue, Bernt Schiele, and Trevor Darrell. Generating visual explanations. In European conference on computer vision, pages 3-19, 2016.
12

---

[32] Lisa Anne Hendricks, Ronghang Hu, Trevor Darrell, and Zeynep Akata. Generating counterfac- tual explanations with natural language. arXiv preprint arXiv:1806.0900 09809, 2018.
[33] Lisa Anne Hendricks, Anna Rohrbach, Bernt Schiele, Trevor Darrell, and Zeynep Akata. Generating visual explanations with natural language. Applied Al Letters, 2(4), December 2021.
[34] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning, Adv. Neural Inf. Process. Syst., 29, 2016.
[35] Alon Jacovi and Yoav Goldberg. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4198 4205, Online, July 2020. Association for Computational Linguistics.
[36] Alon Jacovi, Ana Marasović, Tim Miller, and Yoav Goldberg. Formalizing trust in artificial intelligence: Prerequisites, causes and goals of human trust in AL. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 624 635, 2021.
[37] Max Jaderberg. Volodymyr Mnih, Wojeiech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In Sth International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. DpenRevview.net20111 2017.
[38] Sarthak Jain and Byron C Wallace. arXiv:1902 10186, February 2019.
Attention is not explanation. arXiv preprint
[39] Zoe Juozapaitis, Anurag Koul, Alan Fern, Martin Erwig, and Finale Doshi-Velez. Explainable reinforcement learning via reward decomposition. In IJCA//ECAI Workshop on explainable artificial intelligence, 2019.
[40] Bilal Kartal, Pablo Hernandez-Lewt and Matthew Taylor. Terminal prediction as an auxiliary task for deep reinforcement learning. In Proceedings of the Fifteenth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, AlIDE' 19, Atlanta, Georgia, 2019.
AAAI Press.
[41] Huda Khayrallah, Sean Trott, and Jerome Feldman. Natural language for human robot interac- tion. In International Conference on Humaan-Robot Interaction (HRII. 2015.
[42] Been Kim, Cynthia Rudin, and Julie A Shah. The bayesian case model: A generative approach for case-based reasoning and prototype classification. Advances in neural information processing systems, 27, 2014.
[43] Been Kim, Rajiv Khanna, and Oluwasanmi O Koyejo. Examples are not enough, learn to criticize! criticism for interpretability. Advances in neural information processing systems, 29, 2016.
[44] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory Sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). arXiv preprint
November 2017.
[45] Jinkyu Kim, Suhong Moon, Anna Rohrbach, Trevor Darrell, and John Canny. Advisable learning for self-driving vehicles by internalizing observation-to-aceee rules. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR). IEEE, June 2020.
[46] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1885-1888 1894. PMLR, 2017.
[47] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. arXiv preprint arXiv:2007.0440 1,,.0000..44221111 July 2020.
[48] Guillaume Lample and Devendra Singh Chaplot. Playing FPS games with deep reinforce- ment learning. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI 17, pages 2140-2144 San Francisco, California, USA, 2017. AAAI Press.
13

---

[49] Zachary C Lipton. The mythos of model interpretability; In machine learning, the concept of interpretability both important and slippery. Queueing Syst., 16(3):31-57, 2018.
[50] Prashan Madumal, Tim Miller, Liz Sonenberg, and Frank Vetere. Explainable reinforcement learning through a causal lens. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 2493-2500, 2020.
[51] Cynthia Matuszek, Evan Herbst, Luke Zettlemoyer, and Dieter Fox. Learning to parse natural language commands to a robot control system. In Experimental robotics, pages 403-415, 2013.
[52] Thomas McGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik. Acquisition of chess knowledge in AlphaZero. arXiv preprint 1tx::::2//1.0920 November 2021.
[53] Tim Miller. Explanation in artificial intelligence: Insights from the social sciences. Artificial intelligence, 267:1-38, 2019.
[54] Piotr W Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andy Ballard, Andrea Banino, Misha Denil, Ross Goroshin, I. Sifre, Koray Kavukcuoglu, Dharshan Kumaran, and Raia Hadsell. Learning to navigate in complex environments. arXiv preprint arXiv:1611.0367 2017.
[55] Alexander Mordvintsew, Christopher Olah, and Mike Tyka. Inceptionism: Going deeper into neural networks. Google Research Blog, 2015.
[56] Alexander Mott, Daniel Zoran, Mike Chrzanowski, Daan Wierstra, and Danilo Jimenez Rezende. Towards interpretable reinforcement learning using attention augmented agents. Adv Neural Inf. Process. Syst., 32, 2019.
[57] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Danny Hernandez, Scott Johnston, Andy Jones, Jackson
Deep Ganguli, Zac Hatfield- Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. Transformer Circuits Thread, March 2022.
[58] Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Darrell, and Marcus Rohrbach. Multimodal explanations: Justifying decisions and pointing to the evidence. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, June 2018
[59] Abhilasha Ravichander, Yonatan Belinkov, and Eduard Hovy. Probing the probing paradigm: Does probing accuracy entail task relevance? arXiv preprint arXiv:2005.007/ 2020.
[60] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Anchors: High-precision model- agnostic explanations. In Proceedings of the AAAI conference on artificial intelligence, vol- ume 32, 2018
[61] Junha Roh, Chris Paxton, Andrzej Pronobis, Ali Farhadi, and Dieter Fox. Conditional driving from natural language instructions. In Conference Robot Learning, pages 540-551, 2020.
[62] Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206-21 215, 2019.
[63] Tianmin Shu, Caiming Xiong, and Richard Socher. Hierarchical and interpretable skill ac- quisition in multi-task reinforcement learning. arXiv preprint arXiv::712.00794. December 2017.
[64] Peter Spirtes, Clark N Glymour, Richard Scheines, and David Heckerman. Causation, prediction. and search. MIT press, 2000.
[65] Ian Tenney, James Wexler, Jasmijn Bastings, Tolga Bolukbasi, Andy Coenen, Sebastian Gehrmann, Ellen Jiang, Mahima Pushkarna, Carey Radebaugh, Emily Reif, and Ann Yuan. The language interpretability tool: Extensible, interactive visualizations and analysis for NL. models. arXiv preprint arXiv:2008.05122 August 2020.
14

---

[66] Nicholay Topin and Manuela Veloso. Generation of policy-level explanations for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 33, pages 2514-2521, 2019.
[67] S Vygotsky. Thought and Language. The MIT Press. MIT Press, London, England, 2012.
[68] Tom Ward, Andrew Bolt, Nik Hemmings, Simon Carter, Manuel Sanchez, Ricardo Barreira, Seb Noury, Keith Anderson, Jay Lemmon, Jonathan Coe, Piotr Trochim, Tom Handley, and Adrian Bolton. Using unity to help solve intelligence. arXiv preprint 1.002940 November 2020.
[69] Sarah Wiegreffe and Yuval Pinter. DEt8::9900.4422. August 2019.
Attention is not not explanation. arXiv preprint
[70] Herman Yau, Chris Russell, and Simon Hadfield. What did you think would happen? explaining agent behaviour through intended outcomes. Adv. Neural Inf. Process. Syst., 33:18375-18386. 2020.
[71] Chih-Kuan Yeh, Joon Kim, Ian En-Hsu Yen, and Pradeep K Ravikumar. Representer point selection for explaining deep neural networks. Advances in neural information processing systems, 31, 2018.
[72] Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, and Others. Deep reinforcement learning with relational inductive biases. In International conference on learning representations, 2018.
[73] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 818-833, 2014.
[74] Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba. Interpretable basis decomposition for visual explanation. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Discussion.
(c) Did you discuss any potential negative societal impacts of your work? [No] The work itself is focused on achieving explainability, which is a topic focused on positive socictal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or URL)? [No] Our agent training code is built on a core library that has yet to be open-sourced. If this library is published prior NeurIPS camera ready deadline, the code for the design presented in this paper in the camera-ready version.
will endeavor open-source
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix A.
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] All results figures include error bars.
15

---

(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix A.
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "eO8YOBTFHY", "reviewer": "Reviewer_KW43", "summary": "This paper focuses on the problem of improving the faithfulness and explainability of AI system. The authors proposed a architecture based on POMDP and add a decoder to the internal representation to generate semantically meaningful information that could be further leveraged for internal state update and future policies. The authors showed that by intervening on the semantically grounded information, models could learn from the causal self-talk by incorporating prior internal representations with new semantic information. The experimental results show improvement of the models causal faithfulness (i.e., when passed in new semantic information, how often will the model act following the guidance) in a synthetic environment.", "questions": "As stated previously, my questions are as follows:\n1. How significant is the self-talk process compared to grounded representation learning that requires $z_t$ and $m_t$ to be aligned?\n2. The proposed 3 causal-self talk models have similar performance under current experimental settings. I was wondering if there will be specific scenarios where some of them would be best (e.g., is CST-MR always subordinate compared to the other two or could it be useful in some specific scenario). It will also be interesting if the authors could discuss if the composition of the three methods is feasible and will perform well (since each of them works toward the same goal)\n", "limitations": "The authors have stated their limitations thoroughly in the paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[+] In this paper the authors listed five desiderata for explainable AI systems and designed a corresponding model architecture that contains semantic outputs at each time-step. The proposed architecture is general enough to be a common backbone of explanatory models with different design choices of intermediate representation used and decoder architectures.\n\n[+] The proposed causal self-talk asks the model to condition on the semantic output $z_t$ of itself as input and do counterfactual reasoning given a state $m_t'$ that is not aligned to the $z_t$. With this intuitive design, the model can learn to ground its internal representation and policy on the semantically meaningful representation. This makes the model follows from the explanations $z_t$ it provided and could be acting faithfully with new semantic guidance.\n\n[-] One concern of this paper is on the causal faithfulness. The current definition provided by the authors is to what degree would the model act consistently with the intervention of semantic input $z_t$. This definition to me seems directly related to some current studies in language-conditioned policies learning or grounded policy learning. As the overall goal is to have model that not only acts with its own semantic output but also act consistently given new semantic output, there should be other offline learning methods that solves similar issue. This makes potentially makes the self-talk process less significant.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "OyJBosXDyHs", "reviewer": "Reviewer_AH1b", "summary": "This paper introduces a new approach to explainable models called CST, or causal self-talk. There are 4 variations presented. The paper also defines several “desiderata” that should guide explainable AI systems such as faithfulness and scalability. Authors claim only self-models can maximize desiderata and CST is an example of that. There are experiments of CST-based models in a virtual environment called DaxDucks. ", "questions": "Questions:\n1.Pg. 1, line 30 “ We argue that only the AI system itself is capable of supplying a model that fulfills all the key desiderata”: this seems to be a big claim. I’m not convinced this is argued considerably in the paper or really the focus. The paper does propose one such method but doesn’t build a case for it being the only such system. \n2.Section 2: I would elaborate this section more. Especially “faithful” since that seems to be multiple desiderata in and of itself and vague. Also maybe add in-line citations.\n3.Does this method only work for single time-step instances?\n\nTypos and Writing suggestions:\n1.Pg. 3, line 86, “au naturale”: odd word choice\n2.pg . 1 line 58, “to apply it to any network” change to “to apply it to any size network”\n3.Pg. 1, line 30 “key desiderata”: I would flesh out these desiderata (which are discussed in section 2. \n4.Pg. 3, line 117 “feed the model’s outputs as inputs”: this seems to be the crux of the whole paper. I would introduce this way earlier.\n5.Pg.2 line 59 “accurate, factual, and truthful”: seems to all mean the same thing?\n", "limitations": "I can tell this paper was a lot of work to write. There’s lots of parts (good) but they're all kind of underdeveloped (bad). The main limitation is that it’s very hard to follow. The methods and experiments sections can be clarified. If the authors can do that successfully, this could be a huge improvement. Also, the paper tries to do a lot of things in just 9 pages. There’s the desiderata definitions and start off sounding like it’s a conceptual/theoretical paper. Then it introduces and experiments with four different variations of CST. ", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality:\nThis paper is original in introducing and testing out the new concept of causal self-talk for explainable AI. It gives a pretty thorough coverage (concepts, methods, and experiments) of the new idea and offers several variations. The paper also defines concepts good to optimize in explainable AI.\n\nQuality:\nThis paper has mixed quality. Some sections look underdeveloped. Section 2, for instance, seems pretty important considering their claimed contribution but looks not fully fleshed out. I can tell the authors did a lot of work for this paper and perhaps there’s much valuable insights here but it’s lost in the presentation and explanation.\n\nClarity:\nThe paper is well written and clear in some sections (lit review and historical overview) but not others (methods and experiment). The writing is somewhat experimental/creative(?) in some parts. Section 4 in particular, is confusing. Maybe Figure 1 can be improved to better relay the idea of time-step inputs? \n\n\nSignificance:\nTheir newly introduced concept is interesting and paves a path forward for the XAI community on causal methods. Their limitation of memory states not currently workable for general multi-time steps is a big one. Overall it’s an interesting early idea and implementation that can be built on with future work.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "N8dgj6jjHNL", "reviewer": "Reviewer_g2gH", "summary": "In this paper, the authors define 5 desiderata for AI explainability: grounding, flexibility, minimal-interference, scalability, and faithfulness. They show how these desiderata are satisfied by a system that learns an internal model via \"causal self-talk\" (a data augmentation strategy in which externally grounded information is decoded and then rerouted back into the internal state). Intuitively, the decoded information functions as an explanation for why the system behaved in a particular way. To enforce \"causal faithfulness\" (the decoded information exerts a causal influence on the system's output), the authors train the decoder to communicate information to earlier versions of the same agent. This creates the information asymmetry necessary for the agent to use the decoder output. The authors explore 3 different objective functions for training the decoder (based on reinforcement learning, memory reconstruction, and policy distillation). They find that the reinforcement learning variant performed best in terms of causal faithfulness and \"semantic control\" (using decoded information to obtain reward).", "questions": "If the authors can address the two main weaknesses raised above (somewhat contrived experiments and lack of comparison to other approaches) then I will be happy to increase my rating.\n\nUPDATE: the authors have addressed these concerns. Although nothing really substantive has changed in the paper, I understand their reasoning as well as the page constraints.", "limitations": "I don't see any major negative societal impact of this work. I thought the authors did a good drop addressing limitations and future directions.", "rating": 9, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The work is original, as far as I can tell (I am not an expert in this area).\n- The paper is clearly written and logically argued.\n- The ideas are interesting.\n- The experiments demonstrate the efficacy of the proposed method.\n- Given that explainability is an important real-world problem, the results are potentially significant.\n\nWeaknesses:\n- I thought the experiments were useful for demonstrating the efficacy, but they also felt rather contrived relative to real-world problems where explainability is needed.\n- Only variations of the proposed system were studied. This has the advantage of provided well-controlled comparisons, but it doesn't establish to what extent the system outperforms previously proposed systems. \n- This is perhaps less important, but I would have thought that human judgment of explanatory adequacy is important for evaluating real-world explanation systems. This matters at the end of the day, right?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GhUF4ugzR1g", "reviewer": "Reviewer_1Udf", "summary": "The authors delineate a set of desiderata for interpretabiliy of RL systems and then build a causally inspired interpretable solution in which the agent generate a textual output to describe its internal state and utilizes it as a form of memory to update its internal state during the course of a trajectory rollout. The causal aspect stems from interventions on the internal state of the agent being set to its initialization value. The authors evaluate their method on their own benchmark.", "questions": "1. Do you think the intervention could hamper policy learning in the complex environments? It appears that “resetting” the internal state constantly to the initialization value could potentially be harmful. \n2. Do you have any plans for releasing code + environment?\n3. Have you considered how the method might be used for language-conditioned imitation learning? One could feed in instructions instead of z from the previous time step. This could work like a conversational RL agent where it takes an instruction and performs some actions and then returns a string of text for conversation. \n", "limitations": "I believe the paper in its current form needs significant improvement. There seems to be a disconnect between the evaluation and the desiderata which can be bridged through thorough experiments. Additional work is necessary before the work is ready for publication. I wish the authors all the best for this. ", "rating": 5, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Well Written paper: The authors describe not only the intuition behind their work but also how the intuition came into being. This is useful when learning to do research in a new field. \n2. Important problem: The authors have selected an important problem. Explainability of RL agents is key to making RL agents safer and interpretable. \n3. New evaluation environment. The authors have built their own RL environment which (if open-sourced) could yield future results. \n4. Detailed Desiderata: The authors define their own set of desiderata for interpretable RL systems. \n\nWeaknesses:\n1. No pictorial representation of architecture. The authors should provide a figure depicting a rollout of the RL agent in the environment. I can conceive of this by representing an MDP along with a rollout of the policy for a few steps. \n2. Biggest Gains are in the most simple settings: Providing the textual input as a one-hot encoded vector generates most gains. The authors could try to pretrained embeddings for text to improve this.\n3. Highly simplistic evaluation setting: There exist multiple environments with textual annotations, e.g., https://openreview.net/forum?id=eCPCn25gat which could allow for similar experiments. Experiments on them would be useful. My worry is that the approach will not scale.\n4. Not enough explanation delineating the relationship between experiments and desiderata: While defining desiderata is helpful, a callback to the desiderata is necessary in the experiments sections describing how they were achieved. This is missing. \n5. Missing citing work: https://arxiv.org/abs/2010.03110 and https://proceedings.neurips.cc/paper/2021/file/c1722a7941d61aad6e651a35b65a9c3e-Paper.pdf and https://ieeexplore.ieee.org/document/9561439 \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept