Tell me why!—Explanations support learning relational and causal structure
Authors: Andrew Kyle Lampinen
Nicholas Andrew Roy
Ishita Dasgupta
Stephanie C.Y. Chan
Allison Tam
Chen Yan
Adam Santoro
Neil Charles Rabinowitz
Jane X Wang
Felix Hill
Venue: ICLR 2022 Submitted (2022)
Keywords: Explanation
RL
Language
Relations
Causality

Abstract: Explanations play a considerable role in human learning, especially in areas that remain major challenges for AI—forming abstractions, and learning about the relational  and  causal  structure  of  the  world. Here, we explore whether machine learning models might likewise benefit from explanations.  We outline a family of relational tasks that involve selecting an object that is the odd one out in a set (i.e., unique along one of many possible feature dimensions). Odd-one-out tasks require agents to reason over multi-dimensional relationships among a set of objects. We show that agents do not learn these tasks well from reward alone, but achieve >90% performance when they are also trained to generate language explaining object properties or why a choice is correct or incorrect. In further experiments, we show how predicting explanations enables agents to generalize appropriately from ambiguous, causally-confounded training, and even to meta-learn to perform experimental interventions to identify causal structure. We show that explanations help overcome the tendency of agents to fixate on simple features, and explore which aspects of explanations make them most beneficial. Our results suggest that learning from explanations is a powerful principle that could offer a promising path towards training more robust and general machine learning systems.

Under review as a conference paper at ICLR 2022
TELL ME WHY!-EXPLANATION SUPPORT LEARNING OF RELATIONAL AND CAUSAL STRUCTURE
Anonymous authors Paper under double-blind review
ABSTRACT
Explanations play a considerable role in human learning, especially in areas that remain major challenges for Al-forming abstractions, and learning about the re- lational and causal structure of the world. Here, we explore whether machine learning models might likewise benefit from explanations. We outline family of relational tasks that involve selecting an object that is the odd one out in a set (i.c.. unique along one of many possible feature dimensions). Odd-one-out tasks re- quire agents to reason over multi dinliiddmmenionnll relationships among a set of objects. We show that agents do not learn these tasks well from reward alone, but achieve > 90% performance when they are also trained to generate language explaining object properties or why choice is correct or incorrect. In further experiments, we show how predicting explanations enables agents to generalize appropriately from ambiguous, sausally-confounnn experimental interventions to identify causal structure. We show that explanations help overcome the tendency of agents to fixate on simple features, and explore which aspects of explanations make them most beneficial. Our results suggest that learning from explanations is a powerful principle that could offer a promis- ing path towards training more robust and general machine learning systems.
training, and even to meta-leamn to perform
Explanations- that provides explicit information about the abstract, causal structure of the world are central to human learning (Keil al. 2000 Lombrozo 2006). Explanations help solve the credit assignment problem, because they link a concrete situation to generalizable abstractions that can be used in the future (Lombrozo) 2006 Lombrozo and Carey] (006). Thus explanations allow us to learn efficiently. from otherwise underspecified examples (Ahn et al. 1992). Human ex- planations selectively highlight generalizable causal factors and thereby improve our causal under- standing (Lombrozo and Carey 2006). Similarly, they help us to make comparisons and master re- lational and analogical reasoning (Gentner and Christie 2008 Lupyan 2008 Edwards et al.. 2019) Even explaining to ourselves, without feedback, can improve our ability to generalize (Chi et al.)
1994 Rittle- JohnsonJ 2006 Williams and Lombrozo, 2010), potentially because explanations ab- stract knowledge, thus making it easter recall and generalize (ef. Dasgupta and Gershman 2021
causality, relations and analogies, and generalization -are are often cited areas where deep learning has not yet achieved human-level performance (e.g. FFdor and Pylyshyn 1988 Lake et al. [017 [Parl]] 2018; Marcus 2020 Ichien et al. 2021 Holyoak and Lu 2021 Puebla and Bowers 2021: Geirhos al. (020). Thus, there has been increasing interest in using explanations as learning signal (c.g. Ross et al. 2017, Mu et al. 2020 Camburu et al. 2018 Schramowski et al. 2020 see related work). That is, rather than seeking explanations only post- hoc, in an attempt to help humans understand the system (e.g. Chen et al. 2018 Topin and Veloso 2019 Xie et al. 2020), these works use explanations to help the system understand the task (cf. Santoro et al.] (022)) However, most research on learning from explanations has been within super- vised learning. There have been a few explorations within reinforcement learning (RI:; Guan al.) 1 Tulli et al. 2020), but these have not focused relational causal reasoning. Yet explana- tions may be particularly useful in these contexts, to augment the sparse learning signals of rewards. Furthermore, explanations help humans to learn causal structure, and RL agents learn causal structure (e.g. DDaguptt et al., 2019), while supervised learners cannot (Pearl 2018). We therefore explore the benefits of language explanations for RL tasks involving relational and causal structure.
These ciilittes-abstraccee
We define a language explanation to be a string that indicates relationships between a situation, the agent's behavior, and abstract task structure. Thus, explanations can highlight aspects of

---

Under review as a conference paper at ICLR 2022
situation that are generalizable other instances of that task, by elevating task-relevant features above idiosyncratic or task-irreles -ireleaaee ones. We show that language explanations improve agent learning and generalization, and explore how different aspects of explanations affect their benefits.
We first outline a set of challenging relational tasks involving from set that is the odd-one-out along some dimension. These tasks are difficult to learn, because agents need to consider the relationships among all the present objects along various dimensions, rather than simply the properties of a single object or even pairs of objects. We show that these tasks are difficult for agents to learn from reward alone in multiple environments and learning paradigms
the object
However, learning and generalization improve substantially when agents are also trained generate language explanations- even without prior knowledge of language and without telling agents how to use the explanations. Explanations highlight abstract task structure, and thus outperform task- agnostic auxiliary losses (even ones that offer more supervision). Explanations allow agents to learn more effectively, without fixating on easy-but-inan butiiiuuuee "shorteut" features (Geirhos et al.) 2020 Hermann and Lampinen 2020). Explanations allow agents to disentangle confounded features and generalize appropriately cutoof-distribution to deconfounded evaluation, and to meta-learn to perform interventions to identify causal structure. We then explore different how different aspects of explanations contribute to their benefits.
Our results suggest that generating explanations could be powerful tool for learning challenging RL tasks. Language explanations may be simpler for humans to produce than other forms of super- vision (e.g. Cabi al., 2019 Guan et al. 2021), and could selectively identify the key generalizable features of a situation. Thus, training agents to generate explanations might provide a path towards improving learning and generalization. We highlight the following as our main contributions:
1. The family of cognitively rellevnnt odd-one-out tasks ($I) provide challenging measures of agents' ability to extract abstract relations, across various settings and paradigms.
2. Generating explanations ( substantially improves RL agent learning on these tasks
(3 (33n without prior knowledge of language and without telling agents how to use explanations. Explanations selectively highlight the generalizable structure of particular task, yielding better performance than less-selective unsupervised auxiliary losses.
3. Explanations can disentangle confounded features from ambiguous training (33.2)
4. Explanations help agents learn perform experiments identify causal structure $3.3)
5. Explanations help agents move beyond simple biases. Explanations that respond to agent behavior are best, and outputting explanations is better than receiving them as input 3.4
THE ODD-ONE-OU OUT TASKS
Properties explanation: This is large, striped. parple pentagon.
Properties explanation: This is large, striped, blue rectangle.
Reward explanation: Correct, it is uniquely green.
Reward explumation: Incorrect, others are red, solid, or circks.
(a) Example 1.
Example 2.
Figure 1: Conceptual illustrations of two possible odd-one-out tasks, and corresponding possible explanations. This figure depicts odd-one-out tasks with feature dimensions of color, texture, shape, and size, and the two types of explanations we consider. Property explanations identify relevant ob- ject features, while reward explanations specify which feature(s) make the choice correct or incor- rect. (ap The second object is the odd one out, because it is unique color. (b) The first object is the odd one out, because it is uniquely large. Explanations of incorrect choices identify all features.
We outline a challenging family of fundamentally of objects, i.c. the one that somehow unique (Fig, []]. Odd-one-out tasks have been used exten- sively in cognitive science (e.g. |Stephens and Navarro 2008 Crutch et al.) 2009), and proposed in perceptual settings robotics (Sinapov and Stoytchev 2010). These tasks are challenging, because they involve both relational reasoning (same vs. different) and abstraction (identifying uniqueness requires reasoning over all objects, and all dimensions along which objects may be related). Further- more, these tasks permit informative explanations, of relevant dimensions, properties, and relations.
tasks: finding the odd one out in set
2

---

Under review as a conference paper at ICLR 2022
Investigating these challenging and abstract- yet pellainable-relttiin tasks is particularly inter- esting. because relational reasoning and abstraction are critical human abilities (Gentner 20033] Penn
iet al.
2008), but the capacity of deep learning to acquire these skills is disputed (c.g. (aantoro al.
2017
2018 Geiger et al., 2020; Ichien et al. 2021; Puebla and Bowers, 2021). However, expla-
nations important to human relational learning (Gentner and Christie 2008 Lupyan 2008 Ed-
wards et al.
2019), suggesting that explanations might similarly help machines acquire these skills.
In Fig. we conceptually illustrate some odd-one-out tasks. In Fig. Tajthe second object is a uniquely green, while the rest are purple. We thus denote color as the relevant dimension in this episode. Along the other, irrelevant dimensions- shape,, texture, and aize-attributes appear in pairs. For example, there are two pentagons and two triangles. These pairs force the agent to consider all the objects. If the agent considered only the first three objects it would be unable to tell whether the first object was the odd one out (uniquely large), the second (uniquely green), or the third (uniquely a triangle or uniquely solid textured). Thus, the agent must consider all objects to identify the correct dimension and the correct unique feature. This makes the relational reasoning particularly challenging, since the agent must consider many possible relationships. The agent is rewarded for selecting the odd-one-out, either by picking it up, or naming it, depending on the task instantiation.
We emphasize that in principle these tasks can be learned from reward alone- alone-language is not nec- essary for performing them, we evaluate without language. Nevertheless, we find that in practice language explanations are critical for learning these tasks in our settings. We consider two types of explanations: reward explanations and property explanations (see Fig. for examples). Reward ex- planations are produced after the agent chooses, and identify the feature(s) that make the choice cor- rect incorrect. Property explanations are produced before the agent chooses, and explain the iden- tity of the object the agent is facing by specifying its task-relevant properties. Both types satisfy our above criterion for explanations: they link the situation and the agent's ' behavior to the task structure.
1.1 ENVIRONMENTS
Odd-one-out tasks can be instantiated in various settings, from games to language or images, and can incorporate various latent structures (e.g. meta meta-learning) We instantiate these tasks in 2D and 3D RL. environments (Fig. 3a). In 2D, the agent has simple directional movement actions, while in 3D it can move, look around, and grasp nearby objects at which it is looking. In both environments we place an agent in room containing four objects, which vary along feature dimensions of color, texture, position, and either shape (2D) or size (3D). In each episode, one object will be unique along one dimension. The 3D environment compounds the difficulty of the odd-one-out tasks, because the agent's ' limited view often forces it to compare objects in memory. See Appx. C.2 for full details.
2 METHOD: GENERATING EXPLANATION AS AUXILIARY TRAINING
We focus on language explanations provided during training. We synthetically generate the explanations online, conditional on agent behavior. However, explanations could be produced by humans, e.g. as annotations of past trajectories (cf. [Ross et al. 2017]. We train the agent to generate explanations as an aux- iliary signal to shape its representations (Fig. 2), as opposed to providing explanations as direct inputs (which is less effective; Appx. A.3 our approach thus does not require explanations at test time Note that we do not directly supervise behaviour through explanations, nor tell the agent how to use them. The agent simply predicts explanations as an auxiliary output.
Reconsinuct
Genernte Explanation 2202s
Image
MLP
MLP
ResNer
LSTM
Heads
GTTXL
Memory
ResNet
Encoders
Image
Figure 2: RI. agent with auxil-
iary prediction of explanations.
We train agents using the IMPALA (Espeholt et al. 2018) framework. Our agent (Fig. 2 consists of a visual encoder, a memory. and output heads. The encoder is CNN or ResNet (task desk-deeenddnt).. The agent remembers using a 4-layer Gated TransformerXL Memory (Parisotto et al., 2020)), with the visual encoder output and previous reward as inputs. The output of the memory is input to the heads. The policy and value heads are MLPs, trained with V-trace. Another head reconstructs the input images, to learn better representations (though this is not necessary; Appx. Finally, the explanation head is a single-layer LSTM, which generates language explanations. We train the agent to predict these explanations using a summed softmax cross-entropy loss. See Appx. C.1 for details.
3

---

Under review as a conference paper at ICLR 2022
3 EXPERIMENTS
3.1 PERCEPTUAL ODD-ONE-OUT TASKS IN 2D AND 3D RL ENVIRONMENTS
100
Accuracy 199)
Accuracc (96)
50
50
25
Chance
25
Explarations explanations
Chance
0
Training steps
Training steps
(a) 2D & 3D environments.
(b) 2D results.
(c) 3D results.
Figure 3: Explanations help agents learn the perceptual odd-one-out tasks in both RL environments. @ Our environments in 2D (top) and 3D (bottom). In 2D, the agent is the white square, while in 3D it has a first-person view. The objects appear in varying positions, colors, textures, and shapes (2D) sizes (3D). (b) 2D results. Agents trained with explanations achieve high performance; agents trained without do not. (c) 3D results. Only agents trained with explanations learn the tasks. (Training steps denotes actor/environment steps, number of parameter updates 10 smaller. seeds per condition in 2D, 3 per in 3D, lines=means, dark region=11SD, light region=range.)
Explanations help agents to learn perceptual odd-one-out tasks. We first evaluate the benefit of explanations for learning the odd-one-out tasks, by comparing agents trained without explanations both 2D and 3D environments, agents
agents trained with property and reward explanations. trained with explanations learn to solve the tasks over 90% of the time (Figs. 3b Agents trained without explanations perform worse; in the easier 2D environment they exhibit partial learning (see 3.4). while in the more challenging 3D environment they barely perform above chance. All agents in were trained with an auxiliary unsupervised reconstruction loss. However, agents trained without reconstruction but with explanations perform well (Appx. A.7), while agents trained with reconstruc- tion but without explanations do not. Because explanations highlight abstract task structure, they outperform task-agnostie unsupervised objectives, even ones that provide strictly more supervision.
3.2 EXPLANATIONS CAN DECONFOUND
Train (confounded):
100
Chosen by color
Evalustion choice matches
100
Choson by shape
training explonattons ont
Evaluation behavior
75
Chosen by textlure
75
50
50
Evaluation (deconfounded)::
Chance
Explain shape
Explain texture
Training
168
Training
le8
(a) Training & evaluation setup.
(b) Bias without explanations.
(c) Choices with explanations.
Figure 4: Explanations can deconfound perfectly correlated features. (ab Schematic depictions and environment screenshots from train and test. The agent is trained in confounded settings, where the target object is unique in color, shape, and texture. The agent is tested in deconfounded settings, where one object unique along each dimension (and an additional distractor object has no unique attributes). (b) When trained without explanations, the agent is biased towards using color (the simplest feature) in evaluation. (c) However, the agent is trained with explanations that target any particular feature, the agent tends to use that feature to choose in the deconfounded evaluation. (3 seeds per condition, chance random choice among valid objects.)
For humans, explanations help identify which specific aspects a situation are generalizable (Lom- brozo and Carey] 2006). Could explanations also help RL agents disentangle confounded features, and generalize to -ut-of-distribution tests? We explore this with a different training and testing setup (Fig. 4ap. In training, object is the odd-one-out along three feature dimensions (color, shape, and
4

---

Under review as a conference paper at ICLR 2022
texture). Thus, any all of these features could be used solve the task the dimensions per- fectly confounded. In test, however, the features are deconfounded: there is a different odd-one-out along each dimension. We explore the effect of explanations that consistently refer single fea- ture dimension (without mentioning others) on the agent's 's behavior in deconfounded evaluation. We train agents in four conditions: no explanations, color-only explanations, shape-only explanations, or texture-only explanations. Although color, shape, and texture are confounded within each episode, their values appear in different combinations across episodes-e.g. all triangles are purple in Fig. Hal but in another episode they may all be red. Thus, siggledimeesion explanations can potentially draw the agent's attention to particular dimension, and thereby disentangle these features, even though the explanations do not alter the relationship between these dimensions and the reward signal.
We found that agents trained without explanations were biased towards using color (the simplest feature) in the deconfounded evaluation (Fig. 4b). However, the agents trained with explanations generalized in accordance with the dimension that they were trained to explain > 85% of the time (Fig. Ac). even though there were no direct cues linking the reward to that dimension over the others. In this setting, shaping an agent's ' internal representations through explanations draws its attention to the desired dimension, and allows > 85% out-of-distributie generalization along that dimension.
3.3 EXPLANATIONS HELP AGENTS META-LEARN TO EXPERIMENT TO IDENTIFY CAUSAL STRUCTURE
Clock
100
Final choice accuracy (%6)
texualon alags.
Final accuracy
75
75
Explanations No expla na tions
Trandeen
50
50
Ind
Trandtern SucRart
25
25
Chance
Thai
8.00 0.25
0,50 0.75
1.00
1.25
1.50
8.00
0.25
0.50 0.75
Training steps
1e9
Training steps
1c9
Trundbern odar!
Traesfora cake!
Thal
Choose!
(a) A single intervention trial (top). and multi trial structure (bottom).
(b) Easy level structure (bot- tom) and results (top).
(c) Hard level structure (bot- tom) and results (top).
Figure 5: Explanations allow agents to meta-learn to perform experiments. (a) Each episode consists of 3 trials where the agent gets to experiment with a magic wand in order to discover which feature dimension is relevant, followed by a final trial where must choose the unique object along that dimension. In this case the relevant dimension is color. In the first trials the agent transforms the shape and texture of the objects, but is not rewarded for picking them up. In the third trial, transforms the color and is rewarded for picking the object up. The agent can then infer that it should choose the differeet-colored object in the final trial. (b) In some episodes, the experiments are easy, because all the object attributes are the same, and the agent only needs to transform an object and select that object. Agents trained with explanations learn these tasks, while agents trained without explanations do not. (c) other episodes, the experiments are harder, because the object attributes are all paired- the agent must transform one object, and then pick up another which has been made unique. With explanations, agents learn these difficult levels as well. (4 seeds per condition.)
Explanations help humans to understand causal structure (Lombrozo) 2006 Lombrozo and Carey 2006). The ability of deep learning to learn causal structure is sometimes questioned (e-g. |earl 2019). but while thcoretical limitations hold for passive learners, RL agents can intervenc and can therefore learn causal structure. Indeed, agents can meta-learn causal reasoning in simple settings (Dasgupta al. 2019). We therefore investigate whether explanations could help agents meta-learn to identify causal structure in more challenging odd-one-out tasks in richer environments.
We consider a meta-learning setting where agents complete episodes with four trials each. On each trial, the agents perform the odd-one-out task. There is only one causally important dimension per episode-rewaa is determined by uniqueness on only one of the feature dimensions (e.g. color).
5

---

Under review as a conference paper at ICLR 2022
The "correct" dimension is not directly observable, the agents must learn to perform experiments on the first three trials to identify the causally relevant dimension, in order to select the correct object early trial correctly,
on fourth test trial (Fig. Sap The agent receives 1 reward for completing but 10 reward for completing the final trial correctly. Thus, the agent is incentivized to experiment and discover the correct dimension in the early trials, in order to gain a large reward in the final trial.
In the first three trials of each episode, we give the agent a magic wand that can perform one causal intervention per trial: changing an object's color, shape, or texture. The agent forced to use the wand to create an odd-one-out, because each trial's initial configuration lacks objects with unique features along each dimension the features are either all the same. or appear in pairs. When the features are all the same, the experiments are relatively easy (Fig. [5b): the agent must simply trans- form an object and then select the same object. When the features are all paired, however, the exper- iments are harder (Fig. 5c): the agent must transform one object, which will change to match other objects and then must select different object that was formerly paired with this one, but is now unique. The final trial is always deconfounded test, where different object is unique along each dimension, and with no access to the magic wand. On all trials, we reward the agent only if it selects an object which is unique along the "correct" dimension. Thus, the agent cannot consistently choose correctly unless it has already experimented with the magic wand to infer the correct dimension.
We again compare agents that receive property and reward explanations to agents without, but in this case the explanations are augmented to identify the correct dimension (e.g. "incorrect, the dimension is shape, and other objects are squares"). We find that agents trained without explanations cannot learn these tasks, while agents trained with explanations achieve high success at both easy (Fig, 5b)) and harder levels (Fig. (c). We find that in these complex, intervention-focused tasks, behaviourally- relevant explanations are necessary (Fig. c). Furthermore, reward explanations are necessary for any learning to occur, but having property explanations in addition is helpful (Fig. 6c
3.4
EXPLORING THE BENEFITS OF EXPLANATION IN MORE DETAIL.
100
100
All
Final choice sccuracy (30
All
Prop.
Properties
75
75
Reward
75
Reward
None
None
Accuracy
50
50
50
25
Prop.
Chance
0
0
8.00
0.75
1.00
1.50
Training steps
168
Training steps
Training steps
(a) Basic 2D.
(b) Basic 3D.
(c) Meta-learning (hard levels).
Figure 6: Different explanation types offer complementary, separable benefits. We compare agents trained with all explanations or none (as above) to those trained with only property explanations (red), or only reward explanations (blue). (a) In the basic 2D tasks, either kind of explanations is sufficient for learning, but having both types together is substantially faster. (b) In the 3D tasks, property explanations result comparable learning, while reward explanations are not as effective, but still better than none. This is likely due to the memory challenge of these masks-property ex- planations can help the agent discover what to encode to make its choice, while reward explanations cannot. c In the meta-learning to experiment setting, by contrast, only reward explanations (or both together) result in any learning on the hard levels. (2 seeds per sub-type condition.)
In order to better understand the henefits of explanations, we explored our results further in a variety of analyses, ablations, and control and experiments. We outline our findings here; see Appx. A full results. We found that explanations as input are not helpful (Appx. (A.3). and can interfere with the benefits of explanations as targets. We also found that a curriculum of tasks that teaches object properties is not as effective as explanations (Appx. A.6). and that explanations are more beneficial in more complex tasks (Appx. A.5). We highlight three particularly interesting findings here.
Explanations help agents overcome biases toward easy features (Appx. A.I): In 2D, agents without explanations fixate on positions and colors, and learn to solve the task only when those

---

Under review as a conference paper at ICLR 2022
dimensions happen to be relevant. Shape and texture are generally not learned at all. This explains the moderate performance without explanations. With explanations, by contrast, agents learn to solve the tasks with any feature. Similarly, in the confounded features setting color is preferred without explanations, but again with explanations agents can learn to use other features. [Hermann and Lampinen (2020) show similar feature-diffiuule rankings for CNNs, and that CNNs lazily prefer casier features. Similarly. Geirhhs et al. (2020) discuss "shortcut features" that networks prefer, despite the fact that those features do not correctly solve the task. Thus, explanations may help an agent to overcome biases towards easy- bussy-bu--ineee but-inaccurate solutions, to more completely master the task.
Both explanation types provide complementary benefits; their relative value depends on the environment (Fig. 6): In the above experiments we provided agents with both property and reward explanations. Here, we compare to agents trained to generate only a single type of explanations. We found that having both types of explanations is generally better than single type, but the relative benefits of different types depend on the setting. In the 2D environment (Fig. 6a) either type of ex- planations alone results in learning, but both types together result in substantially faster learning. In the 3D setting (Fig. 6bb. we find that property explanations are relatively more beneficial; perhaps because predicting explanations on encountering the objects helps the agent overcome the memory challenges in the 3D environment by helping it to encode the relevant features in an easily decodable way. By contrast, for the meta-learning tasks (Fig. [c). we find that the reward explanations are nec- essary for any learning. The likely reason for this is clear when considering the episode structure- the relevance of a transformation to the final reward much more directly conveyed by the reward explanations than the property ones. However, both types of explanations together are required for complete learning within the training budget we considered. In summary, the relative benefits of the explanation types depend on the demands of the environment, but generally having both types is best.
Behaviourally- and contextually-relevart explanations are most effective (Appx. A.2 : Hu-
man explanations are pragmatic communication--thee depend on context, knowledge, and behav- ior (Van Fraassen 1988). We therefore compared to control explanations that referred to objects in the room, but independent of behaviour (on 10% of steps we randomly chose an explanation that could occur in the current room, regardless of agent actions), and irrelevant explanations (randomly sampled from those possible in any room). We found that behavior-relevant explanations were much more beneficial than behavior-irrelevant ones, and completely irrelevant explanations had no bene- fit. Explanations should respond to the agents actions rather than passively conveying information.
RELATED WORK
Explanations play many roles in human learning, such as enabling efficient learning, even from a single example (Ahn et al. 1992). Explanations play several roles: highlighting both causal factors, and relationships between present situation and broader principles (Lombrozo 2006). They there- fore depend strongly on prior knowledge, and the relationship between explainer, the recipient, and the situation to be explained (Van Fraassen 1988, Cassens et al. 2021). As Wood et al. (1976) say: "one must recognize the relation between means and ends in order to benefit from 'knowledge of re- sults."' " Explanations link specific situation to more general principles that can be used in the future.
Relational and analogical reasoning are often considered crucial to human intelligence (Gentnrr 2003). and possibly absent in other animals (Penn et al. 2008). The relations same and different are central to many of these accounts, but their origins are heavily disputed (Penn et al.) 2008 Katz and Wright, 202T e.g.). But language and culture likely play a critical role in learning these concepts and skills (Gentner and Christie, 2008 Lupyan (r08)~rreational concepts are not simply given in the natural world: they are culturally and linguistically shaped" (Gentner 2003). Thus, explanations may be particularly key to these abilities. This may explain disagreements over neural networks' capacity for relational reasoning (Geiger et al., 2020; Puebla and Bowers 2021
Relations:
Ichien al. 2021). at least without relational inductive biases (Santoro et al. 2017).
Causality: Humans focus on causal structure, even as children (Gopnik et al. 1999 Gopnik and So- bell 2000), and our causal understanding is closely linked to explanations (Lombrozo and Vasilyeva, 2017 ). Human explanations are not just causal, but emphasize important causal factors that are use- Lombroo ful for future prediction and intervention (Lombrozo and Carey 2006). Furthermore, Lombrozo and
Carey (006) emphasize that children accept various explanations, while adults selectively endorse
causally generalizable ones, suggesting that this focus may be at least partly learned.
7

---

Under review as a conference paper at ICLR 2022
Self-explanation: Asking humans to produce explanations for themselves, without providing feed- back, can improve generalization (e.g. Chi et al. 1994. Rittle-Johnson 2006; Williams and Lom- brozo 2010). Furthermore, Nam and McClelland (202T) find that the ability to produce explanations is strongly related to the ability to learn a generalizable problem-solving strategy involving rela- tional reasoning. Furthermore, ing these abilities. Are the ability to generate explanations and generalize (meta-)learned together?
eeppaiiilnnessllllllllllll in formal mathematics- is related to develop-
4.1 RELATED WORK IN AI
We are certainly not the first to observe that human studies like the above suggest that explanations might help in Al-there has been a variety of prior work on explanation in AI, which review here. We also relate to the broader set of approaches for auxiliary supervision that help agents (or models) learn more effective representations for a task. Explanations are particularly targeted form of auxiliary supervision that focuses on the causally-relevant, generalizable elements of a situation.
Language as representation, or to shape representations? Andreas et al. 2018 used language as a latent bottleneck representation in meta-learning, and found benefits. However, Mu et al. (2020) showed that it was better to not bottleneck through language, but merely use descriptions to shape latent representations in supervised classification tasks. This is closely related to our approach, but we evaluate varied explanations more challenging tasks, across a broader range of settings.
Natural Language Processing: Explanations fit naturally into NLP tasks, and Hase and Bansal (202I) highlight the many ways that explanations could enter in NLP tasks, e.g. as targets, inputs, or priors. Surprisingly, they find no improvement from using explanations survey. However, they show some positive effects of explanation retrieval during both training and test, including improved performance on relational tasks and better disentangling of causal factors.
targets in their
Feature explanations as learning tools: Some prior work has refined models using imput atten- tion or gradients targets for explanatory feedback (e.g. from humans). Ross et al. (2017) show that penalizing gradients to irrelevant features can improve generalization on a variety of image and language tasks. [Lertvitayakumjorn and Toni (202T) survey works on tuning NLP models using planatory feedback on features, word-level attention, etc. (Schramowski et al. (020)) highlight an in- triguing interactive-leanning caweninn--foo--eeddbree can be used for similar counter- exxmpll---- or gradient- based training. Stammer et al. (2021) extend this approach in neurosymbolic models intervene on ambolically-cooveyeee semantics rather than purely visual features. In RI., however, applications of feature explanations have been more limited, although Guan et al. (20T)) used human annotations of relevant visual features (together with binary feedback) to generate augmentations that varied the task-irrelevant features, and showed benefits over other feature-based explanation techniques or standard augmentations in video game playing.
setting where an expert in the loop gives feedback which
Language in RL: Language is used broadly in RL, whether as instructions (e.g. Hermann et al.) 2017 Kaplan et al. 2017). to target exploration (Goyal et al. 2019), or as an abstraction to structure hierarchical policies (Jiang et al., 2019). Luketina et al. (2019) review wide variety of recent uses of language in RL, and argue for further research. However, they do not even mention explanations. Tulli et al. 2020) consider using natural language explanations of actions in RI. However, they only evaluate on a simple, symbolic MDP, and they do not observe benefits, perhaps because the explanations they use do not relate to the abstract structure of the task.
Auxiliary tasks: Predicting explanations is part of the general paradigm of shaping agent represen- tations with auxiliary signals (e.g. Jaderberg et al. 20166) However, explanations are fundamentally unsuperviee objectives are task-independent by definition, while explanations selectively emphasize the causally relevant features of a situation, and the rela- tionship to general task principles (Lombrozo and Carey] 2006 Lombrozo 2006). Some supervised auxiliary objectives are more similar to explanations; the boundaries of explanation are blurry. In the Alchemy environment (Wang et al. (021), which involves learning latent causal structure, predicting task-relevant features improves performance. Similarly, [Santoro et al. (2018) show that predicting a "meta-target" an abstract label encoding some task structure improves learning of relational reasoning task. More broadly, supervising task inference can improve meta-learning (Rakelly et al. 2019 Humplik et al. 2019). Since these predictions directly relate to task structure, they are closer to explanations than unsupervised agnostic predictions. However, they do not necessarily ac- tively link the details of the present situation the principles of the task, human explanations do.
different from unsupervised
8

---

Under review as a conference paper at ICLR 2022
5 DISCUSSION
We outlined the odd-one-out tasks, and showed that these tasks are challenging for RL agents to learn from reward alone. However, learning to generate language explanations significantly im- proved performance. Even though our agents lacked prior knowledge of language, learning to gen- erate the language explanations helped them discover the reasoning processes necessary for the task. Explanations helped agents to learn challenging and important abilities, such as relational and causal reasoning. This corresponds with human uses of explanations identify the causally-relevant factors of a task, and thereby allow generalization (Lombrozo and Carey 2006).
Because of this focus on task-specific structure, explanations outperform task-agnostic unsupervised auxiliary objectives. Indeed. we found that explanations helped agents to move beyond a fixation on easy shortcuts that do not fully solve the task, but that models nevertheless prefer (cf. [Hermann) and Lampinen 2020; Geirhhs et al. 2020). Furthermore, our results with confounded dimensions show that explanations can shape how an agent generalizes out-o--distribution (cf. [Ross et al. 2017) Thus, explanations offer promising route to training RL agents that learn and generalize better.
However, some care in crafting explanations is required to achieve their full benefits. Explanations must relate between the context, the agent's s behavior, and the abstract task pnuucture-epplanatitin that ignore behavior are less useful, and those that ignore context are useless. Receiving explanations input was not useful, likely because it is casier for the agent to ignore inputs than auxiliary targets. Explanations outperform unsupervised auxiliary reconstruction. Thus, simply training agents with more information (as with unsupervised objectives) is often not sufficient; explanations must provide relevant and specific learning targets to be most beneficial.
We also acknowledge that the boundaries of explanation are vague. For example, descriptions can- not name every property, so they tend to pragmatically focus on causally-relevant ones, and thus highlight similar features explanations. This is why we refer to task-relevant property descriptions as explanations. Furthermore, we use "'xpllnatton" to refer to cues to relationships between spe- cific situations, behaviors and abstract principles, which may overlap with other forms of auxiliary supervision. While we focused on language explanations, non-language predictions that highlight abstract task features could likely serve the same purpose. Explanations can also vary in abstraction (cf. Fyfe et al., 2014). The boundaries of explanation should be explored further in future work.
We also do not want to imply that explanations are necessary for learning. Most of the tasks we considered could potentially be learned with sufficient data alone, especially if combined with more complicated techniques, for example data augmentation (Raileanu et al.] 22200: Guan et al.) 202T), or auxiliary generative model learning (Gregor et al., 2019). Furthermore, many promising domains for deep learning Suarningsuuch as protein structure prediction (Jumper et al. 202T) are precisely those areas that humans do not understand well, and challenging domains for humans explain.
Indeed, some domains might be irreducibly complex; in these domains forcing a system to strictly follow simple explanations could be detrimental. Our approach does not force the agent use explanations directly, and therefore might be less harmful in such cases than stronger constraints like requiring symbolic representations (Garcez and Lamb 2020). We leave these issues for future work.
In other domains there may exist simple explanations that humans have not yet discovered. This ob- servation motivates future research direction: learning to explain over diverse task distributions, leveraging human explanations in domains we do understand. A curriculum focused on producing explanations could potentially yield substantial benefits. Humans generalize better after explain- ing, even without feedback (Chi et al. 1994 Rittle-Jonnson 2006), and this ability may be learned through education (cf. Nam and McClelland 2021). An agent that similarly learns to produce ex- planations might similarly learn to generalize better even in some domains for which we lack ground truth explanations, and its explanations might help humans interpret its behaviour, and the domains.
Conclusions: We considered a challenging set of relational tasks, and showed that learning to gen- crate explanations helps RL agents to learn these tasks across a variety of settings and paradigms. Explanations help agents move beyond biases favoring easy features, can determine how agents generalize of distrriiibution from ambiguous experiences, and can allow agents to meta-learn to perform experiments identify causal structure. Because many of these abilities are thought to be challenging for current RL agents, we suggest that generating explanations as an auxiliarly learning signal, rather than purely for post-hoc interpretation, may be fruitful direction for further research.

---

Under review as a conference paper at ICLR 2022
REFERENCES
W.-k. Ahn, W. F. Brewer, and R. J. Mooney. Schema acquisition from a single example. Journal of Experimental Psychology: Learning, Memory, and Cognition, 18(2):391, 1992.
Andreas, D. Klein, and S. Levine. Learning with latent language. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2166-2179, 2018.
S. Cabi, S. G. Colmenarejo, A. Novikov, K. Konyushkova, Reed, R. Jeong, K. Zolna, Y. Aytar, D. Budden, M. Vecerik, et al. Scaling data-driven robotics with reward sketching and batch reinforcement learning. arXiv preprint arXiv:1909. 12200, 2019.
O.-. Camburu, T. Rocktäschel, T. Lukasiewicz, and P. Blunsom. e-snli: Natural language inference with natural language explanations. Advances in Neural Information Processing Systems, 31: 9539 9049,,949405 2018.
Cassens, Habenicht, Blohm, R. Wegener, J. Korman, S. Khemlani. G. Gronchi. R. M. Byrne, G. Warren, M. S. Quinn, et al. Explanation in human thinking. In Proceedings of the Annual Meeting of the Cognitive Science Sociery, volume 43, 2021.
Chen, Song, M. Wainwright, and M. Jordan. Learning to explain: An cnermattiontthooreee perspective on model interpretation. In International Conference on Machine Learning, pages 883-892. PMLR, 2018.
M. Chi, De Leeuw M.-H. Chiu, and C. Lavancher. Eliciting self-explanations improves understanding. Cognitive Science, 18(3):439-477 1994. ISSN 0364-0213. doi: https://dii.on org/ 10:10160036--2113044900000 atticle/s1110336011333343
URL httttt:t///ett sci encedi rect. com/ sci ence/
S. J. Crutch, S. Connell, and E. K. Warrington. The different representational frameworks underpin- ning abstract and concrete knowledge: Evidence from odd-one-out judgements. Quarterly Jour- nal of Experimental Psychology, 62(7):1377-1390, 2009.
I. Dasgupta and S. J. Gershman. Memory as computational resource. Trends in Cognitive Sciences, 2021.
I. Dasgupta, J. Wang. S. Chiappa, J. Mitrovic, P. Ortega, D. Raposo, E. Hughes, P. Battaglia, M. Botvinick, and Z. Kurth-Nelson. Causal reasoning from meta-reinforcement learning, arXiv preprint ar 1901.000 2019.
B. J. Edwards, J. J. Williams, D. Gentner, and Lombrozo. Explanation recruits comparison in a category-learning task. Cognition, 185:21-38, 2019.
L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference Machine Learning, pages 1407-1416. PMLR, 2018.
J. A. Fodor and Z. W. Pylyshyn. Connectionism and cognitive architecture: A critical analysis. Cognition, 28( 11-2):3-71, 1988.
E. R. Fyfe, N. M. McNeil, J. Y. Son, and R. L. Goldstone. Concreteness fading in mathematics and science instruction: A systematic review. Educational psychology review, 26(1):9-25, 2014.
A. d. Garcez and L. C. Lamb, Neurosymbolic ai: the 3rd wave. arXiv preprint arXhv:2012.05876, 2020.
A. Geiger, A. Carstensen, M. C. Frank, and C. Potts. Relational reasoning and generalization using non-symbolic neural networks. arXiv preprint arXiv:2006.079: 2020.
R. Geirhos, J.HH. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665-67) 673, 2020.
D. Gentner. Why we' re so smart. Language in mind: Advances in the study of language and thought, 195235, 2003.
10

---

Under review as a conference paper at ICLR 2022
D. Gentner and S. Christie. Relational language supports relational cognition in humans and apes. Behavioral and Brain Sciences, 31(2):136-133) 2008.
A. Gopnik and D. M. Sobel. Detecting blickets: How young children use information about novel causal powers in categorization and induction. Child development, 1105):1205-1222 2000.
A. Gopnik, A. Meltzoff, and P. K. Kuhl. The scientist the crib: Minds, brains, and how children learn. William Morrow & Co, 1999.
P. Goyal, S. Nickum, and R. J. Mooney. Using natural language for reward shaping in reinforcement learning. arXiv preprint arXiv: 0eti:9903.002,, 2019.
K. Gregor, D. Jimenez Rezende, F. Besse, Y. Wu, H. Merzic, and A. van den Oord. Shaping belief states with generative environment models for rl. In Ad- vances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https: | http::/:s//ppooceiigs..soo 1CC44877554411777777777775510000
cc/pppeer/219//iie//
pdff
I.. Guan, M. Verma, S. Guo, R. Zhang, and S. Kambhampati. Widening the pipeline in human- guided reinforcement learning with explanation and context-aware data augmentation. In Ad- vances in Neural Information Processing Systems, 2021.
Hase and M. Bansal. When models learn from explanations? formal framework for under- standing the roles of explanation data. arXiv preprint arXiv:2102.02201, 2021.
K. L. Hermann and A. K. Lampinen. What shapes feature representations? exploring datasets, architectures, and training. Advances in Neural Information Processing Systems, 2020.
K. M. Hermann, Hill, S. Green, F. Wang, R. Faulkner, H. Soyer, D. Szepesvari, W. M. Czarnecki, M. Jaderberg, D. Teplyashin, al. Grounded language learning in simulated 3d world. arXiv preprint ar drXiv:1706.06530 06551, 2017.
F. Hill, A. Lampinen, R. Schneider, S. Clark, M. Botvinick, J. L. McClelland, and A. Santoro. Environmental drivers of systematicity and generalization in a situated agent. In International Conference on Learning Representations, 2019.
K. J. Holyoak and H. Lu. Emergence of relational reasoning. Current Opinion in Behavioral Sciences, 37:118-1 1-214, 2021.
J. Humplik, A. Galashov, L. Hasenclever, P. A. Ortega, Y. W. Teh, and N. Heess. Meta reinforcement learning task inference. arXiv preprint arXiv: 965.06424, 2019.
N. Ichien, Q. Liu, S. Fu, K. J. Holyoak, A. Yuille, and H. Lu. Visual analogy: Deep learning versus compositional models. In Proceedings of the 30th Annual Meeting of the Cognitive Science Society, 2021.
M. Jaderberg, V. Mnih, M. Czameeki, T. Schaul, Leibo, D. Silver, and Kavukeuoglu. Re- inforcement learning with unsupervised auxiliary tasks. In International Conference Learning Representations, 2016.
Y. Jiang, S. Gu, K. Murphy, and C. Finn. Language as an abstraction for hierarchical deep reinforce- ment learning. arXiv preprint arXiv: 1906. 07340, 2019.
Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. Židek, A. Potapenko, al. Highly accurate protein structure prediction with al- phafold. Nature, 596(7873):583-58) 2021.
R. Kaplan, C. Sauer, and A. Sosa. Beating atari with natural language guided reinforcement learning. arXiv preprint arXiv: 1704.05339 2017.
Katz and A. A. Wright. Issues in the comparative cognition of same/different abstract-concept learning, Current Opinion in Behavioral Sciences, 37:29-34, 2021.
F. C. Keil, R. A. Wilson, and R. A. Wilson. Explanation and cognition. MIT press, 2000.

---

Under review as a conference paper at ICLR 2022
D. P. Kingma and J. Ba. arXiv:1412.6980, 2014.
Adam: A method for stochastic optimization. arXiv preprint
B. M. Lake, T. D. Ullman, J. B. Tenenbaum, and S. J. Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
Lertvitaayakujjrn and Toni. Explanation-bas human debugging of nlp models: A survey. arXiv preprint arXiv:2104. 10.004/5135 15135, 2021.
T. Lombrozo. The structure and function of explanations. Trends in cognitive sciences, 10(10):464- 470, 2006.
T. Lombrozo and S. Carey, Functional explanation and the function of explanation. Cognition, 99 (2):167-204, 2006.
Lombrozo and Vasilyeva. Causal explanation. Oxford handbook of causal reasoning, pages 415-432, 2017.
J. Luketina, N. Nardelli, G. Farquhar, J. Foerster, J. Andreas, E. Grefenstette, S. Whiteson, and T. Rocktäschel. A survey of reinforcement learning informed by natural language. arXiv preprint urXiv:I906.000 03926, 2019.
G. Lupyan. Taking symbols for granted? is the discontinuity between human and nonhuman minds the product of external symbol systems? Behavioral and Brain Sciences, 3 31(2):140-14) 2008.
G. Marcus. The next decade in ai: four steps towards robust artificial intelligence. arXiv preprint arXiv:2002.06177, 2020.
J. Mu, P. Liang, and N. Goodman. Shaping visual representations with language for few-shot classification. In Proceedings of the 58th Anmual Meeting of the Association for Computa- tional Linguistics, pages 4823-4830, Online, July 2020. Association for Computational Linguis- tics. doi: 10.186 0e.11653/11/2220..m wal.2....l-maii.... mai.. 436. URL https: http:////aaalatttoooooooo/o///// acl-ma iimain. 436
A. J. H. Nam and J. L. McClelland. What underlies rapid learning and systematic generalization in humans. arXiv preprint arXiv:2102.02926, 2021.
E. Parisotto, F. Song, J. Rae, R. Pascanu, C. Gulcehre, S. Jayakumar, M. Jaderberg, R. L. Kaufman, A. Clark, S. Noury, al. Stabilizing transformers for reinforcement learning. In International Conference on Machine Learning, pages 7487-7498. PMLR, 2020.
J. Pearl. Theoretical impediments to machine learning with seven sparks from the causal revolution. arXiv preprint (rrXiv::81) .04016, 2018
J. Pearl. The seven tools of causal inference, with reflections on machine learning. Communications of the ACM, 62(3):54 60, 2019.
D. C. Penn, K. Holyoak, and D. J. Povinelli. Darwin's 's mistake: Explaining the discontinuity between human and nonhuman minds. Behavioral and Brain Sciences, (1(2):109-13) 130, 2008.
G. Puebla and J. S. Bowers. Can deep convolutional neural networks learn same-different relations? bioRxiv preprint, 2021.
R. Raileamu, M. Goldstein, D. Yarats, I. Kostrikov, and R. Fergus. Automatic data augmentation for generalization in deep reinforcement learning. arXiv preprint arXiv:2006. /2862, 2020.
K. Rakelly, A. Zhou, Finn, S. Levine, and D. Quillen. Efficient off off-policy reta-reinforremment learning via probabilistic context variables. In K. Chaudhuri and R. Salakhutdinov, editors, Pro- ceedings of the 36th International Conference on Machine Learning, volume 97 of Proceed- ings of Machine Learning Research, pages 5331-5340. PMLR, 09-15 Jun 2019, URL. https: /pproceedings.. . mlr. www.mmwwt...............m..
raw.llyllallla.. html
B. Rittle-Johnson. Promoting transfer: Effects self-explanation and direct instruction. Child development, 77(1):1-15, 2006.
12

---

Under review as a conference paper at ICLR 2022
A. S. Ross, M. C. Hughes, and F. Doshi-Velez. Right for the right reasons: Training differentiable models by constraining their explanations. In Proceedings of the Twenty- International Joint Conference Artificial Intelligence, IJCAI-17, pages 2662 2670, 2017. doi: 10.24963/ijcai.
2017/371. URL holthhtps:///i
org/10.249633 24963/ 1a0////////// jeai 20127377
A. Santoro, D. Raposo, D. G. Barrett, M. Malinowski, R. Pascanu, P. Battaglia, and T. Lillicrap. A simple neural network module for relational reasoning. arXiv preprint (rXiv:1706001227 2017.
A. Santoro, F. Hill, Barrett, A. Morcos, and Lillicrap. Measuring abstract reasoning in neural networks. In International Conference on Machine Learning, pages 4477-4486, 2018.
A. Santoro, A. Lampinen, K. Mathewson, T. Lillicrap, and D. Raposo. Symbolic behaviour in artificial intelligence. arXiv preprint arXiv:2102.03406, 2021.
P. Schramowski, W. Stammer, S. Teso, A. Brugger, F. Herbert, X. Shao, H.-G. Luigs, A.-K. Mahlein, and K. Kersting. Making deep neural networks right for the right scientific reasons by interacting with their explanations. Nature Machine Intelligence, 2(8) 2776:476,48, 2020.
J. Sinapov and A. Stoytchev. The odd one out task: Toward an intelligence test for robots. In 2010 IEEE 9th International Conference on Development and Learning, pages 126-131. IEFF, 2010.
W. Stammer, P. Schramowski, and K. Kersting. Right for the right concept: Revising neuro-symbolic concepts by interacting with their explanations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3619-3629, 2021.
R. G. Stephens and D. J. Navarro. One of these greebles is not like the others: Semi-supervised models for similarity structures. Proceedings of the 30th Annual Meeting of the Cognitive Science Society, 2008.
N. Topin and M. Veloso. Generation of policy-level explanations for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 2514-2521, 2019.
S. Tulli, S. Wallkötter, A. Paiva, F. S. Melo, and M. Chetouani. Learning from explanations and demonstrations: A pilot study. In 2nd Workshop on Interactive Natural Language Technology for Explainable Artificial Intelligence, pages 61-66, 2020.
B. Van Fraassen. The pragmatic theory of explanation. Theories of Explanation, 8: 135-1555, 1988.
Wang, M. King, Porcel, Z. Kurth-Nelson, T. Zhu, C. Deck, P. Choy, M. Cassin, M. Reynolds, F. Song, al. Alchemy: A structured task distribution for meta-reinfforeement learning. arXiv preprint arXiv:2102.02920 2021.
J. J. Williams and T. Lombrozo. from category learning. Cognitive science, 34(5):776-806, 2010.
The role of explanation in discovery and generalization: Evidence
D. Wood, S. Bruner, and Ross. The role of tutoring in problem solving. Journal of child psychology and psychiatry, 17(2):89-100, 1976.
N. Xie, G. Ras, M. van Gerven, and D. Doran. Explainable deep learning: A field guide for the uninitiated, arXiv preprint 200::2004.14555, 2020.
13

---

Under review as a conference paper at ICLR 2022
In Appendix show additional experiments and analyses. In Appendix B we report numerical values for our main experimental comparisons. In Appendix we report details of the environments, agents, and training.
A ABLATION EXPERIMENTS & FURTHER ANALYSES
In this section, we perform a variety of control, ablation, and auxiliary experiments that identify which attributes of explanations are useful in different settings. We perform most of these experi- ments in the 2D RL setting because of the efficiency of running and training agents in this environ- ment.
A.1 AGENTS TRAINED WITHOUT EXPLANATIONS FIXATE ON THE EASIEST FEATURE DIMENSIONS
100
100
100
100
50
50
50
25
No explination
Trapingsoss
Traingg
Trainig stops
Trainin staps
(a) Position.
Color.
(c) Shape.
(d) Texture.
Figure 7: In the 2D setting, agents trained with explanations learn all dimensions, but agents trained without explanations learn to fully solve the tasks only if the relevant dimension position (the easiest dimension), and only partly learn to solve the tasks with color (the next easiest dimension). (5 seeds per condition.)
In the basic 2D odd-one-out tasks, the agent achieves off-chance performance without explanations (while in more complicated settings such as the causal interventions, it cannot learn at all without explanations). In Fig. we show that what the agent is doing is latching to the feature dimen- sion(s) that are most sahent and easiest (Hermann and Lampinen 2020), and only correctly solving episodes involving these features. Specifically, position is the most salient feature and is learned rapidly even without explanations, followed by color which is partially learned without explana- tions. However, shape and texture much more difficult and not learned well without explana- tions. These results concord with the features that Hermann and Lampinen (2020) found were casi- est for CNNs and ResNets learn, suggesting that explanations may help overcome the preference of agents (or other networks) to be "lazy" and prefer "shortcut features"' (Geirhos et al., 2020).
A.2 EXPLANATIONS ARE MOST USEFUL IF THEY ENGAGE WITH THE AGENT'S BEHAVIOR; SHUFFLED EXPLANATIONS ARE USELESS
We next investigate whether explanations need to be relevant to the agent's behavior, or even the situation at all, in order to be useful. To do this, we provide the agent with explanations that either are siquation-ellvvan, but behavior irrelevant, are irrelevant to both behavior and situational context. To produce the iituation-relevan but inrelveatrrelevant explanations, we first construct an episode as before. We then enumerate all the property and reward explanations that it would be possible to receive in that episode, and present randomly selected one to the agent on approximately 10% of steps, regardless of the agent's actions. These explanations do contain information about the objects in the scene, and can therefore potentially still benefit learning, but they do not directly react to the agent's actions.
We also considered context-ireelevant explanations that were randomly sampled from the set of all possible explanations (we chose either a property explanation or a post choice one with 50% probability, and then sampled random set of attributes to fill out the template). This condition is essentially a control for the possibility that predicting structured information even meaningless information unassociated with the task-coull be acting form of regularization.
14

---

Under review as a conference paper at ICLR 2022
accuracy (86)
Final choice accuracy (86)
Final choice accuracy (%s)
75
75
Full
75
Behavior-imelevant context-irelevent None
50
50
50
25
25
23
0
8.00
0.25
0.50
8.00
1.25
Training stops
Training steps
Training stops
(a) Basic 2D.
(b) Meta-learning (easy levels).
(c) Meta-learning (hard levels).
Figure 8: Explanations must be behaviourally (as well as contextually) relevant to be useful in chal- lenging settings; explanations that are contextually-irelevaa are useless in every experiment. (a) In the basic 2D odd-one-out tasks, behavior-irrelevant explanations eventually result in relatively com- parable performance compared to full explanations, but produce much slower learning. Context ir- relevant explanations are not substantially different than no explanations. (b o both easy and hard
learning-too cermning-tt-xxperringg experiment levels, only agent with full, behavior and context-relevant explanations is able to learn the tasks at all. Thus, more challenging task settings require more specific, behavior- releavnt explanations. (3 seeds for behavior-irrllewalee inrelevant/context autt/contextiimeeeennn irrelevant conditions.)
Our results (Fig. S show that explanations that are relevant both situation and behavior are most useful, situation-relevant but behavior-irrelevant explanations can be better than nothing in some cases, and totally irrelevant explanations are not beneficial all. Specifically, for the basic tasks behavior-irreevaae explanations still result in some learning, but are much slower than full behavior- relevant explanations.
A.3 PROVIDING EXPLANATIONS AS AGENT INPUTS IS NOT BENEFICIAL, AND INTERFERES WITH LEARNING FROM EXPLANATION TARGETS
In the main text, we focused on explanations as targets during training, rather than inputs. That approach heneficial, because it does not require explanations test time, while explanations as input generally does. In Fig. 9 we show that furthermore providing explanations the agent is not beneficial, and is actively detrimental if explanations are also used as targets, presumably because in the latter case the agent can just "pass through" the explanations, without having to learn the task structure. However, it is possible that providing explanations as input on the timestep after the agent predicts them could be useful (as in language model prediction, where each word is input after the model predicts it); we leave evaluating this possibility to future work.
input to
Final choice accuracy (96)
100
75
50
As target (orig.)
25
None (orig.) Inputs targets
Chance
0
Training steps
1e8
Figure 9: Providing explanations as agent inputs is not beneficial (performs better than no explana- tions), and is actively detrimental if explanations are also used as targets. seeds per as-input con- dition, 5 seeds for main conditions.)
15

---

Under review as a conference paper at ICLR 2022
A.4 DIFFERENT KINDS OF EXPLANATIONS HAVE COMPLEMENTARY, SOMETIMES SEPARABLE BENEFITS
We generally provided agents with both property explanations and reward explanations. Is one of these explanations more useful than the other? Are they redundant? To answer these questions, we considered providing the agent with each kind of explanation independently. We generally find that having both types of explanations is best, and the benefits of different types depend on the setting,
In the 2D setting (Fig. bap either type of explanations alone results in learning, but both types together result in substantially faster learning. In the 3D setting (Fig. [bb, we find that property explanations are uniquely beneficial; perhaps because predicting explanations on encountering the objects helps the agent overcome the challenges of generating good representations for its memory.
For the meta-interventions tasks involving experimentations, we find (Fig- 66b that the reward ex- planations are uniquely beneficial, while properties explanations are not useful alone. The likely reason for this is clear when considering the episode structure-the relevance of a transformation to the final reward is much more directly conveyed by the reward explanations than the property ones. However, both types of explanations together are required for complete learning within the learning time we considered.
A.5 THE BENERITS OF EXPLANATIONS DEPEND ON TASK COMPLEXITY
100
Accuracy (%6)
75
Accuraty (%6)
75
50
50
Esplanations No explanations
0
Training steps
108
Training steps
1e8
(a) Position & color only.
(b) Shape & texture only.
The benefits of explanations depend on task difficulty. We train agents in the 2D environ- ment on easier odd-one-out tasks where only two of the four dimensions are ever relevant. (a) When only the easy dimensions of position and color are ever relevant, the agents trained without explana- tions learn just as rapidly as the agents trained with explanations. (b) When the agents are trained on levels where only the harder dimensions of shape and texture are relevant, explanations still accel- erate learning substantially. However, the agents trained without explanations achieve some learn- ing this condition, while they do not achieve any learning on these dimensions in the harder tasks used for the main experiments (see Fig. dc.... (2 seeds per condition.)
Figure 10:
While we generally considered tasks with many feature dimensions that might be relevant, here we show that simpler tasks in which only two dimensions vary do not always require explanations for learning. However, task complexity depends on both the number of possibly relevant dimensions and the base difficulty of those dimensions. In Fig. 10 we show that explanations are not beneficial compared no-explanations when only the easy features of position and color are relevant. Ex- planations are still beneficial when the features are more difficult (shape and texture). But even in this condition, the agent without explanations exhibits some learning, while it does not learn these dimensions at all in the main experiments, where the casier dimensions are also included (see Fig. d). Note also our results on deconfounding- in some cases explanations may help the agent to generalize in desired way even if they are not necessary for learning the training task.
16

---

Under review as a conference paper at ICLR 2022
A.6 LEARNING PROPERTIES THROUGH CURRICULUM RATHER THAN AUXILIARY LOSSES
100
100
Accuracy (96)
Accuracy
50
-50
Explanations
25
No explanations
Chance
Curriculum (shared) Curriallum (separate)
3 Training stops
166
Training stops
166
(a) Property-curricllum tasks.
(b) Odd-one-out tasks.
Figure 11: Agents trained with curriculum of tasks that teach the properties (a) do not learn the odd- one-out tasks (b) any better than agents trained with no explanations. Results are similar whether the agent uses shared policy (light green) for both the currieulum and odd-one-out tasks, separate policies for each (dark green). (2 seeds per condition for curriculum conditions, 5 seeds for main conditions.)
Because predicting property explanations alone can be beneficial, we next consider whether the agent could benefit from learning properties through auxiliary tasks which teach those properties, rather than through explanations. Specifically, we provide the agent with simpler property-learning task in 50% of episodes, where it receives a property like "red" as an input instruction, and has to choose the corresponding object (all objects are different along each feature dimension). These tasks provide a different way to force the agent to learn the properties of the objects. On the odd-one-out tasks, the agent receives the instruction "find the odd out" to distinguish its goal.
Surprisingly, we find that learning these tasks does not substantially accelerate learning the odd- one-out tasks compared to no-explanations and no-property-tee -tasks baseline (Fig. lTb). We initially thought this might be due to interference due to the shared policy being used for different tasks, so we reran the experiment with separate policy heads for the curriculum tasks and odd-one-out tasks, but this did not substantially change results. Auxiliary prediction of explanations may therefore be a more efficient way to encourage learning of task-relevare features, perhaps because actively engages with the agent's behavior in the settings where those dimensions are particularly relevant.
A.7 AUXILIARY UNSUPERVISED LOSSES ARE NEITHER NECESSARY NOR SUFFICIENT; THUS THE BENEFITS OF EXPLANATIONS ARE NOT SIMPLY DUE TO MORE SUPERVISION
In Fig. 12 we show that the auxiliary reconstruction losses are not necessary for learning the odd- one-out tasks. Furthermore, the main text results without explanations show that these losses are not sufficient for learning either. This shows that the benefits of explanations not simply due to having more supervision for the agent, but rather are specific to supervision that highlights the abstract task structure.
17

---

Under review as a conference paper at ICLR 2022
100
Accuracy (%)
75
50
Explanations Exp. -recon
25
No explanations
0
Training steps
Figure 12: Ablating the auxiliary reconstruction losses does not alter the pattern of results in the 2D) environment thus reconstruction is not necessary for learning these tasks. Comparing the results in Fig. 3bj which does include reconstruction losses, shows that reconstruction losses are also not sufficient without explanations. (2 seeds per condition.)
Table I: Numerical results from main experiments/figuee in each domain-mean + standard devi- ation across seeds. Results are average performance (% correct) across evaluations during the last 1% of training.
Experiment
Level
Fig.
Condition
Performance
Explanations
91.3 0.7
Perceptual 2D
3b
No explanations
61.9 2.2
Explanations
92.7 1.4
Perceptual 3D
3c
No explanations
29.5
Chose color
No explanations
55.4
Chose shape
4b
No explanations
2.62
Chose texture
No explanations
1.74
Deconfounding
Chose color
Explain color
9.9
Chose shape
4c
Explain shape
2.9
Chose texture
Explain texture
0.9
Explanations
96.
Easy
5b
No explanations
24.0
Meta-learning
Explanations
2..6
Hard
5c
No explanations
61.9
18

---

Under review as a conference paper at ICLR 2022
B QUANTITATIVE RESULTS
C
METHODS
C.I RL AGENTS & TRAINING
Table 2: Hyperparameters used in main experiments. Where only one value is listed across both columns, it applies to both.
2D
3D
All activation Iis
ReLU
State dimension
512
Mernrry dimension
312
Mcmory layers
Memory num. heads
TiXL extra length
128
Visual encoder
CNN
ResNet
Vis. enc. channels
(16. 32, 32)
Vis. enc. fill, size
(9, 3, 3)
(3, 3. 3)
Vis, enc. filt. stride
(9, 1)
(2,
Vis. enc. num. blocks
NA
(2, 2,
Policy & value nets
MLP with hidden layer with 512 units.
Recunstruction decoder
Architectural transpose of the encoder, with independent weights. I-layer LSTM
Explanation decoder
Explanation LSTM dimension
256
Recon, loss weight
0.
V-trace loss weight
V V-trace bascline weight
Explanation loss weight
3.3 10
main, 0.2 meta-learning.
3.3 10
Entropy weight
10-
1.10
Batch size
24
Training trajectory length
50
Optimizer
Adam Kingma and Ba 2014
LR
10
with explamations, 2 10 without
In Table 2 we list the architectural and hyperparameters used for the main experiments. In most cases the hyperparameters were taken from other sources without tuning for our setup. There are two main exceptions: (1) we chose the explanation weight loss to approximately balance the magnitude of this loss with the magnitudes of the RL losses early in training in each experiment, and (2) we swept the learning rate for the main experiments in 2D and 3D (but used the same settings for follow- up experiments, including meta-learning). In the 2D experiments we found that a similar learning rate was best for both agents trained with and without explanations, but in 3D we found that agents trained without explanations needed a slower learning rate to avoid their performance degrading from chance-level to below chance.
Since we ran the 3D experiments after 2D, we used similar byperparameters, except that we found needed to decrease the learning rate as noted above. However, some hyperparameters do differ across tasks due to specific task features. For example, the visual encoder for the 2D tasks is set to have filter size of 9 because this is the resolution of each square in the grid, and the entropy cost for 2D tasks was chosen from prior work which used a similar grid world action space (Hill let al. 2019), while the 3D cost is lower because of the more complex action space for these tasks (see below). These decisions were shared across experimental conditions, so should not favor one condition over another.
We trained the agents to predict the language explanation using softmax cross-entropy loss over a word-level vocabulary of 1000 tokens (more than were necessary for the limited language we used). The explanation loss was summed across the sequence of tokens.
Explanation prediction loss:
Self-supervised image reconstruction loss:
We trained the agents reconstruct the image pixels (normalized to range [0, 1] on each color channel) with a sigmoid cross-entropy loss. The image reconstruction loss averaged across all pixels and channels. However, we found in follow-up
19

---

Under review as a conference paper at ICLR 2022
experiments that this did not substantially change results in 2D (Appx. A.7 so disabled this loss in 3D.
C.2 RL ENVIRONMENT DETAILS
C.2.1 2D
The tasks were instantiated in 9 9 tile room with an extra I tile wall surrounding all sides, for a total of 11 x 11 tiles. This was upsampled at a resolution of 9 pixels per tile to form 99 x 99 image input to the agent. The agent
placed in the center of the room, and had 9 possible actions, allowing it to move one square in any the possible directions, or to do nothing.
Four objects were placed in the room with the agent. They were chosen that single object was the odd one out, along single dimension, and features appeared two pairs along the other dimensions. The objects varied along the feature dimensions of:
Color: of 19 possible colors (e.g. green lavender).
Shape: one of 11 possible shapes (e.g. triangle tee).
Texture: one of possible textures (e.g. horizontal stripes checkers).
Position: One position types (in corner, against horizontal wall, against vertical wall, or in 3x3 square in the center).
The agent was given 1 188 steps to complete each episode, after which the episode would immediately terminate. The agent had choose an object by walking onto the grid cell containing It would be immediately rewarded 1 if the object was the odd one out, and 0 otherwise. However, the episode would last for additional few steps give the agent time to learn from the reward explanation (if provided); this extra time was provided even if the agent was not trained to predict explanations, in order precisely match the training experience across conditions. No additional reward would be received during this period, but the agent would be asked output the reward explanation at every timestep. This period would last either 16 steps, until the agent touched any object, or until the full reached, whichever came soonest.
episode limit of 128 steps
the agent was trained to predict property explanations, whenever was adjacent to an object would be asked to predict a string of the form:
This is red
sped triangle in-the-corner
The properties always appeared in the order color texture shape position. These sentences were tokenized word level, with the hyphenated phrases treated as single words. Hence, a single token was attached to each possible feature value along each dimension.
the agent was adjacent to multiple objects, which description it received was determined randomly. Once the agent made a choice, property descriptions were disabled.
the agent was asked to predict reward explanations, for the period after receiving the reward (see above), it would be asked to predict a string in one of the following forms:
Correct because it is uniquely horizontal-striper
Incorrect because other objects are red triangles or in-the-corner
Thus, the reward explanations identify all features that contributed to a decision being incorrect.
C.2.2 3D
The agent was placed in room with a randomly placed door and windows (the agent could not interact with these). It had 10 possible actions: moving forward backard, moving left right, looking left or right or up or down, grabbing an object it was facing (if within certain distance), and doing nothing.
20

---

Under review as a conference paper at ICLR 2022
Four objects were placed in the room, with attributes sampled as in the 2D environment from the dimensions:
Color: one of 10 possible colors (e.g. blue or magenta).
Size: of 3 possible sizes (small, medium, large).
Texture: one of possible textures/materials (e.g. metallic or wood-grain).
Position: One of 3 position types (in corner, against wall, or in the center).
The episode lasted for 60 seconds, at 30 FPS; but the agent took an action only once every 4 frames (the action then repeated until the next agent step), the episode lasted for most 450 agent steps. The agent had to use its "grab" action on an object to make a choice; colliding with the objects would simply cause them move. 25 steps were allocated for reward explanations (if any).
the agent was asked to predict property explanations, they were given when the agent was facing an object and close enough grab it.
The property and reward explanations for the 3D environment were analogous to the ones for the 2D environment, except that the reward explanaitons did not have the "or" before the final attribute.
21


Reviews:
[{"review_id": "vqEnBmX2MgP", "reviewer": "Reviewer_DFpa", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is mostly a position paper supported by dedicated experiments in specific setups intended to support the various claims. I was not familiar with sort-one-out identifications, but I have found the arguments of the authors convincing. Similarly the discussion about the related works seems to be quite detailed. Having this in mind, my educated guess about the paper is that it might be interesting to have it presented at ICLR. A weakness I noticed is the very dry description of the experimental setup.On the other side, the rest of the presentation is very clear and easy to follow. I should also notice that the claims are mostly supported by empirical analysis in very specific environments, deciding whether or not this is enough to claim a general validity could be controversial.\n", "summary_of_the_paper": "This is a paper about sort-one-out identification. The goal is to emphasise that standard reinforcement learning is not optimal for this task, while adding (language) explanations to the learning phase might be highly beneficial. Dedicated experiments are developed for 2- and 3-D simulation environments. Each experiment is used to advocate a specific point such as the effect of the explanations on accuracy, robustness wrt correlated features, and other more detailed analyses.", "summary_of_the_review": "A well-written paper advocating the advantages of adding explanations to RL setups wrt to sort-one-out identifications. As a non expert of this field I have found the presentation clear and convincing. The arguments to support the claims are mostly empirical, but restricted to a specific simulation architecture.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "lL7nzhKWo7P", "reviewer": "Reviewer_AvzX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Despite the idea of including textual explanations in the learning pipeline to improve performance is interesting and trending these days in the DL research community, the overall contribution of this paper is very modest.\nThe proposed odd-one-out tasks do not cast a significant new challenge with respect to existing tasks. Authors justify their introduction because they require relational reasoning and abstraction, described as \"identifying uniqueness requires reasoning over all objects, and all dimensions along which objects may be related.\" Yet, existing tasks already cover such requirements, e.g. CLEVR [1]. \nThe paper's central claim -namely that learning to generate explanations is useful to solve relational tasks - is not adequately supported. In this regard, there are two major concerns: \n- What are explanations? To give substance to a claim involving *explanations*, these should be rigorously defined. This definition is missing. Moreover, the proposed *property explanation (PE)* clashes with the intuitive idea of explanation, PEs resemble more a description. Can we consider any additional information about objects as an explanation? Should this information be structured in a particular way to gain the status of explanation? If yes, how? If not, what is the point of introducing the concept of explanation?\n- Experiments show a comparison between an agent trained using only input images and an agent trained also using additional relevant information about the properties of each object. This setup could only say something about the performance of agents trained using more or less information. Instead, we expect to see the comparison between agents trained using the same amount of information, but in one case shaped as an explanation and in the other case not. It is not clear what the authors mean by \"prior knowledge of language.\"\n\nThe paper is not properly structured. The methods section is far too short. Many aspects should be expanded, such as the description of the RL environment and the training datasets for the NN.\nThe experiment section is poor despite its length, it should be enriched with a comparison against existing approaches, even if slightly out-of-task for odd-one-out. The related work section suffers the same problem. It is far too long and, in the end, not very informative for the proposed approach. \nSection 4 provides interesting references to literature in psychology and neuroscience. Yet, it should be shortened by removing less relevant parts, such as the \"Self-explanation\" paragraph. \nSection 5 does not do a good job in placing this work in AI literature. References are too generic (e.g. the NLP paragraph) and should tackle more competitive approaches.\n\n[1] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning. In Proc. CVPR, pp. 1988–1997, 2017.", "summary_of_the_paper": "The authors introduce 2D and 3D Reinforcement Learning tasks that ask the agent to find the odd object in a collection of objects. The experiments show that an agent trained using only input images performs worse than an agent trained also using clean abstract information about the properties of each object.", "summary_of_the_review": "Despite the idea of including textual explanations in the learning pipeline to improve performance is interesting and trending these days in the DL research community, the overall contribution of this paper is very modest. The paper's central claim is ambiguous due to a missing operative definition of explanation and not adequately supported by experiments. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "djnxs8tSKUY", "reviewer": "Reviewer_x1cM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I enjoyed reading this paper, and thank the authors for their clear exposition.\n\nThe intro and background were clear, and well-situated in the computer and cognitive science, and the psychology literature. The experimental setup was also one I had not seen before (the authors say it is one of their contributions on page 2; so thank you for this interesting setup).\n\nDespite the authors' good grasp of the state of the art, and in fact, because it, I would ask that the authors refrain from over-claiming in their experimental results. As this paper will likely be read by many researchers, I would emphasize the potential for impact that the paper has to demonstrate proper drawing of scientific conclusions. Some examples here:\n* bot of page 3: \"never perform above chance\"; they do, in the middle and towards the end of training --> \"barely/hardly/seldomly perform above chance\"\n* top of page 5: \"allows strong ood generalization\"; what is strong? have you quantified this? even qualitatively/relatively speaking, one needs some metric to compare diff settings.\n* mid of page 8: \"causally-relevant and generalizable explanations\": what is meant by this? causality has clear definitions, which are absent here. on the other hand, explanation relevance is difficult to ascertain (as the authors have mentioned), thus \"causally relevant explanations\" is doubly vague here. Furthermore, causal relations, at least in the SCM framework and through the independent mechanisms assumptions, is considered \"generalizable\" by definition.\n* mid of page 9: \"Generally, having more types of explanations is better\"; this is false, as you had shown that random/irrelevant explanations hurt performance. In your investigated setting, and for the two types of explanations, more = better most of the time.\n* title of App A.3: the wording of \"actively detrimental\" is misleading, or confusing at best, and this depends on what you compare with. If explanations are inputs, or input+targets, then there is no change compared to having explanations at all. Explanations only help (in the context of your experiment) if they are only the targets (and relative to this setting, adding them additionally as inputs is detrimental, yes, but one naturally expects the baseline to be the case presented in the abstract/intro, where explanations are absent everywhere).\n\nClearly, the paper investigates many empirical setups, which in the end show that explanation targets are useful in various settings. I wonder:\n* the training objective (not seen in the main body) comprises of various parts; the explanation part is essentially a regularizer, no?\n* are there settings in which agents don't require 10^8 (in 2D) or 10^9 (in 3D and in causal 2D) training steps to start performing?\n* is human language the best regularizer? are there other parsimonious languages that would better suit this task (and perhaps reduce the number of training steps in the process)?\n* curious why you chose to have an explicit agent to explore and environment to do the odd-one-out task; this adds more complexity to the system and makes the subsequent analysis of the impact of explanation targets more difficult.\n* how can one compare the effect of the presented auxiliary loss of explanability with other auxiliary losses?", "summary_of_the_paper": "This paper investigates the use of explanations during the training of an agent for downstream tasks. The investigated tasks are variants of the odd-one-out setting, and the experiments demonstrate improved performance on this task (relative to training without explanation targets) for static (observational) and interactive (interventional) settings when the agent is trained with description-based and/or reward-based explanations. Additional experiments are performed to empirically show the ability of such an agent to \"deconfound\" the shape, color, texture, and sometimes size of the objects, to study the effect of random/irrelevant explanations, and to perform various other ablation studies.", "summary_of_the_review": "The exposition is clear and the background and relevant literature are well-represented. The paper only lightly presents a theoretical analysis of the work, and the value of the work seems to be primarily based in the comprehensive set of experiments presented, which serve to show that training an agent with an auxiliary loss to generate explanations (alongside a scene reconstruction and policy optimization) would allow the agent to perform better on various odd-one-out settings, compared to had it not been trained to give explanations. Notwithstanding over-claimed statements (to-be-addressed) and some open discussion questions highlighted above, this paper seems like one that the ICLR would enjoy reading and referring to.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "V99j2cJd9-", "reviewer": "Reviewer_8zdV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I really enjoyed reading this paper.  It is very well written, most concepts are\npresented clearly, and it explores an important research direction -- namely\nhow explanations provide invaluable supervision during learning.\n\nThe paper has two main downsides:\n\n- It ignores most literature on learning from explanations.  The authors cite\nthe works of Camburu et al and Mu et al, but that's about it.  They go as far\nas calling these \"exceptions\".  Quoting: \"most recent work in AI neglects explanations as a learning signal.\"\n\nThis is however false.  In the last few years, the literature on this topic has\ngrown considerably and I could consider it unfair to not mention at least the\nfollowing works:\n\n- Stammer, Wolfgang, Patrick Schramowski, and Kristian Kersting. \"Right for the Right Concept: Revising Neuro-Symbolic Concepts by Interacting with their Explanations.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3619-3629. 2021.\n\nwhich specifically studies learning from explanations in the context of relational\nprediction tasks.  The problem of \"where do the explanations come from\", namely\nfrom human supervisors, has been studied in detail here:\n\n- Schramowski, Patrick, Wolfgang Stammer, Stefano Teso, Anna Brugger, Franziska Herbert, Xiaoting Shao, Hans-Georg Luigs, Anne-Katrin Mahlein, and Kristian Kersting. \"Making deep neural networks right for the right scientific reasons by interacting with their explanations.\" Nature Machine Intelligence 2, no. 8 (2020): 476-486.\n\nThe authors write \"explanations could be produced by humans, e.g. as annotations of past trajectories\" (p 3) and this is *exactly* the setting that the above paper studies.  This\nwork is based on an earlier paper:  \n\n- Ross, Andrew Slavin, Michael C. Hughes, and Finale Doshi-Velez. \"Right for the right reasons: training differentiable models by constraining their explanations.\" In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pp. 2662-2670. 2017.\n\nwhere the authors develop a way to integrate supervision on saliency maps.  This\nis very relevant although different from the linguistic explanations used in this\nwork.  The literature on debugging NLP models using explanations has been surveyed\nin:\n\n- Lertvittayakumjorn, Piyawat, and Francesca Toni. \"Explanation-Based Human Debugging of NLP Models: A Survey.\" arXiv preprint arXiv:2104.15135 (2021).\n\nWorks on this topic already study whether and why explanations are useful for\ndebugging and deconfouding learned models.  Finally, concerning reinforcement\nlearning, a key paper is:\n\n- Guan, Lin, Mudit Verma, Sihang Guo, Ruohan Zhang, and Subbarao Kambhampati. \"Explanation augmented feedback in human-in-the-loop reinforcement learning.\" arXiv preprint arXiv:2006.14804 (2020).\n\nNotice that I am *not* claiming that the current work is completely subsumed\nby existing studies;  but I am saying that the authors neglec a large set of\nworks and that this is not acceptable.\n\nThe authors should at the bare minimum distinguish their original contributions\nfrom those that were already made in this literature.\n\nA more complete list of works on using explanations as supervision can be found\nhere:\n\n- https://github.com/stefanoteso/awesome-explanatory-supervision\n\nBesides this complaint, I liked the contents of the paper.  The main issue is,\nto my eyes, that little is done to position the proposed work against existing\nresearch.\n\n\n** POST-REBUTTAL UPDATE **: The authors substantially improved their coverage of the related work.  I have increased the score accordingly.", "summary_of_the_paper": "The authors introduce a meta-architecture for learning from explanations.  This\narchitecture is showcased by solving different instantiations of the same\nrelational task (\"one-odd-out\", identifying the one object that stands out\nin a set of objects).  Empirical results showcase the usefulness of learning\nfrom explanations and the ability of the proposed approach to do so.", "summary_of_the_review": "Explores an important research direction but neglects existing literature.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the use of natural language explanations during the training of an agent for odd-one-out tasks. Experiment results show that using quality explanation as abstract information about object properties helps with the agent performance, as compared with the vanilla method.

Strengths:
- Experiment results are conducted thoroughly to support the major claims made by the paper
- The problem is well motivated and has an important implication

Weakness:
- There has been extensive discussion about whether the paper lacks a more formal and rigorous definition of "explanation" as considered in the scope of this paper. 
- Concerns are raised regarding the gaps between the broad claims in the paper and the restricted experiment settings
Decision: Reject