Tell me why!—Explanations support learning relational and causal structure

Explanations—language that provides explicit information about the abstract , causal structure of the world—are central to human learning ( Keil et al. , 2000 ; Lombrozo , 2006 ) . Explanations help solve the credit assignment problem , because they link a concrete situation to generalizable abstractions that can be used in the future ( Lombrozo , 2006 ; Lombrozo and Carey , 2006 ) . Thus explanations allow us to learn efficiently , from otherwise underspecified examples ( Ahn et al. , 1992 ) . Human explanations selectively highlight generalizable causal factors and thereby improve our causal understanding ( Lombrozo and Carey , 2006 ) . Similarly , they help us to make comparisons and master relational and analogical reasoning ( Gentner and Christie , 2008 ; Lupyan , 2008 ; Edwards et al. , 2019 ) . Even explaining to ourselves , without feedback , can improve our ability to generalize ( Chi et al. , 1994 ; Rittle-Johnson , 2006 ; Williams and Lombrozo , 2010 ) , potentially because explanations abstract knowledge , thus making it easier to recall and generalize ( cf . Dasgupta and Gershman , 2021 ) . These abilities—abstraction , causality , relations and analogies , and generalization—are often cited as areas where deep learning has not yet achieved human-level performance ( e.g . Fodor and Pylyshyn , 1988 ; Lake et al. , 2017 ; Pearl , 2018 ; Marcus , 2020 ; Ichien et al. , 2021 ; Holyoak and Lu , 2021 ; Puebla and Bowers , 2021 ; Geirhos et al. , 2020 ) . Thus , there has been increasing interest in using explanations as learning signal ( e.g . Ross et al. , 2017 ; Mu et al. , 2020 ; Camburu et al. , 2018 ; Schramowski et al. , 2020 ; see related work ) . That is , rather than seeking explanations only posthoc , in an attempt to help humans understand the system ( e.g . Chen et al. , 2018 ; Topin and Veloso , 2019 ; Xie et al. , 2020 ) , these works use explanations to help the system understand the task ( cf . Santoro et al. , 2021 ) . However , most research on learning from explanations has been within supervised learning . There have been a few explorations within reinforcement learning ( RL ; Guan et al. , 2021 ; Tulli et al. , 2020 ) , but these have not focused on relational or causal reasoning . Yet explanations may be particularly useful in these contexts , to augment the sparse learning signals of rewards . Furthermore , explanations help humans to learn causal structure , and RL agents can learn causal structure ( e.g . Dasgupta et al. , 2019 ) , while supervised learners can not ( Pearl , 2018 ) . We therefore explore the benefits of language explanations for RL tasks involving relational and causal structure . We define a language explanation to be a string that indicates relationships between a situation , the agent ’ s behavior , and abstract task structure . Thus , explanations can highlight aspects of a situation that are generalizable to other instances of that task , by elevating task-relevant features above idiosyncratic or task-irrelevant ones . We show that language explanations improve agent learning and generalization , and explore how different aspects of explanations affect their benefits . We first outline a set of challenging relational tasks involving uniqueness—identifying the object from a set that is the odd-one-out along some dimension . These tasks are difficult to learn , because agents need to consider the relationships among all the present objects along various dimensions , rather than simply the properties of a single object or even pairs of objects . We show that these tasks are difficult for agents to learn from reward alone in multiple environments and learning paradigms However , learning and generalization improve substantially when agents are also trained to generate language explanations—even without prior knowledge of language and without telling agents how to use the explanations . Explanations highlight abstract task structure , and thus outperform taskagnostic auxiliary losses ( even ones that offer more supervision ) . Explanations allow agents to learn more effectively , without fixating on easy-but-inadequate “ shortcut ” features ( Geirhos et al. , 2020 ; Hermann and Lampinen , 2020 ) . Explanations allow agents to disentangle confounded features and generalize appropriately out-of-distribution to deconfounded evaluation , and to meta-learn to perform interventions to identify causal structure . We then explore different how different aspects of explanations contribute to their benefits . Our results suggest that generating explanations could be a powerful tool for learning challenging RL tasks . Language explanations may be simpler for humans to produce than other forms of supervision ( e.g . Cabi et al. , 2019 ; Guan et al. , 2021 ) , and could selectively identify the key generalizable features of a situation . Thus , training agents to generate explanations might provide a path towards improving learning and generalization . We highlight the following as our main contributions : 1 THE ODD-ONE-OUT TASKS . We outline a challenging family of fundamentally-relational tasks : finding the odd one out in a set of objects , i.e . the one that is somehow unique ( Fig . 1 ) . Odd-one-out tasks have been used extensively in cognitive science ( e.g . Stephens and Navarro , 2008 ; Crutch et al. , 2009 ) , and proposed in perceptual settings in robotics ( Sinapov and Stoytchev , 2010 ) . These tasks are challenging , because they involve both relational reasoning ( same vs. different ) and abstraction ( identifying uniqueness requires reasoning over all objects , and all dimensions along which objects may be related ) . Furthermore , these tasks permit informative explanations , of relevant dimensions , properties , and relations . Investigating these challenging and abstract—yet explainable—relational tasks is particularly interesting , because relational reasoning and abstraction are critical human abilities ( Gentner , 2003 ; Penn et al. , 2008 ) , but the capacity of deep learning to acquire these skills is disputed ( e.g . Santoro et al. , 2017 ; 2018 ; Geiger et al. , 2020 ; Ichien et al. , 2021 ; Puebla and Bowers , 2021 ) . However , explanations are important to human relational learning ( Gentner and Christie , 2008 ; Lupyan , 2008 ; Edwards et al. , 2019 ) , suggesting that explanations might similarly help machines acquire these skills . In Fig . 1 we conceptually illustrate some odd-one-out tasks . In Fig . 1a the second object is a uniquely green , while the rest are purple . We thus denote color as the relevant dimension in this episode . Along the other , irrelevant dimensions—shape , texture , and size—attributes appear in pairs . For example , there are two pentagons and two triangles . These pairs force the agent to consider all the objects . If the agent considered only the first three objects it would be unable to tell whether the first object was the odd one out ( uniquely large ) , the second ( uniquely green ) , or the third ( uniquely a triangle or uniquely solid textured ) . Thus , the agent must consider all objects to identify the correct dimension and the correct unique feature . This makes the relational reasoning particularly challenging , since the agent must consider many possible relationships . The agent is rewarded for selecting the odd-one-out , either by picking it up , or naming it , depending on the task instantiation . We emphasize that in principle these tasks can be learned from reward alone—language is not necessary for performing them , so we evaluate without language . Nevertheless , we find that in practice language explanations are critical for learning these tasks in our settings . We consider two types of explanations : reward explanations and property explanations ( see Fig . 1 for examples ) . Reward explanations are produced after the agent chooses , and identify the feature ( s ) that make the choice correct or incorrect . Property explanations are produced before the agent chooses , and explain the identity of the object the agent is facing by specifying its task-relevant properties . Both types satisfy our above criterion for explanations : they link the situation and the agent ’ s behavior to the task structure . 1.1 ENVIRONMENTS . Odd-one-out tasks can be instantiated in various settings , from games to language or images , and can incorporate various latent structures ( e.g . meta-learning ) . We instantiate these tasks in 2D and 3D RL environments ( Fig . 3a ) . In 2D , the agent has simple directional movement actions , while in 3D it can move , look around , and grasp nearby objects at which it is looking . In both environments we place an agent in a room containing four objects , which vary along feature dimensions of color , texture , position , and either shape ( 2D ) or size ( 3D ) . In each episode , one object will be unique along one dimension . The 3D environment compounds the difficulty of the odd-one-out tasks , because the agent ’ s limited view often forces it to compare objects in memory . See Appx . C.2 for full details . 2 METHOD : GENERATING EXPLANATIONS AS AUXILIARY TRAINING . We focus on language explanations provided during training . We synthetically generate the explanations online , conditional on agent behavior . However , explanations could be produced by humans , e.g . as annotations of past trajectories ( cf . Ross et al. , 2017 ) . We train the agent to generate explanations as an auxiliary signal to shape its representations ( Fig . 2 ) , as opposed to providing explanations as direct inputs ( which is less effective ; Appx . A.3 ) ; our approach thus does not require explanations at test time . Note that we do not directly supervise behaviour through explanations , nor tell the agent how to use them . The agent simply predicts explanations as an auxiliary output . We train agents using the IMPALA ( Espeholt et al. , 2018 ) framework . Our agent ( Fig . 2 ) consists of a visual encoder , a memory , and output heads . The encoder is a CNN or ResNet ( task-dependent ) . The agent remembers using a 4-layer Gated TransformerXL Memory ( Parisotto et al. , 2020 ) , with the visual encoder output and previous reward as inputs . The output of the memory is input to the heads . The policy and value heads are MLPs , trained with V -trace . Another head reconstructs the input images , to learn better representations ( though this is not necessary ; Appx . A.7 ) . Finally , the explanation head is a single-layer LSTM , which generates language explanations . We train the agent to predict these explanations using a summed softmax cross-entropy loss . See Appx . C.1 for details . 3 EXPERIMENTS . 3.1 PERCEPTUAL ODD-ONE-OUT TASKS IN 2D AND 3D RL ENVIRONMENTS . Explanations help agents to learn perceptual odd-one-out tasks . We first evaluate the benefit of explanations for learning the odd-one-out tasks , by comparing agents trained without explanations to agents trained with property and reward explanations . In both 2D and 3D environments , agents trained with explanations learn to solve the tasks over 90 % of the time ( Figs . 3b-c ) . Agents trained without explanations perform worse ; in the easier 2D environment they exhibit partial learning ( see 3.4 ) , while in the more challenging 3D environment they barely perform above chance . All agents in 2D were trained with an auxiliary unsupervised reconstruction loss . However , agents trained without reconstruction but with explanations perform well ( Appx . A.7 ) , while agents trained with reconstruction but without explanations do not . Because explanations highlight abstract task structure , they outperform task-agnostic unsupervised objectives , even ones that provide strictly more supervision .

Summary: This is a paper about sort-one-out identification. The goal is to emphasise that standard reinforcement learning is not optimal for this task, while adding (language) explanations to the learning phase might be highly beneficial. Dedicated experiments are developed for 2- and 3-D simulation environments. Each experiment is used to advocate a specific point such as the effect of the explanations on accuracy, robustness wrt correlated features, and other more detailed analyses.