Tell me why!—Explanations support learning relational and causal structure

Explanations—language that provides explicit information about the abstract , causal structure of the world—are central to human learning ( Keil et al. , 2000 ; Lombrozo , 2006 ) . Explanations help solve the credit assignment problem , because they link a concrete situation to generalizable abstractions that can be used in the future ( Lombrozo , 2006 ; Lombrozo and Carey , 2006 ) . Thus explanations allow us to learn efficiently , from otherwise underspecified examples ( Ahn et al. , 1992 ) . Human explanations selectively highlight generalizable causal factors and thereby improve our causal understanding ( Lombrozo and Carey , 2006 ) . Similarly , they help us to make comparisons and master relational and analogical reasoning ( Gentner and Christie , 2008 ; Lupyan , 2008 ; Edwards et al. , 2019 ) . Even explaining to ourselves , without feedback , can improve our ability to generalize ( Chi et al. , 1994 ; Rittle-Johnson , 2006 ; Williams and Lombrozo , 2010 ) , potentially because explanations abstract knowledge , thus making it easier to recall and generalize ( cf . Dasgupta and Gershman , 2021 ) . These abilities—abstraction , causality , relations and analogies , and generalization—are often cited as areas where deep learning has not yet achieved human-level performance ( e.g . Fodor and Pylyshyn , 1988 ; Lake et al. , 2017 ; Pearl , 2018 ; Marcus , 2020 ; Ichien et al. , 2021 ; Holyoak and Lu , 2021 ; Puebla and Bowers , 2021 ; Geirhos et al. , 2020 ) . Thus , there has been increasing interest in using explanations as learning signal ( e.g . Ross et al. , 2017 ; Mu et al. , 2020 ; Camburu et al. , 2018 ; Schramowski et al. , 2020 ; see related work ) . That is , rather than seeking explanations only posthoc , in an attempt to help humans understand the system ( e.g . Chen et al. , 2018 ; Topin and Veloso , 2019 ; Xie et al. , 2020 ) , these works use explanations to help the system understand the task ( cf . Santoro et al. , 2021 ) . However , most research on learning from explanations has been within supervised learning . There have been a few explorations within reinforcement learning ( RL ; Guan et al. , 2021 ; Tulli et al. , 2020 ) , but these have not focused on relational or causal reasoning . Yet explanations may be particularly useful in these contexts , to augment the sparse learning signals of rewards . Furthermore , explanations help humans to learn causal structure , and RL agents can learn causal structure ( e.g . Dasgupta et al. , 2019 ) , while supervised learners can not ( Pearl , 2018 ) . We therefore explore the benefits of language explanations for RL tasks involving relational and causal structure . We define a language explanation to be a string that indicates relationships between a situation , the agent ’ s behavior , and abstract task structure . Thus , explanations can highlight aspects of a situation that are generalizable to other instances of that task , by elevating task-relevant features above idiosyncratic or task-irrelevant ones . We show that language explanations improve agent learning and generalization , and explore how different aspects of explanations affect their benefits . We first outline a set of challenging relational tasks involving uniqueness—identifying the object from a set that is the odd-one-out along some dimension . These tasks are difficult to learn , because agents need to consider the relationships among all the present objects along various dimensions , rather than simply the properties of a single object or even pairs of objects . We show that these tasks are difficult for agents to learn from reward alone in multiple environments and learning paradigms However , learning and generalization improve substantially when agents are also trained to generate language explanations—even without prior knowledge of language and without telling agents how to use the explanations . Explanations highlight abstract task structure , and thus outperform taskagnostic auxiliary losses ( even ones that offer more supervision ) . Explanations allow agents to learn more effectively , without fixating on easy-but-inadequate “ shortcut ” features ( Geirhos et al. , 2020 ; Hermann and Lampinen , 2020 ) . Explanations allow agents to disentangle confounded features and generalize appropriately out-of-distribution to deconfounded evaluation , and to meta-learn to perform interventions to identify causal structure . We then explore different how different aspects of explanations contribute to their benefits . Our results suggest that generating explanations could be a powerful tool for learning challenging RL tasks . Language explanations may be simpler for humans to produce than other forms of supervision ( e.g . Cabi et al. , 2019 ; Guan et al. , 2021 ) , and could selectively identify the key generalizable features of a situation . Thus , training agents to generate explanations might provide a path towards improving learning and generalization . We highlight the following as our main contributions : 1 THE ODD-ONE-OUT TASKS . We outline a challenging family of fundamentally-relational tasks : finding the odd one out in a set of objects , i.e . the one that is somehow unique ( Fig . 1 ) . Odd-one-out tasks have been used extensively in cognitive science ( e.g . Stephens and Navarro , 2008 ; Crutch et al. , 2009 ) , and proposed in perceptual settings in robotics ( Sinapov and Stoytchev , 2010 ) . These tasks are challenging , because they involve both relational reasoning ( same vs. different ) and abstraction ( identifying uniqueness requires reasoning over all objects , and all dimensions along which objects may be related ) . Furthermore , these tasks permit informative explanations , of relevant dimensions , properties , and relations . Investigating these challenging and abstract—yet explainable—relational tasks is particularly interesting , because relational reasoning and abstraction are critical human abilities ( Gentner , 2003 ; Penn et al. , 2008 ) , but the capacity of deep learning to acquire these skills is disputed ( e.g . Santoro et al. , 2017 ; 2018 ; Geiger et al. , 2020 ; Ichien et al. , 2021 ; Puebla and Bowers , 2021 ) . However , explanations are important to human relational learning ( Gentner and Christie , 2008 ; Lupyan , 2008 ; Edwards et al. , 2019 ) , suggesting that explanations might similarly help machines acquire these skills . In Fig . 1 we conceptually illustrate some odd-one-out tasks . In Fig . 1a the second object is a uniquely green , while the rest are purple . We thus denote color as the relevant dimension in this episode . Along the other , irrelevant dimensions—shape , texture , and size—attributes appear in pairs . For example , there are two pentagons and two triangles . These pairs force the agent to consider all the objects . If the agent considered only the first three objects it would be unable to tell whether the first object was the odd one out ( uniquely large ) , the second ( uniquely green ) , or the third ( uniquely a triangle or uniquely solid textured ) . Thus , the agent must consider all objects to identify the correct dimension and the correct unique feature . This makes the relational reasoning particularly challenging , since the agent must consider many possible relationships . The agent is rewarded for selecting the odd-one-out , either by picking it up , or naming it , depending on the task instantiation . We emphasize that in principle these tasks can be learned from reward alone—language is not necessary for performing them , so we evaluate without language . Nevertheless , we find that in practice language explanations are critical for learning these tasks in our settings . We consider two types of explanations : reward explanations and property explanations ( see Fig . 1 for examples ) . Reward explanations are produced after the agent chooses , and identify the feature ( s ) that make the choice correct or incorrect . Property explanations are produced before the agent chooses , and explain the identity of the object the agent is facing by specifying its task-relevant properties . Both types satisfy our above criterion for explanations : they link the situation and the agent ’ s behavior to the task structure . 1.1 ENVIRONMENTS . Odd-one-out tasks can be instantiated in various settings , from games to language or images , and can incorporate various latent structures ( e.g . meta-learning ) . We instantiate these tasks in 2D and 3D RL environments ( Fig . 3a ) . In 2D , the agent has simple directional movement actions , while in 3D it can move , look around , and grasp nearby objects at which it is looking . In both environments we place an agent in a room containing four objects , which vary along feature dimensions of color , texture , position , and either shape ( 2D ) or size ( 3D ) . In each episode , one object will be unique along one dimension . The 3D environment compounds the difficulty of the odd-one-out tasks , because the agent ’ s limited view often forces it to compare objects in memory . See Appx . C.2 for full details . 2 METHOD : GENERATING EXPLANATIONS AS AUXILIARY TRAINING . We focus on language explanations provided during training . We synthetically generate the explanations online , conditional on agent behavior . However , explanations could be produced by humans , e.g . as annotations of past trajectories ( cf . Ross et al. , 2017 ) . We train the agent to generate explanations as an auxiliary signal to shape its representations ( Fig . 2 ) , as opposed to providing explanations as direct inputs ( which is less effective ; Appx . A.3 ) ; our approach thus does not require explanations at test time . Note that we do not directly supervise behaviour through explanations , nor tell the agent how to use them . The agent simply predicts explanations as an auxiliary output . We train agents using the IMPALA ( Espeholt et al. , 2018 ) framework . Our agent ( Fig . 2 ) consists of a visual encoder , a memory , and output heads . The encoder is a CNN or ResNet ( task-dependent ) . The agent remembers using a 4-layer Gated TransformerXL Memory ( Parisotto et al. , 2020 ) , with the visual encoder output and previous reward as inputs . The output of the memory is input to the heads . The policy and value heads are MLPs , trained with V -trace . Another head reconstructs the input images , to learn better representations ( though this is not necessary ; Appx . A.7 ) . Finally , the explanation head is a single-layer LSTM , which generates language explanations . We train the agent to predict these explanations using a summed softmax cross-entropy loss . See Appx . C.1 for details . 3 EXPERIMENTS . 3.1 PERCEPTUAL ODD-ONE-OUT TASKS IN 2D AND 3D RL ENVIRONMENTS . Explanations help agents to learn perceptual odd-one-out tasks . We first evaluate the benefit of explanations for learning the odd-one-out tasks , by comparing agents trained without explanations to agents trained with property and reward explanations . In both 2D and 3D environments , agents trained with explanations learn to solve the tasks over 90 % of the time ( Figs . 3b-c ) . Agents trained without explanations perform worse ; in the easier 2D environment they exhibit partial learning ( see 3.4 ) , while in the more challenging 3D environment they barely perform above chance . All agents in 2D were trained with an auxiliary unsupervised reconstruction loss . However , agents trained without reconstruction but with explanations perform well ( Appx . A.7 ) , while agents trained with reconstruction but without explanations do not . Because explanations highlight abstract task structure , they outperform task-agnostic unsupervised objectives , even ones that provide strictly more supervision .

Summary: The authors introduce a meta-architecture for learning from explanations.  This architecture is showcased by solving different instantiations of the same relational task ("one-odd-out", identifying the one object that stands out in a set of objects).  Empirical results showcase the usefulness of learning from explanations and the ability of the proposed approach to do so.