Leveraging the Inductive Bias of Large Language Models for Abstract Textual Reasoning

1 Introduction . Natural language processing ( NLP ) has seen major progress thanks to probabilistic language models ( LMs ) like GPT-2 [ 1 ] , BERT [ 2 ] , and T5 [ 3 ] . These are pre-trained in a general , task-agnostic way on large corpora of unstructured text and then fine-tuned on specific tasks . This method has achieved state of the art performance on popular NLP tasks like question-answering , textual entailment , text summarization , neural machine translation , and more [ 4 ] . These models are not just impressive for the high scores they achieve on quantitative language tasks ; the text they generate often reflects patterns of “ real-world ” structure , suggesting that embedded in the weights of the LM is implicit knowledge of physics , object persistence , containment , spatial relationships , causal mechanisms , material properties , and other common-sense knowledge central to human reasoning and intuition . If that is so , then they ought to provide a useful inductive bias in learning to perform symbolic reasoning tasks that mirror real-world tasks . In this paper , we attempt to leverage and characterize this inductive bias by training reasoning engines that demonstrate some of the hallmarks of human reasoning ability : learning ( a ) rules ( b ) that generalize well ( c ) from few examples . Concretely , we fine-tune T5 on a suite of symbolic reasoning tasks and study generalization along multiple different axes beyond the normal test/train split : we examine cardinality generalization , object generalization , part-of-speech generalization , and show ( in the spirit of curriculum learning ) that LMs can leverage combinations of learned subskills to master complicated composite tasks with both better sample efficiency and higher terminal performance than learning directly on the complicated tasks . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) We see our contributions as four-fold . First , we demonstrate a high level of performance by connectionist models on tasks resembling symbolic classical AI tasks , demonstrating some symbolic ability of such models in light of recent calls to unify the principles of both symbolism and connectionism . Secondly , we demonstrate the breakdown of reasoning ability by manufacturing our own reasoning datasets that can be tweaked in systematic ways , instead of simply split into training/validation/test sets . This means that we can assess our models ’ ability to both interpolate and extrapolate in systematic , symbolic , and grammatical ways , and otherwise flex with changing distributions . Thirdly , we demonstrate the ability of large LMs to reason compositionally , that is , to learn two kinds of reasoning separately and then to combine those different kinds of reasoning on a novel composite task , to which they are both relevant . Lastly , we demonstrate the inductive bias we hypothesize is present in large LMs and can be leveraged to assist in the formation of reasoning engines . 2 Related Work . Transformer-based LMs [ 5 ] are the dominant LM architecture , including the original GPT models [ 1 , 6 ] , BERT [ 2 , 3 ] and the recent 175 billion parameter GPT-3 [ 4 ] . They are trained via generic maximum likelihood learning on vast , unstructured text corpora , but often exhibit zero/few-shot learning abilities on a variety of NLP tasks , having grasped key mechanics of natural language by learning to simply predict a missing word ; it is this inductive bias we hope to harness . These models implicitly house a rich world model with concepts and relations . Petroni et al . query a language model ( instead of a traditional , symbolic knowledge base ) for relational data explicitly expressed in natural langauge [ 7 ] . Bosselut et al . expand this scope , attempting to generate explicit commonsense knowledge graphs using pre-trained LMs [ 8 ] , and Bouraoui et al . perform relation extraction with BERT , which can be construed as finding and specifying edges [ 9 ] . This work treats the formation of explicit knowledge graphs using LMs , whereas ours treats the leveraging of implicit knowledge graphs using LMs . Our tasks are inspired by classic examples of good old-fashioned AI , including Blocks World [ 10 ] and STRIPS planners [ 11 ] , where a system ’ s state , composed of symbols such as toy blocks , must be manipulated to achieve a desired end given a rule set . In contrast to these original problems , we learn the symbolic rules governing a similar system from examples . Our work is therefore similar in spirit to program induction , where a connectionist model must learn a structured program from examples ; examples include the Neural Turing Machine [ 12 ] and the Differentiable Neural Computer [ 13 ] . Some work has attempted to teach a language model rules by explicit teaching through specialized datasets [ 14 ] or knowledge graphs [ 15 ] . We count on these rules being learned implicitly , as a side effect of the general learning objective of LMs . Weber et al . combine neural networks with logic programming to solve multi-hop reasoning tasks in natural language [ 16 ] . We don ’ t insist on expressing explicit representations of symbolic rules learned , instead reasoning about rules learned by examining out-of-distribution performance . Our tasks are similar to the bAbI dataset [ 17 ] . However , we focus on systematic study of specific dimensions of generalization that is impossible with bAbI . bAbI questions evaluate whether certain skills are possessed ; our questions evaluate to what extent these skills are possessed , how they generalize , and when they break down . Finally , the idea of compositional learning draws inspiration from multi-task learning , which was explored in [ 18 ] and more recently in a deep learning approach [ 19 ] . Supplementary Training on Intermediate Labeled-data Tasks ( STILTs ) was introduced in [ 20 ] , which uses two stages of pretraining , where the first stage is unsupervised like with other pre-trained models , but where the second is on some data-rich intermediate supervised task . 3 Data Generation , Evaluation , and Training Protocol . The central questions we ask in this paper revolve around whether large LMs possess some world model or inductive bias that helps them to learn reasoning rules from few examples . These rules should be used to successfully extrapolate not just to novel instances , but to novel tasks . Upon learning to track objects across containers , for example , a language model should be able to generalize to new types of objects , marginally more complicated scenarios than it has seen before , and leverage skills already learned to progress more quickly than if it hadn ’ t . We show how LMs can generalize in these ways and others by systematically fine-tuning them on classes of scenarios where they can learn rules . Across tasks , our reasoners deduce the final state of an environment given natural language descriptions of an initial state and actions taken on it . 3.1 Task Types . Containers . In the first task , which we call containers , we manipulate objects in various containers and ask the reasoner to track the state of the environment . The initial state of the environment is a random allotment of n_objects objects into n_containers containers . The names of these objects and containers are sampled uniformly and without replacement from lists of candidates . Examples of such candidates are given in the appendix . Once sampled and organized , the objects and containers are converted into a plain English expression describing their organization . Then , there is a random manipulation of this initial state , where an object is randomly taken from a container and placed into another container . The task of the reasoning engine is to describe the final state of the environment . For training , each scenario uses 2-8 objects and 2-3 containers , sampled uniformly . We construct sentences by sampling without replacement from a uniform distribution of container names and object names and filling a template accordingly . Our training set of object and container names comes from a proprietary linguistic dataset , which has data on commonness , part of speech , etc . of each word . We divide this dataset into nouns and verbs and subtract their intersection from both . We split the unique nouns into a train set ( n=36566 ) and a validation set ( n=12189 ) and a container set ( n=9 ) , the former two of which are to be used as object names and the latter of which is to be used for container names during training . For each set ( train and val ) , we find the 2000 most common and join the dataset on concreteness ratings from Brysbaert et al . [ 21 ] to take the 2000 most concrete nouns from each set . We also generate a list of random strings by sampling uniformly from single digit integers and lowercase English characters , ranging in length randomly from 5 to 10 . Navigation . In both versions of navigation , a natural-language description of a map of the environment is provided to the reasoning engine . It is generated by sampling from a list of common locations including “ kitchen '' , “ garden '' , and others . These locations are then composed into a grid where they are in north-south-west-east orientation to each other . In the first task , called navigation route , the reasoner is additionally given a starting point and a destination . The reasoner must provide a valid route from origin to destination . In the second task , called navigation result , we require that the reasoner , given a starting point and a route , determine the location where they would find themselves in the map . For training , the maximum number of locations is 8 and the minimum is 3 . Composite task . We are interested in assessing the ability of a reasoning engine to perform a task which combines multiple elemental types of knowledge . Specifically , can a learner master two skills separately , and then leverage knowledge from both to perform the composite task better and more quickly than it would have by learning directly on the composite task ? This task , called hard object , is intended to be a composition of navigation and containers . The reasoner is given a verbal map of the world and an action taken . In this action , an object is taken from its container , carried on a route indicated by successive moves in cardinal directions , and placed . The reasoner must then describe the new state of the containers . 3.2 Generalization Types . Given our overarching interest in large LMs ’ ability to yield reasoners that generalize well , we craft several types of experiments to probe the bounds of different kinds of generalization , namely : Cardinality generalization . This tests a model ’ s structural understanding of the domain . If a reasoner is trained on scenarios with k objects , steps of navigations , rooms , etc . can it generalize to more than k objects , steps of navigations , rooms , etc. ? Object generalization . This is a semantic test of whether or not the reasoner can leverage prior knowledge of English to generalize to new , never-before-seen objects . For example , if several different container training scenarios are composed of objects from different distributions of words ( e.g . 2000 concrete nouns vs. 2000 most common nouns vs. 2000 randomly sampled nouns ) , which distribution of training scenarios results in the best model generalization to scenarios composed with new , previously unseen nouns ? Part-of-speech generalization . This is another semantic test based on the idea that a reasoner that understands language will perform better on “ right '' words than on “ wrong '' words . For example , on the container task , the model should perform well when nouns are objects and poorly when verbs or random strings are . Intuitively , a scenario such as “ The bin contains a dethrone and a transpose '' makes less sense , and is statistically less likely , than a scenario such as “ The bin contains a ball and a snake '' ; the naturalness of the scenario descriptions should positively correlate with generalization . We hypothesize that natural nouns will work best , since nouns are often sensible things to move from one container to the other , and that arbitrary strings and verbs will both decrease performance.On the other hand , random strings have probably never been encountered by the model , and it might learn to simply copy whatever tokens are in specific places , treating strings as opaque IDs , symbols without meaning . Verbs should actively confuse the model , since a verb ’ s linguistic role is structurally and semantically different than a noun ’ s ; it is a symbol not without meaning , but with the wrong meaning . Reasonable phrasing generalization . Finally , what if we replace the templates themselves ? Instead of replacing the objects in a scenario , we replace the English scaffolding we insert them into by mapping deterministically from each English word to a gibberish word composed of English morphemes but possessing no meaning ( See 4.5 ) . A reasoner for which language is meaningful will have a harder time adjusting to this task , while a reasoner for which language is simply a sequence of meaningless strings will struggle no more with this task than with its original English version .

Summary: This paper investigates whether pre-trained language models (T5 in this case) learn priors which support symbolic reasoning tasks in the process of optimizing their purely linguistic objective.  The performance of fine-tuned pre-trained LMs is evaluated on two symbolic reasoning tasks -- object location across a number of boxes, and a room navigation task -- and compared with identical models trained from scratch.  The results touch on a number of topics, but generally show (1) good reasoning accuracy when starting from the pre-trained t5 model, (2) abysmal performance by the from-scratch models, (3) a good degree amount of generalization performance for t5 to reason beyond the fine-tune scenario, and (4) that learning subtasks improves performance on a compositional task.