Leveraging the Inductive Bias of Large Language Models for Abstract Textual Reasoning

1 Introduction . Natural language processing ( NLP ) has seen major progress thanks to probabilistic language models ( LMs ) like GPT-2 [ 1 ] , BERT [ 2 ] , and T5 [ 3 ] . These are pre-trained in a general , task-agnostic way on large corpora of unstructured text and then fine-tuned on specific tasks . This method has achieved state of the art performance on popular NLP tasks like question-answering , textual entailment , text summarization , neural machine translation , and more [ 4 ] . These models are not just impressive for the high scores they achieve on quantitative language tasks ; the text they generate often reflects patterns of “ real-world ” structure , suggesting that embedded in the weights of the LM is implicit knowledge of physics , object persistence , containment , spatial relationships , causal mechanisms , material properties , and other common-sense knowledge central to human reasoning and intuition . If that is so , then they ought to provide a useful inductive bias in learning to perform symbolic reasoning tasks that mirror real-world tasks . In this paper , we attempt to leverage and characterize this inductive bias by training reasoning engines that demonstrate some of the hallmarks of human reasoning ability : learning ( a ) rules ( b ) that generalize well ( c ) from few examples . Concretely , we fine-tune T5 on a suite of symbolic reasoning tasks and study generalization along multiple different axes beyond the normal test/train split : we examine cardinality generalization , object generalization , part-of-speech generalization , and show ( in the spirit of curriculum learning ) that LMs can leverage combinations of learned subskills to master complicated composite tasks with both better sample efficiency and higher terminal performance than learning directly on the complicated tasks . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) We see our contributions as four-fold . First , we demonstrate a high level of performance by connectionist models on tasks resembling symbolic classical AI tasks , demonstrating some symbolic ability of such models in light of recent calls to unify the principles of both symbolism and connectionism . Secondly , we demonstrate the breakdown of reasoning ability by manufacturing our own reasoning datasets that can be tweaked in systematic ways , instead of simply split into training/validation/test sets . This means that we can assess our models ’ ability to both interpolate and extrapolate in systematic , symbolic , and grammatical ways , and otherwise flex with changing distributions . Thirdly , we demonstrate the ability of large LMs to reason compositionally , that is , to learn two kinds of reasoning separately and then to combine those different kinds of reasoning on a novel composite task , to which they are both relevant . Lastly , we demonstrate the inductive bias we hypothesize is present in large LMs and can be leveraged to assist in the formation of reasoning engines . 2 Related Work . Transformer-based LMs [ 5 ] are the dominant LM architecture , including the original GPT models [ 1 , 6 ] , BERT [ 2 , 3 ] and the recent 175 billion parameter GPT-3 [ 4 ] . They are trained via generic maximum likelihood learning on vast , unstructured text corpora , but often exhibit zero/few-shot learning abilities on a variety of NLP tasks , having grasped key mechanics of natural language by learning to simply predict a missing word ; it is this inductive bias we hope to harness . These models implicitly house a rich world model with concepts and relations . Petroni et al . query a language model ( instead of a traditional , symbolic knowledge base ) for relational data explicitly expressed in natural langauge [ 7 ] . Bosselut et al . expand this scope , attempting to generate explicit commonsense knowledge graphs using pre-trained LMs [ 8 ] , and Bouraoui et al . perform relation extraction with BERT , which can be construed as finding and specifying edges [ 9 ] . This work treats the formation of explicit knowledge graphs using LMs , whereas ours treats the leveraging of implicit knowledge graphs using LMs . Our tasks are inspired by classic examples of good old-fashioned AI , including Blocks World [ 10 ] and STRIPS planners [ 11 ] , where a system ’ s state , composed of symbols such as toy blocks , must be manipulated to achieve a desired end given a rule set . In contrast to these original problems , we learn the symbolic rules governing a similar system from examples . Our work is therefore similar in spirit to program induction , where a connectionist model must learn a structured program from examples ; examples include the Neural Turing Machine [ 12 ] and the Differentiable Neural Computer [ 13 ] . Some work has attempted to teach a language model rules by explicit teaching through specialized datasets [ 14 ] or knowledge graphs [ 15 ] . We count on these rules being learned implicitly , as a side effect of the general learning objective of LMs . Weber et al . combine neural networks with logic programming to solve multi-hop reasoning tasks in natural language [ 16 ] . We don ’ t insist on expressing explicit representations of symbolic rules learned , instead reasoning about rules learned by examining out-of-distribution performance . Our tasks are similar to the bAbI dataset [ 17 ] . However , we focus on systematic study of specific dimensions of generalization that is impossible with bAbI . bAbI questions evaluate whether certain skills are possessed ; our questions evaluate to what extent these skills are possessed , how they generalize , and when they break down . Finally , the idea of compositional learning draws inspiration from multi-task learning , which was explored in [ 18 ] and more recently in a deep learning approach [ 19 ] . Supplementary Training on Intermediate Labeled-data Tasks ( STILTs ) was introduced in [ 20 ] , which uses two stages of pretraining , where the first stage is unsupervised like with other pre-trained models , but where the second is on some data-rich intermediate supervised task . 3 Data Generation , Evaluation , and Training Protocol . The central questions we ask in this paper revolve around whether large LMs possess some world model or inductive bias that helps them to learn reasoning rules from few examples . These rules should be used to successfully extrapolate not just to novel instances , but to novel tasks . Upon learning to track objects across containers , for example , a language model should be able to generalize to new types of objects , marginally more complicated scenarios than it has seen before , and leverage skills already learned to progress more quickly than if it hadn ’ t . We show how LMs can generalize in these ways and others by systematically fine-tuning them on classes of scenarios where they can learn rules . Across tasks , our reasoners deduce the final state of an environment given natural language descriptions of an initial state and actions taken on it . 3.1 Task Types . Containers . In the first task , which we call containers , we manipulate objects in various containers and ask the reasoner to track the state of the environment . The initial state of the environment is a random allotment of n_objects objects into n_containers containers . The names of these objects and containers are sampled uniformly and without replacement from lists of candidates . Examples of such candidates are given in the appendix . Once sampled and organized , the objects and containers are converted into a plain English expression describing their organization . Then , there is a random manipulation of this initial state , where an object is randomly taken from a container and placed into another container . The task of the reasoning engine is to describe the final state of the environment . For training , each scenario uses 2-8 objects and 2-3 containers , sampled uniformly . We construct sentences by sampling without replacement from a uniform distribution of container names and object names and filling a template accordingly . Our training set of object and container names comes from a proprietary linguistic dataset , which has data on commonness , part of speech , etc . of each word . We divide this dataset into nouns and verbs and subtract their intersection from both . We split the unique nouns into a train set ( n=36566 ) and a validation set ( n=12189 ) and a container set ( n=9 ) , the former two of which are to be used as object names and the latter of which is to be used for container names during training . For each set ( train and val ) , we find the 2000 most common and join the dataset on concreteness ratings from Brysbaert et al . [ 21 ] to take the 2000 most concrete nouns from each set . We also generate a list of random strings by sampling uniformly from single digit integers and lowercase English characters , ranging in length randomly from 5 to 10 . Navigation . In both versions of navigation , a natural-language description of a map of the environment is provided to the reasoning engine . It is generated by sampling from a list of common locations including “ kitchen '' , “ garden '' , and others . These locations are then composed into a grid where they are in north-south-west-east orientation to each other . In the first task , called navigation route , the reasoner is additionally given a starting point and a destination . The reasoner must provide a valid route from origin to destination . In the second task , called navigation result , we require that the reasoner , given a starting point and a route , determine the location where they would find themselves in the map . For training , the maximum number of locations is 8 and the minimum is 3 . Composite task . We are interested in assessing the ability of a reasoning engine to perform a task which combines multiple elemental types of knowledge . Specifically , can a learner master two skills separately , and then leverage knowledge from both to perform the composite task better and more quickly than it would have by learning directly on the composite task ? This task , called hard object , is intended to be a composition of navigation and containers . The reasoner is given a verbal map of the world and an action taken . In this action , an object is taken from its container , carried on a route indicated by successive moves in cardinal directions , and placed . The reasoner must then describe the new state of the containers . 3.2 Generalization Types . Given our overarching interest in large LMs ’ ability to yield reasoners that generalize well , we craft several types of experiments to probe the bounds of different kinds of generalization , namely : Cardinality generalization . This tests a model ’ s structural understanding of the domain . If a reasoner is trained on scenarios with k objects , steps of navigations , rooms , etc . can it generalize to more than k objects , steps of navigations , rooms , etc. ? Object generalization . This is a semantic test of whether or not the reasoner can leverage prior knowledge of English to generalize to new , never-before-seen objects . For example , if several different container training scenarios are composed of objects from different distributions of words ( e.g . 2000 concrete nouns vs. 2000 most common nouns vs. 2000 randomly sampled nouns ) , which distribution of training scenarios results in the best model generalization to scenarios composed with new , previously unseen nouns ? Part-of-speech generalization . This is another semantic test based on the idea that a reasoner that understands language will perform better on “ right '' words than on “ wrong '' words . For example , on the container task , the model should perform well when nouns are objects and poorly when verbs or random strings are . Intuitively , a scenario such as “ The bin contains a dethrone and a transpose '' makes less sense , and is statistically less likely , than a scenario such as “ The bin contains a ball and a snake '' ; the naturalness of the scenario descriptions should positively correlate with generalization . We hypothesize that natural nouns will work best , since nouns are often sensible things to move from one container to the other , and that arbitrary strings and verbs will both decrease performance.On the other hand , random strings have probably never been encountered by the model , and it might learn to simply copy whatever tokens are in specific places , treating strings as opaque IDs , symbols without meaning . Verbs should actively confuse the model , since a verb ’ s linguistic role is structurally and semantically different than a noun ’ s ; it is a symbol not without meaning , but with the wrong meaning . Reasonable phrasing generalization . Finally , what if we replace the templates themselves ? Instead of replacing the objects in a scenario , we replace the English scaffolding we insert them into by mapping deterministically from each English word to a gibberish word composed of English morphemes but possessing no meaning ( See 4.5 ) . A reasoner for which language is meaningful will have a harder time adjusting to this task , while a reasoner for which language is simply a sequence of meaningless strings will struggle no more with this task than with its original English version .

Summary: In this work, the authors investigate if large scale pre-trained language models (such as T5) can provide inductive biases that are useful for solving language-based symbolic reasoning tasks, and furthermore, generalize to unseen settings.   First, the authors define four types of tasks. While all the four tasks are formulated as sequence generation given prompts (prefixes), they require different kinds of reasoning:  1. Container. The prefix text describes an initial states of the world and a sequence of transitions, the target sequence needs to describe the ending states.  2. Navigation Route. The prefix text describes a map as well as two positions in the map, the target sequence is the navigation path from one position to the other.  3. Navigation Result. The prefix text describes a map, a starting position, and a navigation path, the target is the ending position.  4. Composite Task. A mixture of container and navigation tasks.   Then, the authors define a set of experimental settings targeting the generalizability of models, from a variety of aspects such as map sizes, object/container numbers, reasoning steps required as well as some linguistic properties (e.g., replacing nouns with words with other POS tags, or even made-up words).  The authors provide plenty of experiments, suggesting the pre-trained language models indeed have capability of learning and to some extent generalizing symbolic rules via language-based tasks.