Leveraging the Inductive Bias of Large Language Models for Abstract Textual Reasoning
Authors: Christopher Michael Rytting
David Wingate
Venue: NeurIPS 2021 Poster (2021)
Keywords: natural language processing
reasoning
transfer learning
generalization

Abstract: Large natural language models (LMs) (such as GPT-3 or T5) demonstrate impressive abilities across a range of general NLP tasks. Here, we show that the knowledge embedded in such models provides a useful inductive bias, not just on traditional NLP tasks, but also in the nontraditional task of training a symbolic reasoning engine. We observe that these engines learn quickly and generalize in a natural way that reflects human intuition. For example, training such a system to model block-stacking might naturally generalize to stacking other types of objects because of structure in the real world that has been partially captured by the language describing it. We study several abstract textual reasoning tasks, such as object manipulation and navigation, and demonstrate multiple types of generalization to novel scenarios and the symbols that comprise them. We also demonstrate the surprising utility of $\textit{compositional learning}$, where a learner dedicated to mastering a complicated task gains an advantage by training on relevant simpler tasks instead of jumping straight to the complicated task. 

Leveraging the Inductive Bias of Large Language Models for Abstract Textual Reasoning
Christopher Michael Rytting Department of Computer Science Brigham Young University
David Wingate
Department of Computer Science Brigham Young University
Provo, UT 84602 chrisryttingdbyu. edu
Provo, UT 84602 wingated@cs byu edu
Abstract
Large natural language models (such as GPT-3 or T5) demonstrate impressive abilities across a range of general NLP tasks. Here, we show that the knowledge embedded in such models provides a useful inductive bias, not just on traditional NLP tasks, but also the nontraditional task of training a symbolic reasoning engine. We observe that these engines learn quickly and generalize in natural way that reflects human intuition. For example, training such a system to model block- stacking might naturally generalize stacking other types of objects because structure in the real world that has been partially captured by the language describing it. We study several abstract textual reasoning tasks, such object manipulation and navigation, and demonstrate multiple types of generalization to novel scenarios and the symbols that comprise them. We also demonstrate the surprising utility of compositional learning, where a learner dedicated to mastering complicated task gains advantage by training relevant simpler tasks instead of jumping straight to the complicated task.
Introduction
Natural language processing (NI.P) has seen major progress thanks to prohabilistic language models (LMs) like GPT-2 (1. BERT 2 and T5 3. These pre-trained in general, task-agnostic way large corpora of unstructured text and then tunettned on specific tasks. This method has achieved state of the art performance on popular NLP tasks like anestion-answering textual entailment, text summarization, neural machine translation, and more 4
These models not just impressive for the high scores they achieve quantitative language tasks; the text they generate often reflects patterns of "real-worke structure, suggesting that embedded in the weights of the LM is implicit knowledge of physics, object persistence, containment, spatial relationships, causal mechanisms, material properties, and other common-sense knowledge central to human reasoning and intuition. If that is so, then they ought provide useful inductive bias in learning to perform symbolic reasoning tasks that mirror real-world tasks.
In this paper, attempt to leverage and characterize this inductive bias by training reasoning engines that demonstrate some of the hallmarks human reasoning ability: learning (a) rules (b) that generalize well (c) from few examples. Concretely, we fine-tune T5 suite of symbolic reasoning tasks and study generalization along multiple different axes beyond the normal test/train split: we examine cardinality generalization, object generalization, part-of-speech generalization, and show (in the spirit of curriculum lcarning) that LMs can leverage combinations of learned subskills to master complicated composite tasks with both better sample efficiency and higher terminal performance than learning directly on the complicated tasks.
35th Conference on Neural Information Processing Systems (NcurIPS 2021)

---

We see our contributions as four-fold. First, we demonstrate a high level of performance by connec- tionist models on tasks resembling symbolic classical AI tasks, demonstrating some symbolic ability of such models in light of recent calls to unify the principles of both symbolism and connectionism. Secondly, we demonstrate the breakdown of reasoning ability by manufacturing our own reasoning datasets that can be tweaked in systematic ways, instead of simply split into training/validationnees sets. This means that we can assess our models' ability to both interpolate and extrapolate in sys- tematic, symbolic, and grammatical ways, and otherwise flex with changing distributions. Thirdly, we demonstrate the ability of large LMs to reason compositionally, that is, to learn two kinds of reasoning separately and then to combine those different kinds of reasoning on novel composite task, to which they are both relevant. Lastly, we demonstrate the inductive bias we hypothesize is present in large LMs and can be leveraged to assist in the formation of reasoning engines.
2 Related Work
Transformer- based LMs [5] are the dominant LM architecture, including the original GPT models 6]. BERT [2 and the recent 175 billion parameter GPT-3 [4]. They are trained via generic maximum likelihood learning on vast, unstructured text corpora, but often exhibit zero/few-shor learning abilities on a variety of NLP tasks, having grasped key mechanics of natural language by learning to simply predict a missing word; it is this inductive bias we hope to harness.
These models implicitly house rich world model with concepts and relations. Petroni et al. query a language model (instead of a traditional, symbolic knowledge base) for relational data explicitly expressed in natural langauge 77. Bosselut al. expand this scope, attempting to generate explicit commonsense knowledge graphs using pre-trained LMs (8), and Bouraoui et al. perform relation extraction with BERT, which can be construed as finding and specifying edges [ This work treats the formation of explicit knowledge graphs using LMs, whereas ours treats the leveraging of implicit knowledge graphs using LMs.
Our tasks are inspired by classic examples of good old-foshioned AI, including Blocks World [0] and STRIPS planners [1], where system's 's state, composed of symbols such toy blocks, must be manipulated to achieve a desired end given a rule set. In contrast to these original problems, we learn the symbolic rules governing a similar system from examples. Our work is therefore similar in spirit to program induction, where a connectionist model must learn a structured program from examples; examples include the Neural Turing Machine [12] and the Differentiable Neural Computer [13) Some work has attempted to teach language model rules by explicit teaching through specialized datasets [14] or knowledge graphs [15). We count on these rules being learned implicitly, as a side effect of the general learning objective of LMs. Weber et al. combine neural networks with logic programming to solve multi-hop reasoning tasks in natural language [6]. We don't 't insist on expressing explicit representations of symbolic rules learned, instead reasoning about rules learned by examining out-of -istrrrbbiiiito performance.
Our tasks are similar to the bAbl dataset (17]. However, we focus on systematic study of specific dimensions of generalization that is impossible with bAbl. bAbl questions evaluate whether certain skills are possessed; our questions evaluate to what extent these skills are possessed, how they generalize, and when they break down.
Finally, the idea of compositional learning draws inspiration from multi-task learning, which was explored in [18] and more recently in deep learning approach [19. Supplementary Training on Intermediate Labeled-data Tasks (STILTs) was introduced in (20,, which uses two stages of pre- training, where the first stage is unsupervised like with other pre-trained models, but where the second is on some data-rich intermediate supervised task.
3 Data Generation, Evaluation, and Training Protocol
The central questions we ask in this paper revolve around whether large LMS possess some world model or inductive bias that helps them to learn reasoning rules from few examples. These rules should be used to successfully extrapolate not just to novel instances, but to novel tasks. Upon learning to track objects across containers, for example, language model should be able to generalize to new types of objects, marginally more complicated scenarios than it has seen before, and leverage skills already learned to progress more quickly than if it hadn't.

---

Table 1: Examples of scenarios for each task. Non-bold words indicate natural language templates that are filled in with randomly sampled words. Additionally, template phrases can repeat to describe variable number of containers, rooms, objects, navigation steps.
Task
Prefix
Target
Container
The bin contains a ball and a snake. The box contains a quilt. Took a quitt from the box and put it in the bin.
The bin contains a ball, a quilt, and a snake. The box contains objects. to the west.
Navigation Route
The garden is to the west of the kitchen. The bedroom is to the south of the kitchen. To get from the kitchen to the garden. you must
Navigation Result
The garden
the west the kitchen. The bedroom is to the south of the kitchen. If you start in the kitchen and go to the west, you will end in the
garden.
Hard Object Composite
The kitchen is to the north of the garage. The garden is to the west of the kitchen. The bedroom garage. There a bin containing a book the bedroom. There is a box in the garden containing no objects. Took a book from the bin in the bedroom. Went to the north twice. then went the west. Placed it.
The box in the garden con- tains book. The bin in the bedroom contains no objects
the south the
Task
We show how LMs can generalize in these ways and others by systematically fine-tuning them on classes of scenarios where they can learn rules. Across tasks, our reasoners deduce the final state of an environment given natural language descriptions of an initial state and actions taken on it.
3.1 Task Types
Containers. In the first task, which we call containers, we manipulate objects in various containers and ask the reasoner to track the state of the environment. The initial state of the environment is a random allotment of n_objects objects into r_containers containers. The names of these objects and containers are sampled uniformly and without replacement from lists of candidates. Examples of such candidates are given in the appendix. Once sampled and organized, the objects and containers are converted into a plain English expression describing their organization. Then, there a random manipulation of this initial state, where object is randomly taken from container and placed into another container. The task of the reasoning engine is to describe the final state of the environment.
For training, each scenario uses 2-8 objects and 2-3 containers, sampled uniformly. We construct sentences by sampling without replacement from uniform distribution of container names and object names and filling template accordingly. Our training set of object and container names comes from a proprietary linguistic dataset, which has data on commonness, part of speech, etc. of each word. We divide this dataset into nouns and verbs and subtract their intersection from both. We split the unique nouns into a train set (n=36566) and a validation set (n=12189) and a container set (n=9), the former two of which are to be used object names and the latter of which is to be used for container names during training. For each set (train and val), we find the 2000 most common and join the dataset on concreteness ratings from Brysbaert et al. [2 to take the 2000 most concrete nouns from each set. We also generate a list of random strings by sampling uniformly from single digit integers and lowercase English characters, ranging in length randomly from 5 to 10.
Navigation. In both versions of navigation, a natural-language description a map the environ- ment is provided to the reasoning engine. It is generated by sampling from a list of common locations including "kitchen", "garden", and others. These locations are then composed into a grid where they in wethh--outh-wwssttt reasoner is additionally given a starting point and a destination. The reasoner must provide a valid route from origin to destination. In the second task, called navigation result, require that the reasoner, given a starting point and a route, determine the location where they would find themselves in the map. For training, the maximum number of locations is 8 and the minimum is 3.
orientation to cach other. In the first task, called navigation ronte, the
Composite task. We are interested assessing the ability a reasoning engine to perform a task which combines multiple elemental types of knowledge. Specifically, can learner master two skills separately, and then leverage knowledge from both to perform the composite task better and more quickly than it would have by learning directly on the composite task? This task, called hard object,
3

---

is intended to be composition of navigation and containers. The reasoner is given a verbal map of the world and an action taken. In this action, an object is taken from its container, carried on route indicated by successive moves in cardinal directions, and placed. The reasoner must then describe the new state of the containers.
3.2 Generalization Types
Given our overarching interest large LMs' ability to yield reasoners that generalize well, we craft several types of experiments to probe the bounds of different kinds of generalization, namely:
Cardinality generalization. This tests a model's structural understanding of the domain. If reasoner is trained on scenarios with & objects, steps of navigations, rooms, etc. can it generalize to more than & objects, steps of navigations, rooms, etc.?
Object generalization. This is a semantic test of whether or not the reasoner can leverage prior knowledge of English generalize to new, never-before-sen objects. For example, if several different container training scenarios are composed of objects from different distributions of words (e.g. 2000 concrete nouns vs. 2000 most common nouns vs. 2000 randomly sampled nouns), which distribution of training scenarios results in the best model generalization to scenarios composed with new, previously unseen nouns?
This is another semantic test based on the idea that a reasoner that understands language will perform better "right" words than on "wrong" words. For example, on the container task, the model should perform well when nouns are objects and poorly when verbs random strings are. Intuitively, scenario such "The bin contains dethrone and transpose" makes less sense, and is statistically less likely, than a scenario such as "The bin contains a ball and a snake"; the naturalness of the scenario descriptions should positively correlate with generalization. We hypothesize that natural nouns will work best, since nouns are often sensible things to move from one container the other, and that arbitrary strings and verbs will both decrease performance. On the other hand, random strings have probably never been encountered by the model, and it might learn to simply copy whatever tokens are in specific places, treating strings as opaque IDs, symbols without meaning. Verbs should actively confuse the model, since a verb's 's linguistic role is structurally and semantically different than a noun's s; it is symbol not without meaning, but with the wrong meaning.
Part-of-speech generalization.
Reasonable phrasing generalization. Finally, what we replace the templates themselves? Instead of replacing the objects in scenario, replace the English scaffolding insert them into by mapping deterministically from each English word to a gibberish word composed of English morphemes but possessing no meaning (See|4.5). A reasoner for which language is meaningful will have a harder time adjusting to this task, while a reasoner for which language is simply a sequence of meaningless strings will struggle no more with this task than with its original English version.
3.3 Base Model and Training Details
On all experiments, we use T5's 3 billion parameter architecture, either fine-tuning a pretrained model or training from scratch. We do so on Nvidia Tesla V100 32GB GPUs with a batch size of 1 and learning rate of 0.003. We train a different reasoning engine for all three elemental tasks for 1000 total steps. After evaluating them on their respective tasks, we train them on the other base task for 1000 steps and eventually on the composite task for 1000 steps (3000 total steps).
3.4 Metrics
For each experiment, we gauge performance using three metrics. The first is exact equality of true final state. This is obviously the highest standard, it mimics perfectly the reasoning would expect a human to carry out after having learned the rules and format the dynamical system. The second metric is substring equality. We want to see how many individual statements from the predicted final state are contained in the true final state. Sometimes the reasoning is flawed, but on target (e.g. if the reasoner predicts, after moving hammer from a box to bin, that it is in both the box and the bin, it should be given partial credit). The third metric is the standard BLEU score, to how similar the sentences are at the individual word level.
Interpolation and Extrapolation. Throughout our experiments, we assess different kinds of gener- alization. The first, which we term interpolation, refers to testing a LM on new instances that are

---

drawn from the same distribution as the training set. For example, in the container task, we might test on scenarios that use already-seen objects and containers, but arranged in novel scenarios. The second, which we term extrapolation, refers to testing a LM on new instances that are drawn from different distribution than the training distribution. For example, in the container task, we might test on scenarios that involve new, never-before-seen objects, containers, or cardinalities.
Structural and Semantic Generalization. Finally, we distinguish between structural generalization, where we test things like cardinality generalization, and semantic generalization, where we explore new words, or new types of words.
Pretrained versus Tabula Rasa
4 Results
1.0
0.8
We now systematically test interpolation and extrapolation, assessing both semantic and structural generalization. We begin with several baselines, and then explore increasingly difficult tasks.
Pretrained
Tabula Rasa
0.4
Interp
SemExtrap
SemExtrap
rnterp
SysExtrap
SysExtrap
SemSysExtrap
SemSysExtrap
4.1 Comparison with Baseline Methods
0.0
600
Since our claims have to do with the inductive bias of large, pre-trained LMs, we first establish a wide baseline gulf between pre-trained models and those trained from scratch (tabula rasa). We train two such versions of TS-3B on the same training set (1000 scenarios with maximums of 8 objects and 3 containers and minimums of 2 objects and 2 containers) and then measure two kinds of both interpolative and extrapolative performance: systematic, meaning the number objects and containers, and seman- tic, meaning the words used for objects in reasoning scenarios (See 3.4 and 3.2 for more info). Thus, the two models' performance is compared on total experiments with 7200 examples each: Interp same number of containers and objects and same words used for object names (as training set); SemExtrap same number of containers and objects (as training set), different words used for object names; SysExtrap- minimums of 4 containers and 10 objects and maximums of 5 containers and 19 objects, same words used for object names (as training set); SemSysExtrap minimums of containers and 10 objects and maximums of 5 containers and 19 objects, different words used for object names (than training set); The average BLEU scores for each model checkpoint (taken every 100 steps during finettning/training)) are displayed in Figure [ pre-trained language models that have been fine-tuned wildly outperform language models trained from scratch.
N Steps Finetuning
Figure 1: Model performance on 4 ex- periments over the course of fine-tuning Compare solid lines (off-the-shelf pre- trained models) and dotted lines (tabula rasa models) of the same color.
These baselines make clear that
4.2 Containers
We now begin our primary experiments, starting by testing cardinality generalization. We train on scenarios with a maximum of 8 objects and 3 containers, and validate structural generalization on scenarios with up 19 objects and 5 containers. The results are shown in Figure The model is able to correctly predict the exact string majority of the time even well outside the training domain. This kind of generalization could only be reached by grasping, to some extent, rules like those used when enumerating long English lists. This is one way in which the inductive bias of LMs aids in the reasoner's 's good performance.
We now test object generalization and part-of-speech generalization. Here, we train models with scenarios generated by parameterizing our templates with various sets of nouns, and then testing by parameterizing with different set of words. The results are shown in Table [ Each row represents a different training set; columns represent performance on different test sets. We tested four different training sets: "all" represents nouns sampled uniformly from our set of all 36,566 nouns. "2k common" " represents the 2000 most common nouns, "2k concrete" represents the 2000 most concrete nouns; "2k random" represents 2000 randomly sampled nouns. There are 3 groups of columns, representing different validation conditions. "Training words" represents a validation condition where the reasoner was tested on new scenarios that used the same set of nouns (eg, different mix of
5

---

Training nouns
Validation nouns
1.0
Numbor of conturate 6o
Bleu Scores
% Exact Matches
Numon of consenees cz
% Substring Matches
Numbor contaihens 10 C
6
objects
Figure 2: The blue box encompasses the training domain. Each cell is the average of performance 100 generated instances for each combination of n_objects and p_containers. There is obvious drop off higher in the number of objects, but this is truncation issue due to long sentences (note that substring scores don't 't degrade nearly much).
Table Detailed results on the container task, showing various aspects of interpolation and extrapolation, both to new scenarios, types of words, and new surrounding texts.
Aug Emtrt Equality
0.8
04
training words
validation words
pos
common
common
0.
bleu
10
all
0.95
0.98
0.99
0.95
0.95
0.98
0.99
0.99
0.95
0.82
Avg. BLEU Score
2k common
0.93
0.96
0.95
0.93
0.93
0.96
0.96
0.99
0.95
0.81
2k concrete
0.93
0,96
0.97
0.93
0.93
0.96
0.96
0.99
0.95
0.78
0.0
Nours
2k random
0.91
0.94
0.95
0.91
0.91
0.94
0.95
0.99
0.95
0.81
Verba
exact
0
Pandom
0.79
0.88
0.91
0.79
0.78
0.87
0.90
0.66
0.44
20
200
2000
2k common
0.60
0.71
0.65
0.59
0.60
0.69
0.67
0.83
0.63
0.37
N Nouns Model
2k conerete
0.60
0.68
0.75
0.59
0.59
0.69
0.71
0.86
0.60
0.31
Trained On
2k random
0.56
0.61
0.68
0.55
0.55
0.60
0.64
0.87
0.63
0.40
Figure 3: Models trained on concrete nouns were most confused by verbs and random strings.
substring
0.97
0.96
0.95
0.97
0.97
0.96
2k
0.90
0.85
0.83
0.88
0.95
containers and objects, or a different number of them). "Validation words" represents a condition with new scenarios using never-before-sede words of specific type. "New POS" represents words that are an entirely different part of speech, or random words. The general conclusion is that, regardless of the metric used, it is best to train on the largest non-specific set of nouns possible. Doing gives good generalization across a wide variety of alternative words, including verbs and random strings.
We now explore the distributions of the training nouns. Many of the words passed between containers were nouns that would not be passed in between containers, like "year", and so we conducted an experiment where we trained on the 2000 most concrete nouns, random subsample of 200 of those nouns, and As can be seen in Figure 3 the model trained on sensible nouns gets most confused at verbs and random strings, relative to peers. This suggests that the distribution of very concrete nouns farther from the distribution of verbs than the larger distribution of concrete nouns.
the most sensible nouns to moved between containers, like "marble" or "mouse".

---

Navigation Route
Navigation Result
Distribution of test data
1402
Number of Rooms
1202
1902
10
200
Number planning
Figure 4: Validation performance the navigation route and navigation result tasks. Entries represent exact string equality. Training regime is outlined in blue box. See text for details.
4.3 Navigation
We now turn our attention to the navigation and navigation route tasks. this section, we use exact string accuracy as our metric, as both substring and BLEU scores ignore the (critical) sequential nature of the predictions.
4.3.1 Interpolation and Extrapolation
Our primary results are in Figure 4 For both the navigation route and navigation result tasks, trained on distributions of maps that contained between 3 to 8 total rooms; this resulted in total path plans that were
steps long.
In the middle, we see results for the navigation result task (where the model must predict the result of several interesting phenomena. Overall, the model does
specific sequence of actions). Here, a good job of making accurate predictions, with a 79% marginal accuracy over the training regime. There two situations where the model performs especially well: the first column represents tasks where there only a single step in the route; the model shows strong ability to extrapolate to any number of rooms. The second case the "diagonal" in the upper-right. This represents situations where the map a linear chain, with very few junctions in the map. In this case, the model also shows number of rooms, and a number
strong ability generalize to situations involving of steps. It is unclear why the model has such trouble with two-step planning problems, although we hypothesize may have something do with the training distribution, discussed later.
In contrast, on the left of Figure 4 we see results for the navigation route task. Here, we see similarly strong performance in the training regime, with 82% marginal accuracy. However, extrapolating to new situations shows mixed results: the model is easily able to extrapolate to any number of rooms, and like the navigation result task, it performs especially well in the case of single-step planning. However, it seems to be completely unable to generalize to new numbers of steps needed in the planning process.
What accounts for the difference? The biggest difference between the navigation route and navigation result tasks is the format of the output: in the navigation result task. the model always outputs single word (the room name), regardless of the complexity of the map or the plan. In contrast, on the navigation route task, longer plans require longer output. The model seems struggle with having to generate sentences that are longer than any it has seen before. While we have used exact string equality in all of the tests reported here, note that longer routes, sometimes the model's output would a substring of the correct output, as in the example below, which is correct, but missing the final step:
Target: to the seat, then to the weat, then to the north, then to the north, then to the north, then to the norsh
Prodiction: west, west, north,
north,

---

Average Exact Score
Average Substring Match
1.0
%
0.6
0.6
Cont Cont-Nav Havd-ooOOOO
ContlaxsO50--Hrdd00
0.4
0.4
HardObj
...
HardObj
0.0
500
1000
2000
2500
3000
0
500
1000
2000
2500
3000
NTaiiiing Steps
N Training Steps
Figure 5: Confidence bands represent one standard deviation above and below mean performance under each metric. The composite learners exhibit better sample efficiency and higher absolute performance ceiling than the learner dedicated exclusively to the composite task.
Uniform path length
Uniformly selected rooms
(A)
Accuracy
(B) Training distribution
(C)
Accuracy
(D) Training distribution
Number of rooms
646
64
3.2
Steps of planning
Figure 6: Extrapolation ability function of training set distribution. See text for discussion.
4.4 Compositional Reasoning the Composite Task
Finally, we tested the hard object task, where the reasoner must consider objects, containers, and maps. To explore this, test four conditions. We train learner exclusively the task (HardObj). We also train a learner first on the navigation task for 1000 steps, then on the container task for 1000 steps, then on the hard object task for 1000 steps HarrD).. We also reverse the order of navigation and container pre-training pretraining, with 2000 steps of 50/50 mixture of sentences drawn from both tasks (ContNavv5500- HardObj). For each regime, we trained 10 models and averaged their performance on a test set of 5000 held-out examples.
and we test a random mixture of
The results in Figure are surprising: the models that learned first on other kinds of tasks learned the composite task more quickly and achieved better ultimate performance. Like curriculum learning, this seems to indicate subskills can persist and be ported to superskills for improved sample efficiency. Instead of fine-tunine finettnnig on a single hard task (and thereby gaining proficiency only that task), better strategy may be to fine-tune on a wide variety of elemental tasks, and focus more on combining them, thereby potentially solving not just one, but a combinatorially large number of complex tasks.
4.4.1 Importance of Training Set Distribution
The T5 model seems unusually sensitive to the training set distribution. exgurrrs this. In our first attempt at creating training distribution for the navigation tasks, we randomly created map with a uniformly selected number of rooms, and then randomly selected source and destination rooms. This resulted in the performance shown in the right-hand panels (C and D) of Figure 6 Panel (D) shows most of the training data is concentrated in the upper-left corner, meaning short paths in small maps. As result (panel (C)), the model was almost completely unable to model length 4 or length 5 plans, because they rarely appeared in the training set.
An alternative is to sample desired path length uniformly, and then generate map, source, and destination with that length. This results the training set distribution shown in panel (B), which concentrates many more examples longer paths in larger maps, but with fewer examples of two-step plans. The resulting accuracy shown in Panel (A). Here, we see that accuracy greatly improved for four- and five-step plans, although two-step accuracy suffers. However, even the

---

Avg. Substrings Match
Avg. BLEU Score
Avg. Exact Equality
1.0
1.0
1.0
0.8
0.8
0.8
0.6
0.6
0.6
%
0.4
0.4
0.4
gibberish
0.2
0.2
0.2
nouns
0.0
0.0
10
10
10
Checkpoints 100)
Checkpoints (n-100)
Checkpoints (1-100)
Figure 7: The advantage of English scenarios over equally consistent reasoning scenarios in an invented language evidence for the aid of language models' inductive bias.
carefully constructed training distribution with uniformly sampled path lengths does not induce generalization paths longer than 5 steps, as shown in Figure Understanding this phenomenon is an important direction for future research.
4.5 Does all that reading really help?
At the heart of our paper is the idea that natural language provides useful inductive bias. To explore this directly, we modify our object-containe task the following way: We devise a one-to-one mapping from each word in our sensible English templates to words in invented gibberish. Since the mapping is one-to-one, the English templates and the gibberish templates are identically structured but with substituted words. For example, the word "The" is mapped the gibberish word "Xrq". the word "contains" to "sixnqkxb", etc. Noun slots in both templates are still filled with the original English nouns (i.e. substituted with gibberish). Figure 7 shows the results: it is harder for T5 to master the gibberish domain. If there were no inductive bias of English-traince models aiding our learners on English tasks, then the mastery of these two grammars would grow at the same pace, since they are just as structured as one another and thus presumably as predictable and learnable as one another. This a bit of evidence favor of the notion that the inductive bias of large LMs does, in fact, aid in learning quickly and generalize well in several different senses.
Ethical and Risk Considerations
Since this paper studies what are essentially toy tasks, we don't consider the risks for our methods be particularly high. However, we believe in the eventual application of these models in real- world domains (e.g. robotics and planning), and this work contributes to needful preliminary understanding these models' behavior, since they are difficult to control and predict. We urge applications of these models to take into account this general risk along with work such as ours which use language models in high-risk domains (consider
helps characterize it. It is dangerous surgical robot or planning agent for defusing explosives) without extensive understanding of their interpolative and extrapolative generalization.
Conclusions and Future Work
The central goal of this paper has been to investigate if connectionist LMs can learn something akin to symbolic rules that generalize in "natural" ways. We have shown strong performance on suite of reasoning tasks, including tasks that we consider to simple-an object manipulation task and navigation
and more difficult-a composite task that combines the other two. On the container task, we show near perfect performance on the training domain and considerably far outside of it and an eventual marked drop-off in performance. On the navigation task, we show differing patterns of generalization on different subtasks, with slightly lower performance than on the container task. We have also demonstrated some of the boundaries of generalization by identifying several different kinds of generalization and measuring them. We do not claim to have explored all the axes of generalization possible in the study of these language models, but rather shown how the generalization ability of

---

LMs can be probed. But these results are exciting because the models perform so well outside of the training domain that it seems as if some general. symbolic rules have been learned.
Beyond generalizing just to new examples on learned tasks, we have demonstrated that these models aided in the performance of complex tasks by learning first elemental tasks. This hints the prospect of learning difficult tasks more quickly and better by learning more simple tasks first.
We have additionally argued that natural language provides a powerful inductive bias for symbolic tasks by comparing on scenarios across different, but equivalent grammars, one in natural language and one in an invented language. Intuitively, learning the distribution of language that describes the world helps a leamnrr to understand the distribution of that world itself. Thus. it is plausible that LMs might be effective ways to provide autonomous agents with priors and world models.
Implicitly, we have demonstrated that connectionist architectures are capable of strong performance on some classically symbolic tasks. This is done in light of recent calls to incorporate guiding principles of symbolic Al into connectionist models; possess at least some of the rule-learning and generalization that humans possess, opposed to the mere ability to recognize patterns and interpolate over a well-explored training domain.
exciting that language models seem to
References
[1] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8):9, 2019.
[2] Jacob Devlin, Ming-War Bing-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 10t::8810.000 10.04805, 2018.
[3] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: arXiv:1910. 10683, 2019.
[4] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
[5] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998 6008,0088800 2017.
[6] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. URL https://s3--s- -ussssst-----st22. amazonaws. com/openai- understanding paper. pdf. 2018.
[7] Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases? arXiv preprins arXiv:1909.01066 2019.
[8] Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. Comet: Commonsense transformers for automatic knowledge graph construction. arXiv preprint 10Xiv:1906.05300 2019
[9] Zied Bouraoui, Jose Camaaho-Collados, and Steven Schockaert. Inducing relational knowledge from bert. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7456-7463, 2020.
Collados,
[10] Terry Winograd. Procedures as a representation for data in computer program for understand- ing natural language. Technical report, Massachusetts Institute of Technology, 1971.
[11] Richard E Fikes and Nils J Nilsson. Strips: A new approach to the application of theorem proving to problem solving. Artificial intelligence, 2(3-4):189 108, 1971.
[12] Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines, 2014. URL http:
//arxiv. org/abs/ absaaba/11111 /ete//110...0 5401
10

---

[13] Alex Graves, Greg Wayne. Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska- Barwiñikka, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, Adrià Puigdomenech Badia, Karl Moritz Hermann, Yori Zwols, Georg Ostrovski, Adam Cain, Helen King, Christopher Summerfield, Phil Blunsom, Koray Kavukcuoglu, and Demis Hassabis. Hybrid computing using neural network with dynamic external memory. Nature, 538(7626): 471-476, 2016.
[14] Peter Clark, Oyvind Tafjord, and Kyle Richardson. Transformers as soft reasoners over language. arXiv preprint arXhv:2002.05867 2020.
[15] Pratyay Banerjee and Chitta Baral. Knowledge fusion and semantic knowledge ranking for open domain question answering. arXiv preprint arXiv:2004.03101, 2020.
[16] Leon Weber, Pasquale Minervini, Jannes Münchmeyer, Ulf Leser, and Tim Rocktäschel. NI- prolog: Reasoning with weak unification for question answering in natural language. arXiv preprint arXiv: 2019.
[17] Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698 2015.
[18] Rich Caruana. Multitask learning, Machine learning, 28(1):4444 1):41-75, 1997.
[19] Xiaodong Liu, Jianfeng Gao, Xiaodong He, Li Deng, Kevin Duh, and Ye- Yi Wang. Representa- tion learning using multi-task deep neural networks for semantic classification and information retrieval. 2015.
[20] Jason Phang. Thibault Févry, and Samuel R Bowman. Sentence encoders on stilts: Supplemen- tary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.0100 2018
[21] Mare Brysbaert, Amy Beth Warriner, and Victor Kuperman. Concreteness ratings for 40 thousand generally known english word lemmas. Behavior research methods, 46(3):904-911, 2014.
References
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We demonstrate the claimed inductive bias across suite of experiments in Section
(b) Did you describe the limitations of your work? [Yes] In Section G we make explicit the obvious truth that we have not demonstrated a general ability of LMS learn anything, nor have we done all the probing that could be done of LMs, lest reader should conclude from the models' impressive performance on toy tasks that they could
learn any reasoning task and perfrom this well.
(c) Did you discuss any potential negative societal impacts of your work? [No] The risks associated with our paper are not unique to it, and are probably shared by all of deep learning. They include environmental risks from intensive computation required for training neural networks and the capacity for eventually automating humans whose skills deep learning could learn automate. Our tasks are, the end of the day, toy tasks, and our paper makes no advance significant enough to threaten the livelihood of human being anytime soon or increase the environmental footprint of training neural networks.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We have.
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]

---

(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as URL)? [Yes] We include the source code and appropriate raw data files in our supplemental material submission, and we outline the data generation procedure, the evaluation protocol, the training regime, and everything else necessary for reproduction either in the main body of the paper or in the appendix. The only exception to this the proprietary dataset mentioned in Subsection 3.1
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Subsection 3.3 and the appendix.
(c) Did you report error bars (c.g., with respect to the random seed after running exper- iments multiple times)? [Yes] We did this in Figure S but we didn't 't do this in other figures where the difference performance was more clear.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Subsection 3.3
4. If you using cxisting assets (e.g., code, data, models) curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite the creators of T5 Section I and the formulator of the concreteness ratings in Subsection [.1] but the proprietary dataset is not public, and so we didn't cite the creator. We created all other assets.
(b) Did you mention the license of the assets? [Yes] We mention the only license we know of (that belonging T5) in the appendix.
(c) Did you include any new assets either the supplemental material as URL? [Yes] We include source code to regenerate our datasets.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No] The pretrained models were generated on huge web scrapes of material in the public domain, and it would be impossible to acquire consent for the use of this data.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No] Language models are known to have privacy leaks, and the only extent to which our methods might lead to privacy leaks is shared by all research htngg/iieettuingg language models, we didn't 't feel the need to document this risk.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "fFbE9__G81a", "reviewer": "Reviewer_gnNv", "summary": "This paper investigates whether pre-trained language models (T5 in this case) learn priors which support symbolic reasoning tasks in the process of optimizing their purely linguistic objective.  The performance of fine-tuned pre-trained LMs is evaluated on two symbolic reasoning tasks -- object location across a number of boxes, and a room navigation task -- and compared with identical models trained from scratch.  The results touch on a number of topics, but generally show (1) good reasoning accuracy when starting from the pre-trained t5 model, (2) abysmal performance by the from-scratch models, (3) a good degree amount of generalization performance for t5 to reason beyond the fine-tune scenario, and (4) that learning subtasks improves performance on a compositional task.\n", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "There's a lot to like in this paper.  The presentation and writing quality overall is just fantastic.  Not only a clear presentation, but a great style that makes the reading really enjoyable.  The authors have also been very thorough in listing the considerations behind each experiment, and approached each from multiple angles to get rid of confounds.  It's clear that a lot of thought lies behind each one.\n\nMy main criticism, and a crucial one, is that the experiments don't go far enough to show what the authors purport to show.  This boils down to a few things:\n\n1) Unclear separation between language understanding and world knowledge.\n\nThe story laid out in the introduction, especially paragraph two, is one of text mirroring patterns in the real world, and the potetial for LMs to encapsulate this knowledge.  The authors provide a long list of potential types of world knowledge, and whether LMs capture any of these is very exciting and a very timely research question.  Just to underscore that this sort of information is central to the paper's argument, L25, \"-If that is so-, then they ought to provide a useful inductive bias in learning to perform symbolic reasoning tasks that mirror real-world tasks\".  I'll call this \"world knowledge\" for the remainder of the review.  Those statements really set the stage for the tone of what this paper is claiming to show, it follows from the reasoning in L7-8 of the abstract, etc.  Suffice to say, from that point I would expect that the paper ends by conclusively showing that this world knowledge is there in their chosen pre-trained LM.  And I think this is where it fails.\n\nDesired interpretation of results: Container task.  Pre-trained model performs vastly better on extrapolation and overall when compared to from-scratch model, because it has embodied world knowledge, (maybe object persistence, containment, from the introduction list).  It shows good generalization ability because it T5 has induced reusable structures of reasoning / world knowledge.  It performs better with concrete nouns, then nouns, then verbs, because utilizing these objects in this manner does not make sense.\n\nAlternate interpretation of results: Container task.  Pre-trained model performs vastly better when compared to from-scratch model because it has some rough representation of the word classes of these words.  It generalizes better to new words because it has similar representations of those words.  It shows good generalization ability because it can do so on top of well defined word class representations (completely apart from their role in reasoning).  It performs worse on verbs because it has trouble refining representations of ungrammatical sentences, and reasoning on top of non-sensical sentences suffer.\n\nIn the first interpretation, I'm really interested to further understand these induced world knowledge patterns more and I really want to accept this paper.  In the second interpretation, it's just shown that pre-trained LMs have really good representations of basic linguistic categories, and that fine-tuning for all sorts of tasks is hugely effective, something the community is already quite aware of.  It's still interesting to see applied to these bAbI-like tasks, but then again, that's a different sort of paper.  In that setting, I would have liked to have seen a comparison with those models and more practical advantages/disadvantages/failure cases described.  \nAnd to the second interpretation, where basic linguistic knowledge accounts for everything,  I don't see any evidence here that would rule this out.  That's not to say that the hypothesis is wrong -- indeed, I would suspect there is a lot of world knowledge in there -- but I don't believe the authors have shown that conclusively.  I think the tasks chosen here make it difficult to really talk about real world knowledge.\n\nThe authors sometimes waffle between these two.  For instance, at the end of section 4.5, the English->gibberish conversion, the authors state that their experiment provides \"evidence in favor of the notion that the inductive bias of large LMs does, in fact, aid in learning quickly and generalize well in several cases\".  In this instances, the story is muddled, because we know that a general inductive bias in large LMs helps learning in many areas, it's basically the state of NLP at the moment, and so it needs no supporting evidence.  It's whether the inductive bias is based in real world knowledge that is unique to this work and pushed for in the introduction, and these experiments do not separate whether the advantage of the English model comes from linguistic knowledge or world knowledge.\n\nIt actually strikes me perhaps as evidence of point (3) below, where the gap between this lines correlates roughly to the combined linguistic and world knowledge the LM learns, while the reasonably high performance of the gibberish model is basically a testament to these training procedures getting the weights in a good position for transfer to unrelated tasks.\n\nAnother possibility of more concretely discovering evidence of world knowledge would be probing the representations in a targetted way.  Combining that with the sorts of experiments provided could strengthen the overall impression of the work.\n\n2) Poor Baselines\n\nIn terms of experimental setup, of course it is tough to get at exactly what types of practical knowledge / inductive bias (apart from linguistic knowledge) these models contain, but I think some problem stem from the chosen baselines.  The authors use the same T5 model for the pre-trained model and for the from-scratch model.  This was a terrible idea in my opinion.  No reader should accept a 3 billion parameter model trained for 1000 steps to be a suitable baseline for comparison -- it is clear that this model was not given any chance to succeed.  I'm curious how smaller-parameter variants fair on the from-scratch setting.  I think it's quite clear that 3B parameters is not required for these sorts of tasks.\n\nSpeaking of similar tasks, the tasks are admittedly close to the FB bAbI in spirit, however, going back to that original work, there was not so much difficulty in achieving good results with models that we would considering quite basic by today's standards.  Why?  Presumably baseline model size and over-capacity.  But also thinking more generally, there were previously good results on bAbI tasks with simple memory-augmented models on top of word embeddings.  Is it similarly sound to claim that word embeddings had strong inductive biases that mirrored the world outside of their linguistic roles?  Of course it's also a question of training size and setting the tasks up to test generalization, but the rationale seems similar to the argument of this work.\n\nI would have also liked to have seen some other models which are maybe capable of inducing linguistic knowledge relevant for the task, without any serious possibility of learning things like physics or spatial relationships, other examples of these properties, etc.\n\n3.) Large pre-trained LMs can improve performance on unrelated tasks\n\nAn important point of comparison is the \"Pretrained Transformers as Universal Computation Engines\"[1], which was not mentioned in this paper, though it appeared on arxiv only a few weeks prior to the submission deadline.  Inn summary, the authors showed that pre-training on large text corpora provide a significant learning advantage when applied to completely unrelated domains (via fine-tuning).  \n\nWhen considering this submission in light of this finding, it raises the question of whether the conclusions in this work are true / whether pre-trained LMs have induced symbolic reasoning knowledge, or merely a result of more universal organization properties in the network.  It would perhaps be interesting to compare these results with models pre-trained on an equally large section of another language and test in English, especially if preposition-style relationships were expressed in different ways (since they seem fundamental to both tasks).\n\nRegardless of the answer to these questions, this is an important point to discuss.\n\n[1]\n@article{DBLP:journals/corr/abs-2103-05247,\n  author    = {Kevin Lu and\n               Aditya Grover and\n               Pieter Abbeel and\n               Igor Mordatch},\n  title     = {Pretrained Transformers as Universal Computation Engines},\n  journal   = {CoRR},\n  volume    = {abs/2103.05247},\n  year      = {2021},\n  url       = {https://arxiv.org/abs/2103.05247},\n  archivePrefix = {arXiv},\n  eprint    = {2103.05247},\n  timestamp = {Mon, 15 Mar 2021 17:30:55 +0100},\n  biburl    = {https://dblp.org/rec/journals/corr/abs-2103-05247.bib},\n  bibsource = {dblp computer science bibliography, https://dblp.org}\n}\n\n\n\n\n\n\nQuestions:\n\nDoes the model size matter?  I'm curious how small a T5 model has to be before it has significant effects on the reasoning task, especially if additional fine-tuning layers were added (of equal size for all T5 variants), and thus isolate the task-specific weights from the \"induced world knowledge\" weights.\n\nThe paper might benefit from more specificity in what the world knowledge the authors feel is relevant here.  For each, they could also describe whether previous NLP work has shown these properties exist in pre-trained LMs or in their necessity in other toy tasks.\nAnd for instance, to choose one from the introduction that may be relevant, \"material properties\"?  For whatever property the authors think relates to their chosen experiments, could they discuss how it is necessary for the task?\n\nL153: for fine-tuning, is this purely using the T5 weights or are any additional layers/heads added to the model?\nL155: what constitutes a step?  A single training example?\n\n\n\nPresentation:\n\nPersonally I find the use of \"symbolic reasoning engine\" to describe a fine-tuned T5 model a failed attempt at inflating the sophistication of the method.  At the very least, it prevents readers from more immediately latching on to the specifics of the model and therefore the study.\n\nFig 1., very difficult to spot the various overlayed lines, especially in a printed version.\n\nIn L19-20, The examples of tasks where pre-trained LMs achieve SoTA accuracy do not require patterns of ``real-world'' structure described in L22.  Where is the citation or evidence that generated text reflects the latter forms of knowledge?\n\nL65, BLEU score is used to see how similar sentences are on subword levels?  But BLEU is a word-level metric?\n\nL92, how are selected objects arranged into sentences?  What kind of diversity is there in these generated sentences or templates?\n\nL102, most common... nouns / and join the dataset -- this description was a bit clunky\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "Fqli0Wb1kzN", "reviewer": "Reviewer_KcKG", "summary": "This paper takes a closer look at the ability of language models to perform reasoning based tasks such as tracking states of entities and answering navigation based questions. The main question is: Can these models generalize outside of the training distribution, thus exhibiting the ability to learn underlying rules instead of learning superficial correlations that only work on in-distribution data. This question is studied through several different generalization splits. From results, we see some evidence that pre-trained models are able to generalize outside of the training distribution though in some cases, it’s inconclusive, (and I elaborate on why below). ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: While I think that the exact settings in this work are somewhat original, Clark et al. 2020 (RuleTaker) also  consider reasoning abilities of transformers on synthetic tasks and look at various generalization abilities. Similarly, Banerjee et al. 2020 also look at how well transformers can be trained to do various reasoning tasks in blocks worlds. In the authors’ opinion, what are some major differences between these works and this paper?  \n\nExperiments: While I really like this direction, I think some of the experiments are inconclusive in determining whether the model has truly learnt the underlying rule. \n- For example, in the containers experiment the model just needs to track the state of 2 containers (which can be easily detected based on the language context) to correctly answer. So, even if the model can generalize to more objects and containers, it is not really impressive since the model can learn to ignore all other containers except the two containers whose states were altered. Indeed, if we look at the navigation results, we see poorer extrapolation since the task is harder. \n- In Section 4.4, we cannot immediately conclude that the model is leveraging previous knowledge. The model could just as well be learning superficial features from the container and navigation tasks that help on the hard object tasks. One way to check for this confounder would be to just do some steps of pre-training on the Nav and Cont tasks and see if that by itself explains the fast learning on HardObj. If not, we can then conclude that perhaps the model is leveraging previously acquired knowledge to do well on a composition.\n\n- I really liked the experiment in Section 4.5. Although, I wonder if the decrease in performance has a simpler explanation like gibberish words like “sixnqkxb” contain more sub-words and are hence harder to reason / track. \n\nQuality: I think the paper is decently written. Some style suggestions:\n- Table-2 could be converted into a barplot (moving numbers into appendix) since it’s really hard to parse so many numbers. And it’s generally good to bold the best numbers for a quick read.\n- In Figure-7, it is unclear what the x axis refers to.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "88JFRyvQiC", "reviewer": "Reviewer_hX8A", "summary": "This paper aims to explore whether the inductive bias of pre-trained language models can support symbolic reasoning tasks. Three different types of tasks are designed for the above objective, including a container-based task, a navigation task and a composite task. In order to verify the generalization ability of the model, the paper also designed different kinds of generalization for probing, including cardinality generalization, object generalization, POS generalization and reasonable phrasing generalization. T5 is fine-tuned and evaluated on these tasks, with some interesting findings observed, which I think are the key contribution of this work: (1) for container-based task, T5 shows good generalization and prediction capability; (2) for navigation task, T5 can also do a good job but when the number of inference steps changes, its performance will drop. (3) for composite task, T5 can do a good job  by learning in a curriculum learning way.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The objective of the paper is clearly clarified. The writing is easy to follow. The findings in the experiments are interesting. It is a good paper to study how well pre-trained LM can perform on reasoning-required tasks and its generalization capabilities on different aspects. I appreciate this work but think the datasets built within this paper is limited to specific domains and scenarios. It will be better if the paper can cover results on other open domain datasets as well, such as commonsense QA, mathematical reasoning, etc. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "no negative societal impact", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5 hours"}, "extra_text": {}}, {"review_id": "2cw6WK-gV-k", "reviewer": "Reviewer_VwQ8", "summary": "In this work, the authors investigate if large scale pre-trained language models (such as T5) can provide inductive biases that are useful for solving language-based symbolic reasoning tasks, and furthermore, generalize to unseen settings. \n\nFirst, the authors define four types of tasks. While all the four tasks are formulated as sequence generation given prompts (prefixes), they require different kinds of reasoning:\n\n1. Container. The prefix text describes an initial states of the world and a sequence of transitions, the target sequence needs to describe the ending states.\n\n2. Navigation Route. The prefix text describes a map as well as two positions in the map, the target sequence is the navigation path from one position to the other.\n\n3. Navigation Result. The prefix text describes a map, a starting position, and a navigation path, the target is the ending position.\n\n4. Composite Task. A mixture of container and navigation tasks. \n\nThen, the authors define a set of experimental settings targeting the generalizability of models, from a variety of aspects such as map sizes, object/container numbers, reasoning steps required as well as some linguistic properties (e.g., replacing nouns with words with other POS tags, or even made-up words).\n\nThe authors provide plenty of experiments, suggesting the pre-trained language models indeed have capability of learning and to some extent generalizing symbolic rules via language-based tasks.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Synthetically generated data\n\nJust to be transparent. I was one of the reviewers of this paper at ICML 2021. The authors have described in the \"submission history\" field about lacking of baselines. In addition to that, there were some ICML reviewers devalued this work because the used tasks are synthetically generated (and thus not natural language).  \n\nI want to emphasize my point that **synthetically generated tasks are equally valuable**. \n\nI understand and agree that the ultimate goal is to model and perform textual reasoning on real world data. However, before stepping into natural language data, I believe it is meaningful to have synthetic environments that are controllable, helping researchers to better understand the problem. Although the tasks introduced in this work use templated language, it is clearly shown in the figures that they are not oversimplified (there are areas in the heatmaps where the models fail to generalize). It might make less sense to use natural language data when strong pre-trained LMs such as T5 still have trouble solving/generalizing on toy-ish tasks. Because otherwise, it might be difficult to analyze, for example, which part of the task is the model struggling on. Furthermore, there are many examples in the NLP communities that use synthetic datasets (templated language) as a starting point of certain research directions, such as bAbI (Weston et al., 2016), TextWorld (Côté et al., 2018) and many more.\n\nOne potentially interesting thing to try might be, back-translating (e.g., EN -> DE -> EN) the teamplated descriptions, and investigate to what degree the language models generalize on this dimension of \"naturalness\".\n\n## Overall comments\n\nThis work provides a set of well-designed experiments with convincing results that explore an interesting and important direction. \n\nSince the flourishing of the large scale pre-trained language models, SOTA scores of neural models on various NLP tasks have increased by a lot. On some tasks, some researchers believe neural models have achieved human-level performance, while others argue that neural models may be somehow exploiting biases and trivial cues injected into the data or model unconsciously, rather than really doing reasoning. \n\nTo that end, the community do require works that help us understanding what and why such language models can learn, and to generalize. This work falls into this category. \n\nAmong the experiment designs and presentations, I especially like the heatmaps, which clearly show how the model gradually loses its generalizability when certain dimension of task becomes more difficult. \n\n\n## Typos and minor things\n\n1. L217: There are 3 groups of columns\n\n## References\n\n1. Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks. Weston et al., 2016.\n\n2. TextWorld: A Learning Environment for Text-based Games. Côté et al., 2018.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Since the current version of the paper uses a set of synthetically generated toy tasks, I do not see immediate negative societal impacts. However, because the work relies heavily on large scale pre-trained language models, which training data is difficult to control. The author should definitely discuss the potential negative effect or harm these techniques may bring in the revision.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper has certainly been put through the ringer of the "moving goalposts" of successive conference reviews, and for that I have a lot of sympathy (although would not make a decision on the basis thereof).

This paper presents an in-depth study of the ability of large pre-trained language models to generalise well on symbolic reasoning tasks, i.e. verifying whether some of the knowledge distilled through the pre-training objective(s) presents a useful inductive bias which aids generalisation in such downstream tasks which have a more structured nature than, say, RTE or QA.

The reviewers concur that the paper is well written, and presents interesting results. Given the interest the community has in this class of models, I concur that this is an important and timely contribution. Reviewer VwQ8 (rating 7) writes in strongest support, and in no small amount of detail, clearly seeking to champion the paper for publication. From reading Reviewer hX8A (rating 6) I could not tell why the rating was not higher. Reviewer KcKG (rating 6) is mildly supporting the paper, while presenting some concerns about whether the experiments are conclusive. I would have liked to read their thoughts following the authors' detailed response, but sadly it seems not to have been forthcoming. Finally Reviewer gnNv (rating 3) wrote at significant length regarding their concerns about this paper, and engaged in what I think must be the most detailed discussion I've witnessed during this round of area chairing.

Academically speaking, I find myself agreeing with a lot of objections and concerns Reviewer gnNv raises. The discussions with the authors here makes for a fascinating read, and I am not clear where the reviewer landed in terms of revising their assessment. I think the fairest way to view the outcome of this conversation is that there are fundamental difficulties with evaluating the sort of question the paper seeks to address, and in a sense the concerns levelled here are a statement about these difficulties more than they are a statement about the appropriateness the authors took, and the rigour thereof. Perhaps the paper is worth publishing on the grounds of its ability to generate such discussion alone.

Taking these reviews holistically, I have formed the strong opinion the paper should be accepted. It generates debate and controversy, but for the right reasons, and attempt to bring further light to the capabilities (and limits) of large pretrained language models. I would hope that the authors could, in as much detail as possible, incorporate their response to reviewers KcKG and gnNv in the final draft of the paper, so that the readers may benefit from these conversations (albeit indirectly).
Decision: Accept (Poster)