Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper takes a closer look at the ability of language models to perform reasoning based tasks such as tracking states of entities and answering navigation based questions. The main question is: Can these models generalize outside of the training distribution, thus exhibiting the ability to learn underlying rules instead of learning superficial correlations that only work on in-distribution data. This question is studied through several different generalization splits. From results, we see some evidence that pre-trained models are able to generalize outside of the training distribution though in some cases, it’s inconclusive, (and I elaborate on why below). 
main_review: Originality: While I think that the exact settings in this work are somewhat original, Clark et al. 2020 (RuleTaker) also  consider reasoning abilities of transformers on synthetic tasks and look at various generalization abilities. Similarly, Banerjee et al. 2020 also look at how well transformers can be trained to do various reasoning tasks in blocks worlds. In the authors’ opinion, what are some major differences between these works and this paper?  

Experiments: While I really like this direction, I think some of the experiments are inconclusive in determining whether the model has truly learnt the underlying rule. 
- For example, in the containers experiment the model just needs to track the state of 2 containers (which can be easily detected based on the language context) to correctly answer. So, even if the model can generalize to more objects and containers, it is not really impressive since the model can learn to ignore all other containers except the two containers whose states were altered. Indeed, if we look at the navigation results, we see poorer extrapolation since the task is harder. 
- In Section 4.4, we cannot immediately conclude that the model is leveraging previous knowledge. The model could just as well be learning superficial features from the container and navigation tasks that help on the hard object tasks. One way to check for this confounder would be to just do some steps of pre-training on the Nav and Cont tasks and see if that by itself explains the fast learning on HardObj. If not, we can then conclude that perhaps the model is leveraging previously acquired knowledge to do well on a composition.

- I really liked the experiment in Section 4.5. Although, I wonder if the decrease in performance has a simpler explanation like gibberish words like “sixnqkxb” contain more sub-words and are hence harder to reason / track. 

Quality: I think the paper is decently written. Some style suggestions:
- Table-2 could be converted into a barplot (moving numbers into appendix) since it’s really hard to parse so many numbers. And it’s generally good to bold the best numbers for a quick read.
- In Figure-7, it is unclear what the x axis refers to.

limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 3: Clear rejection
review_content: summary: This paper investigates whether pre-trained language models (T5 in this case) learn priors which support symbolic reasoning tasks in the process of optimizing their purely linguistic objective.  The performance of fine-tuned pre-trained LMs is evaluated on two symbolic reasoning tasks -- object location across a number of boxes, and a room navigation task -- and compared with identical models trained from scratch.  The results touch on a number of topics, but generally show (1) good reasoning accuracy when starting from the pre-trained t5 model, (2) abysmal performance by the from-scratch models, (3) a good degree amount of generalization performance for t5 to reason beyond the fine-tune scenario, and (4) that learning subtasks improves performance on a compositional task.

main_review: There's a lot to like in this paper.  The presentation and writing quality overall is just fantastic.  Not only a clear presentation, but a great style that makes the reading really enjoyable.  The authors have also been very thorough in listing the considerations behind each experiment, and approached each from multiple angles to get rid of confounds.  It's clear that a lot of thought lies behind each one.

My main criticism, and a crucial one, is that the experiments don't go far enough to show what the authors purport to show.  This boils down to a few things:

1) Unclear separation between language understanding and world knowledge.

The story laid out in the introduction, especially paragraph two, is one of text mirroring patterns in the real world, and the potetial for LMs to encapsulate this knowledge.  The authors provide a long list of potential types of world knowledge, and whether LMs capture any of these is very exciting and a very timely research question.  Just to underscore that this sort of information is central to the paper's argument, L25, "-If that is so-, then they ought to provide a useful inductive bias in learning to perform symbolic reasoning tasks that mirror real-world tasks".  I'll call this "world knowledge" for the remainder of the review.  Those statements really set the stage for the tone of what this paper is claiming to show, it follows from the reasoning in L7-8 of the abstract, etc.  Suffice to say, from that point I would expect that the paper ends by conclusively showing that this world knowledge is there in their chosen pre-trained LM.  And I think this is where it fails.

Desired interpretation of results: Container task.  Pre-trained model performs vastly better on extrapolation and overall when compared to from-scratch model, because it has embodied world knowledge, (maybe object persistence, containment, from the introduction list).  It shows good generalization ability because it T5 has induced reusable structures of reasoning / world knowledge.  It performs better with concrete nouns, then nouns, then verbs, because utilizing these objects in this manner does not make sense.

Alternate interpretation of results: Container task.  Pre-trained model performs vastly better when compared to from-scratch model because it has some rough representation of the word classes of these words.  It generalizes better to new words because it has similar representations of those words.  It shows good generalization ability because it can do so on top of well defined word class representations (completely apart from their role in reasoning).  It performs worse on verbs because it has trouble refining representations of ungrammatical sentences, and reasoning on top of non-sensical sentences suffer.

In the first interpretation, I'm really interested to further understand these induced world knowledge patterns more and I really want to accept this paper.  In the second interpretation, it's just shown that pre-trained LMs have really good representations of basic linguistic categories, and that fine-tuning for all sorts of tasks is hugely effective, something the community is already quite aware of.  It's still interesting to see applied to these bAbI-like tasks, but then again, that's a different sort of paper.  In that setting, I would have liked to have seen a comparison with those models and more practical advantages/disadvantages/failure cases described.  
And to the second interpretation, where basic linguistic knowledge accounts for everything,  I don't see any evidence here that would rule this out.  That's not to say that the hypothesis is wrong -- indeed, I would suspect there is a lot of world knowledge in there -- but I don't believe the authors have shown that conclusively.  I think the tasks chosen here make it difficult to really talk about real world knowledge.

The authors sometimes waffle between these two.  For instance, at the end of section 4.5, the English->gibberish conversion, the authors state that their experiment provides "evidence in favor of the notion that the inductive bias of large LMs does, in fact, aid in learning quickly and generalize well in several cases".  In this instances, the story is muddled, because we know that a general inductive bias in large LMs helps learning in many areas, it's basically the state of NLP at the moment, and so it needs no supporting evidence.  It's whether the inductive bias is based in real world knowledge that is unique to this work and pushed for in the introduction, and these experiments do not separate whether the advantage of the English model comes from linguistic knowledge or world knowledge.

It actually strikes me perhaps as evidence of point (3) below, where the gap between this lines correlates roughly to the combined linguistic and world knowledge the LM learns, while the reasonably high performance of the gibberish model is basically a testament to these training procedures getting the weights in a good position for transfer to unrelated tasks.

Another possibility of more concretely discovering evidence of world knowledge would be probing the representations in a targetted way.  Combining that with the sorts of experiments provided could strengthen the overall impression of the work.

2) Poor Baselines

In terms of experimental setup, of course it is tough to get at exactly what types of practical knowledge / inductive bias (apart from linguistic knowledge) these models contain, but I think some problem stem from the chosen baselines.  The authors use the same T5 model for the pre-trained model and for the from-scratch model.  This was a terrible idea in my opinion.  No reader should accept a 3 billion parameter model trained for 1000 steps to be a suitable baseline for comparison -- it is clear that this model was not given any chance to succeed.  I'm curious how smaller-parameter variants fair on the from-scratch setting.  I think it's quite clear that 3B parameters is not required for these sorts of tasks.

Speaking of similar tasks, the tasks are admittedly close to the FB bAbI in spirit, however, going back to that original work, there was not so much difficulty in achieving good results with models that we would considering quite basic by today's standards.  Why?  Presumably baseline model size and over-capacity.  But also thinking more generally, there were previously good results on bAbI tasks with simple memory-augmented models on top of word embeddings.  Is it similarly sound to claim that word embeddings had strong inductive biases that mirrored the world outside of their linguistic roles?  Of course it's also a question of training size and setting the tasks up to test generalization, but the rationale seems similar to the argument of this work.

I would have also liked to have seen some other models which are maybe capable of inducing linguistic knowledge relevant for the task, without any serious possibility of learning things like physics or spatial relationships, other examples of these properties, etc.

3.) Large pre-trained LMs can improve performance on unrelated tasks

An important point of comparison is the "Pretrained Transformers as Universal Computation Engines"[1], which was not mentioned in this paper, though it appeared on arxiv only a few weeks prior to the submission deadline.  Inn summary, the authors showed that pre-training on large text corpora provide a significant learning advantage when applied to completely unrelated domains (via fine-tuning).  

When considering this submission in light of this finding, it raises the question of whether the conclusions in this work are true / whether pre-trained LMs have induced symbolic reasoning knowledge, or merely a result of more universal organization properties in the network.  It would perhaps be interesting to compare these results with models pre-trained on an equally large section of another language and test in English, especially if preposition-style relationships were expressed in different ways (since they seem fundamental to both tasks).

Regardless of the answer to these questions, this is an important point to discuss.

[1]
@article{DBLP:journals/corr/abs-2103-05247,
  author    = {Kevin Lu and
               Aditya Grover and
               Pieter Abbeel and
               Igor Mordatch},
  title     = {Pretrained Transformers as Universal Computation Engines},
  journal   = {CoRR},
  volume    = {abs/2103.05247},
  year      = {2021},
  url       = {https://arxiv.org/abs/2103.05247},
  archivePrefix = {arXiv},
  eprint    = {2103.05247},
  timestamp = {Mon, 15 Mar 2021 17:30:55 +0100},
  biburl    = {https://dblp.org/rec/journals/corr/abs-2103-05247.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}






Questions:

Does the model size matter?  I'm curious how small a T5 model has to be before it has significant effects on the reasoning task, especially if additional fine-tuning layers were added (of equal size for all T5 variants), and thus isolate the task-specific weights from the "induced world knowledge" weights.

The paper might benefit from more specificity in what the world knowledge the authors feel is relevant here.  For each, they could also describe whether previous NLP work has shown these properties exist in pre-trained LMs or in their necessity in other toy tasks.
And for instance, to choose one from the introduction that may be relevant, "material properties"?  For whatever property the authors think relates to their chosen experiments, could they discuss how it is necessary for the task?

L153: for fine-tuning, is this purely using the T5 weights or are any additional layers/heads added to the model?
L155: what constitutes a step?  A single training example?



Presentation:

Personally I find the use of "symbolic reasoning engine" to describe a fine-tuned T5 model a failed attempt at inflating the sophistication of the method.  At the very least, it prevents readers from more immediately latching on to the specifics of the model and therefore the study.

Fig 1., very difficult to spot the various overlayed lines, especially in a printed version.

In L19-20, The examples of tasks where pre-trained LMs achieve SoTA accuracy do not require patterns of ``real-world'' structure described in L22.  Where is the citation or evidence that generated text reflects the latter forms of knowledge?

L65, BLEU score is used to see how similar sentences are on subword levels?  But BLEU is a word-level metric?

L92, how are selected objects arranged into sentences?  What kind of diversity is there in these generated sentences or templates?

L102, most common... nouns / and join the dataset -- this description was a bit clunky

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this work, the authors investigate if large scale pre-trained language models (such as T5) can provide inductive biases that are useful for solving language-based symbolic reasoning tasks, and furthermore, generalize to unseen settings. 

First, the authors define four types of tasks. While all the four tasks are formulated as sequence generation given prompts (prefixes), they require different kinds of reasoning:

1. Container. The prefix text describes an initial states of the world and a sequence of transitions, the target sequence needs to describe the ending states.

2. Navigation Route. The prefix text describes a map as well as two positions in the map, the target sequence is the navigation path from one position to the other.

3. Navigation Result. The prefix text describes a map, a starting position, and a navigation path, the target is the ending position.

4. Composite Task. A mixture of container and navigation tasks. 

Then, the authors define a set of experimental settings targeting the generalizability of models, from a variety of aspects such as map sizes, object/container numbers, reasoning steps required as well as some linguistic properties (e.g., replacing nouns with words with other POS tags, or even made-up words).

The authors provide plenty of experiments, suggesting the pre-trained language models indeed have capability of learning and to some extent generalizing symbolic rules via language-based tasks.

main_review: ## Synthetically generated data

Just to be transparent. I was one of the reviewers of this paper at ICML 2021. The authors have described in the "submission history" field about lacking of baselines. In addition to that, there were some ICML reviewers devalued this work because the used tasks are synthetically generated (and thus not natural language).  

I want to emphasize my point that **synthetically generated tasks are equally valuable**. 

I understand and agree that the ultimate goal is to model and perform textual reasoning on real world data. However, before stepping into natural language data, I believe it is meaningful to have synthetic environments that are controllable, helping researchers to better understand the problem. Although the tasks introduced in this work use templated language, it is clearly shown in the figures that they are not oversimplified (there are areas in the heatmaps where the models fail to generalize). It might make less sense to use natural language data when strong pre-trained LMs such as T5 still have trouble solving/generalizing on toy-ish tasks. Because otherwise, it might be difficult to analyze, for example, which part of the task is the model struggling on. Furthermore, there are many examples in the NLP communities that use synthetic datasets (templated language) as a starting point of certain research directions, such as bAbI (Weston et al., 2016), TextWorld (Côté et al., 2018) and many more.

One potentially interesting thing to try might be, back-translating (e.g., EN -> DE -> EN) the teamplated descriptions, and investigate to what degree the language models generalize on this dimension of "naturalness".

## Overall comments

This work provides a set of well-designed experiments with convincing results that explore an interesting and important direction. 

Since the flourishing of the large scale pre-trained language models, SOTA scores of neural models on various NLP tasks have increased by a lot. On some tasks, some researchers believe neural models have achieved human-level performance, while others argue that neural models may be somehow exploiting biases and trivial cues injected into the data or model unconsciously, rather than really doing reasoning. 

To that end, the community do require works that help us understanding what and why such language models can learn, and to generalize. This work falls into this category. 

Among the experiment designs and presentations, I especially like the heatmaps, which clearly show how the model gradually loses its generalizability when certain dimension of task becomes more difficult. 


## Typos and minor things

1. L217: There are 3 groups of columns

## References

1. Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks. Weston et al., 2016.

2. TextWorld: A Learning Environment for Text-based Games. Côté et al., 2018.
limitations_and_societal_impact: Since the current version of the paper uses a set of synthetically generated toy tasks, I do not see immediate negative societal impacts. However, because the work relies heavily on large scale pre-trained language models, which training data is difficult to control. The author should definitely discuss the potential negative effect or harm these techniques may bring in the revision.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper aims to explore whether the inductive bias of pre-trained language models can support symbolic reasoning tasks. Three different types of tasks are designed for the above objective, including a container-based task, a navigation task and a composite task. In order to verify the generalization ability of the model, the paper also designed different kinds of generalization for probing, including cardinality generalization, object generalization, POS generalization and reasonable phrasing generalization. T5 is fine-tuned and evaluated on these tasks, with some interesting findings observed, which I think are the key contribution of this work: (1) for container-based task, T5 shows good generalization and prediction capability; (2) for navigation task, T5 can also do a good job but when the number of inference steps changes, its performance will drop. (3) for composite task, T5 can do a good job  by learning in a curriculum learning way.
main_review: The objective of the paper is clearly clarified. The writing is easy to follow. The findings in the experiments are interesting. It is a good paper to study how well pre-trained LM can perform on reasoning-required tasks and its generalization capabilities on different aspects. I appreciate this work but think the datasets built within this paper is limited to specific domains and scenarios. It will be better if the paper can cover results on other open domain datasets as well, such as commonsense QA, mathematical reasoning, etc. 
limitations_and_societal_impact: no negative societal impact
review_title: null

Meta-review: This paper has certainly been put through the ringer of the "moving goalposts" of successive conference reviews, and for that I have a lot of sympathy (although would not make a decision on the basis thereof).

This paper presents an in-depth study of the ability of large pre-trained language models to generalise well on symbolic reasoning tasks, i.e. verifying whether some of the knowledge distilled through the pre-training objective(s) presents a useful inductive bias which aids generalisation in such downstream tasks which have a more structured nature than, say, RTE or QA.

The reviewers concur that the paper is well written, and presents interesting results. Given the interest the community has in this class of models, I concur that this is an important and timely contribution. Reviewer VwQ8 (rating 7) writes in strongest support, and in no small amount of detail, clearly seeking to champion the paper for publication. From reading Reviewer hX8A (rating 6) I could not tell why the rating was not higher. Reviewer KcKG (rating 6) is mildly supporting the paper, while presenting some concerns about whether the experiments are conclusive. I would have liked to read their thoughts following the authors' detailed response, but sadly it seems not to have been forthcoming. Finally Reviewer gnNv (rating 3) wrote at significant length regarding their concerns about this paper, and engaged in what I think must be the most detailed discussion I've witnessed during this round of area chairing.

Academically speaking, I find myself agreeing with a lot of objections and concerns Reviewer gnNv raises. The discussions with the authors here makes for a fascinating read, and I am not clear where the reviewer landed in terms of revising their assessment. I think the fairest way to view the outcome of this conversation is that there are fundamental difficulties with evaluating the sort of question the paper seeks to address, and in a sense the concerns levelled here are a statement about these difficulties more than they are a statement about the appropriateness the authors took, and the rigour thereof. Perhaps the paper is worth publishing on the grounds of its ability to generate such discussion alone.

Taking these reviews holistically, I have formed the strong opinion the paper should be accepted. It generates debate and controversy, but for the right reasons, and attempt to bring further light to the capabilities (and limits) of large pretrained language models. I would hope that the authors could, in as much detail as possible, incorporate their response to reviewers KcKG and gnNv in the final draft of the paper, so that the readers may benefit from these conversations (albeit indirectly).
Decision: Accept (Poster)