Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper empirically investigates whether pretrained large language models can have abstraction capability --  whether the model can be aware of the grammar instead of memorizing surface word patterns, taking from a transferability perspective between tasks of different characteristics. The paper also reports the effect of the training dynamics on the abstraction. Under this scheme, interesting results were found:  (1) there is a "memorize-then-abstract" two-stage process in the training time; (2) several middle-layer heads capture the abstract concepts, and (3) scaling does help.

strength_and_weaknesses: Strength
======
- An extensive and well-designed study.
- Results are revealing. Limitations are articulated and partly addressed.
- In-depth analysis is provided. The difference from previous negative results on the generalization of neural networks is briefly discussed.

Weaknesses
==========
- The number of terminals is a bit small compared to real languages.
clarity,_quality,_novelty_and_reproducibility: The paper is well-organized, and the study is extensive. The experimental designs are new and results are insightful.
summary_of_the_review: A nice addition to the recent literature of large-language models, helping to explain their recent great performance on different settings.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a novel benchmark for evaluating whether neural networks have learned abstract concepts. This benchmark is applied to pre-trained language models, revealing evidence that they indeed can learn abstract concepts. The paper also presents a range of analyses assessing the factors that contribute to the emergence of abstract concepts.
strength_and_weaknesses: ## Strengths:
- The design of the probing framework includes a number of careful control experiments.
- The analyses cover a wide range of factors that might contribute to the emergence of abstract concepts.

## Weaknesses:
- A key claim is that *pre-trained* language models, according to these results, show evidence of being able to learn abstract concepts. But the paper does not directly evaluate the importance of this pre-training (i.e. the generic pre-training that occurs prior to the experiment-specific pre-training in their probing framework). This should be straightforward to test, by simply using the same architectures (T5 and GPT-2), but training them from scratch on the probing framework training sets. I strongly suspect that the broader background pre-training is very important for this capacity, but it would be good to demonstrate this.

### Other comments:
- In the related work, the authors make a distinction between previous 'in-task' evaluations of abstraction vs. the proposed (between-task) evaluation. Are these really fundamentally different? Is there even a principled way to demarcate what constitutes a 'task'? It would be good for the authors to discuss this issue a bit more.
- The 'memorize-then-abstract' phenomenon is interesting, but since the 'memorization' component of this phenomenon involves performance on a heldout (in-task) test set, it seems like 'interpolate-then-abstract' may be a better term, since it isn't strictly speaking memorization.
- In the analysis of modularization, the paper states that 'we consider the first 36 heads in Figure 4', which are also described as 'abstract heads'. Does 'first' mean the heads with the highest DPC? The way it is phrased makes it sound as though the heads with the lowest value along the X axis were chosen, which would seem to be heads that are not important for abstraction.
- In Figures 6a and 6b, it would be helpful to include the arrow notation (i.e. $A \Rightarrow B$) in the legend.
- The memorize-then-abstract phenomenon seems potentially related to 'grokking' [1], in that both involve improvement in generalization after a model has already largely converged on the training data.

[1] Power, A., Burda, Y., Edwards, H., Babuschkin, I., & Misra, V. (2022). Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written, and the results are novel (to the best of my knowledge).
summary_of_the_review: This paper provides an interesting analysis of the emergence of abstract concepts in pre-trained language models, with a set of well-controlled experiments and extensive analyses. The only major missing element is an analysis of the extent to which pre-training is necessary for the emergence of abstract concepts.

Update after rebuttal: the authors have added new experiments demonstrating the importance of pre-training. This addresses my only major concern with the work, and I have updated my score accordingly.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a framework to evaluate the abstraction capability of deep models in terms of transferability. A number of empirical simulations are run on language models in order to evaluate the dynamics of abstraction.
strength_and_weaknesses: The paper considers a very interesting and relevant problem, and it explores it using a sensible framework and experiments. However, the reviewer is uncertain about its conclusions conceptually and pragmatically. From a conceptual point of view, abstraction is never well defined; all the empirical simulations seem to me deal with transfer learning, thus leading to a sort of identification of abstraction = transferability. I assume (together with some of the authors referenced in the introduction) that abstraction is something more than transfer ability. The author should probably discuss this point. From a pragmatic point of view, even if the authors explored some different settings in Section 6.4, it seems that the cases considered are limited, and hence the generalizability of the results.
Furthermore, I think it would be interesting if the authors were to engage with other relevant papers. For instance, the framework considered by the authors is built similarly to other common datasets (e.g.: ARC by Chollet); beyond modality, how are the two related? The "memorize-and-abstract" is also reminiscent of the learning dynamics in information bottleneck (Tishby); is there an analogy there?
clarity,_quality,_novelty_and_reproducibility: The paper is presents its framework and experiments clearly; results are, in the opinion of the reviewer, mostly concerned with transferability than abstraction. Setup is explained and data shared.
summary_of_the_review: The paper offers a useful probing framework for assessing the learning capabilities of deep neural models. Results are interesting in terms of transfer learning, but their meaning in terms of abstraction may be debatable. Explaining this gap and relating to other literature on abstraction would strongly improve the contribution.
review_title: null

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: This paper studies whether language-pretrained transformer models can abstract grammatical structures and thereby transfer them across superficially unrelated domains. They use carefully controlled experiments, and show some strong transfer effects of grammatical structures, which increase with model scale. 
strength_and_weaknesses: Strengths:
* The idea is interesting, and the experiments are fairly compelling.
* The thoroughness and design of the experiments is a major plus. Comparing to the contrast and control conditions makes the main experiment results much more compelling.
* I also appreciated the detailed evaluation of different conditions, exploring mutated grammars, analyses of head pruning, scaling, etc. There are a lot of fascinating details in this paper!

Weaknesses:
* It would be great to have a comparison of a model that was not pretrained on natural language first (e.g. T5 initial checkpoint or randomly initialized). That would help clarify the role of that initial pretraining in perhaps teaching the relevant structures of language (and might relate to why these results are different from other transfer results).
* The claim that model scale improves transfer seems very clear. However, the claims that data scale and diversity improve performance seem weakly supported at best. I think the authors should demonstrate that these effects are statistically significant (which appears not at all true from Fig. 6c-d), or remove these claims.
* In fact, the weakness of the data scale effect actually makes me more skeptical of the main results. How can 1.7k pretraining sentences have such a strong effect? How few domain A sentences would be needed to actually see a decrement? How can the model be learning abstract structures from such a small amount of data? It would be useful to clarify this (and/or to see if it breaks down when the scale is reduced farther).

Comments / notes:
* There is some older connectionist work on the idea that neural networks could transfer abstract grammar structures across domains with superficially different features [Dienes et al., 1999]; and some more recent work on generalization of relational properties [Geiger et al., 2022]. They are of course far simpler than the present experiments, but I think these would be a nice piece of motivating context to add from the cognitive side. 

References
------ 

Dienes, Z., Altmann, G. T., & Gao, S. J. (1999). Mapping across domains without feedback: A neural network model of transfer of implicit knowledge. Cognitive Science, 23(1), 53-82.

Geiger, A., Carstensen, A., Frank, M. C., & Potts, C. (2022). Relational reasoning and generalization using nonsymbolic neural networks. Psychological Review. (https://arxiv.org/pdf/2006.07968.pdf) 
clarity,_quality,_novelty_and_reproducibility: The paper is moderately clear; I think the presentation could be improved in various ways:
* Showing Table 7 or similar in the main text alongside Table 1 to give a more concrete presentation of qualitatively what individual inputs and outputs like in the different tasks would really help readers to understand the paper. 
* I think the clarity would be improved by using the more-standard term “Control experiment” rather than “Controlled experiment”—using “controlled experiment” makes it sound like the main experiments are uncontrolled, and the controlled version is just better.

The quality seems decent.

The work is novel.

summary_of_the_review: Post-revision update
---------------

The authors have addressed the three main weaknesses I listed in their follow-up experiments, and I believe they have improved the clarity of the paper. I have improved my score accordingly.

Original review
--------------------

I think this paper is a potentially-compelling demonstration of a fascinating transfer phenomenon. I think this is a very interesting work that should lead to lots of follow-on investigations. I have some lingering concerns and suggestions noted above, that I hope the authors will address, and that I expect would increase the impact of the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors present a framework for systematically evaluating whether a language model can learn abstract concepts (syntactic categories) when fine-tuned on a target task. In this case the target task is one whose language has a grammar homomorphic to the source grammar, which ensures that the two tasks share the same abstract concepts. This framework is used to probe whether, after fine-tuning, the model is able to solve a task which a model trained on the fine-tuning dataset alone cannot, demonstrating by transfer the existence of some ability to employ abstractions.

Strengths: All the reviewers agreed that the paper provides a nice, well-controlled set of experiments to investigate the learning of these kinds of abstract categories.

Weaknesses: 

1. I share the concern with reviewer tkML that the term "abstraction" is not very clearly articulated (e.g. line 52). My interpretation was specifically of an abstract category (a set of phrases in a language corresponding to a grammar non-terminal). Perhaps the authors can clarify this in the final version. The "formal" definition given on p. 152 is not formal at all (just sort of formally stated).  My initial concern while reading through was with that the results were not sufficiently general, but after discussion with the reviewers and given the the experiments on grammar mutations, which show a robust degradation in performance when the source and target grammars are not perfectly homomorphic, my worry was largely allayed.

2. The writing is not very clear and could use editing.  A few minor issues:
-line 162: grammar is a set of syntactic rules; in your setting those rules are hidden
table 1: you say you use a double-headed arrow instead of a single-headed one (usual grammar rule) but don’t say how your notation differs.
-why not just say that your source and target tasks require languages with homomorphic grammars?
-line 255: say what “DPC” stands for
-line 266: A module (not "modular") is a subset of the parameters …
The conclusion is a bit too short to be helpful (e.g. “insightful conclusions”). Perhaps just drop it.

The paper specifically focuses on a fine-tuning setting for a pre-trained model. This shows that abstractions can be induced by fine-tuning when the tasks are sufficiently similar. The reviewers asked for additional experiments to evaluate the importance of pre-training which the authors performed. The results show pre-training is essential but don't say much about the nature of this kind of latent ability to be fine-tuned to learn abstraction or what mechanisms are at play in eliciting the abstractions through fine-tuning. This will be interesting to explore in future work.

justification_for_why_not_higher_score: My concern with the paper is largely that the English is poor and it's not very clearly written. I could be convinced to push it up to spotlight or oral.
justification_for_why_not_lower_score: The next lower score is "reject" which is certainly not the right outcome for this paper.
Decision: Accept: poster