Does Deep Learning Learn to Abstract? A Systematic Probing Framework
Authors: Shengnan An
Zeqi Lin
Bei Chen
Qiang Fu
Nanning Zheng
Jian-Guang Lou
Venue: ICLR 2023 poster (2023)
Keywords: Abstraction Capability
Probing Tasks
Deep Learning
Pre-Trained Language Model

Abstract: Abstraction is a desirable capability for deep learning models, which means to induce abstract concepts from concrete instances and flexibly apply them beyond the learning context. At the same time, there is a lack of clear understanding about both the presence and further characteristics of this capability in deep learning models. In this paper, we introduce a systematic probing framework to explore the abstraction capability of deep learning models from a transferability perspective. A set of controlled experiments are conducted based on this framework, providing strong evidence that two probed pre-trained language models (PLMs), T5 and GPT2, have the abstraction capability. We also conduct in-depth analysis, thus shedding further light: (1) the whole training phase exhibits a "memorize-then-abstract" two-stage process; (2) the learned abstract concepts are gathered in a few middle-layer attention heads, rather than being evenly distributed throughout the model; (3) the probed abstraction capabilities exhibit robustness against concept mutations, and are more robust to low-level/source-side mutations than high-level/target-side ones; (4) generic pre-training is critical to the emergence of abstraction capability, and PLMs exhibit better abstraction with larger model sizes and data scales.

Published as a conference paper at ICLR 2023
DOES DEEP LEARNING LEARN TO ABSTRACT? A SYSTEMATIC PROBING FRAMEWORK
Shengnan An"i, Zeqi Lin', Bei Chen', Qiang Ful, Nanning Zheng', Jian-Guang LOUI Institute of Artificial Intelligence and Robotics, Xi'an an Jiaotong University
$ Microsoft Corporation
(an1 00066344930staa nnzhengemail) xjtu. edu. Zeqii Lin, beichen, qifu, jlou) ) @emmcrooott.... com
ABSTRACT
Abstraction is a desirable capability for deep learning models, which means to induce abstract concepts from concrete instances and flexibly apply them beyond the learning context. At the same time, there is a lack of clear understanding about both the presence and further characteristics of this capability deep learning models. In this paper, introduce systematic probing framework to explore the abstraction capability of deep learning models from a transferability perspective. A set of controlled experiments are conducted based on this framework, providing strong evidence that two probed pre-trained language models (PLMs), T5 and GPT2, have the abstraction capability. We also conduct in-depth analysis, thus shedding further light: (I) the whole training phase exhibits a "memorize-then- abstract" two-stage process; (2) the learned abstract concepts are gathered in few middle-layer attention heads, rather than evenly distributed throughout the model; (3) the probed abstraction capabilities exhibit robustness against concept mutations, and are more robust to low-level/source-side mutations than high-Level/target-iiie ones; (4) generic pre-training critical to the emergence of abstraction capability, and PLMs exhibit better abstraction with larger model sizes and data scales.
INTRODUCTION
Whereas concrete concepts are typically concerned only with things in the world, abstract concepts are about internal events. Barsalou et al. (7999)
Abstraction means capturing the general patterns (often referred as abstract concepts) efficiently in specific learning context and reusing these patterns flexibly beyond the context (Mitchell 2021
Kumar et al. 2022 Giunchiglia & Walsh 1992 Hull 1920). For instance, the abstraction on language means recognizing the underlying syntax and semantics behind concrete sentences. It is thought to be one of the fundamental faculties in human cognition for effectively learning, understanding and
robustly generalizing, and has been studied for long time in cognitive psychology and behavioral
sciences (Gentner& & Medinal 1998 Barsalou et al. 1999 Shivhare & Kumar 2016 Konidaris 2019).
The abstraction capability is also critical for deep learning, but many previous studies suggested that the surprising success of deep learning may come from the memorization of some surface patterns (also called superficial correlations or shortcuts) (Geirhos et al. 2020 Du et al. (202). such some special tokens (Niven & Kao 2020 Gururangan et al. 2018)) overlapping contexts LLai et al. 2021 Sen & Saffari 2020), and familiar vocabularies (Aji al. 2020). It is still unclear whether the models just memorize these patterns without abstractions, or they do learn abstract concepts (yet overwhelmed by surface patterns when applied in similar context in training). Therefore, this paper aims to take a step forward to probe the abstraction capability of deep learning models, keeping the effects of abstract concepts and surface patterns decoupled and controlled individually.
Our key idea is to probe the abstraction capability from a transferability perspective, since surface patterns are always bounded with fask-specific characteristies while abstract concepts can be more
Work done during an intersship Microsoft Rescarch.

---

Published as a conference paper at ICLR 2023
Outpus
Output
w/o Abstraction
a
AB8
8
3
Task
Task B
2c - A
Context
Updata
ba B
cem B
C
Abstract Concepts
cb- C
Concepts
Dulput
Well Generalte
DA
Output
A; X
X, X
Figure 1: Motivating example: the abstract concepts learned in task A can be effectively reused in task B, but surface patterns are useless. Unused patterns or concepts are whitened after the update.
generally reused. We consider designing multiple tasks with shared abstract concepts and totally different surface patterns, then tracing whether the learning one task can boost the performance another. Figure demonstrates a motivating example.
Motivating Example As shown in Figure D suppose we want to examine whether model can learn the abstract rule (i.e., the symbolic mapping rule 21 - X2, in which I; and X, are general variable slots) from the task A, just memorize surface maps (e.g., ob + AB, in which and A are task-specifi symbols). To reveal the different transferability of two learning mechanisms, utilize probing task B that contains the same underlying abstract rule as task but does not overlap with its symbol set. If the model could learn the abstract rule from task A, it would reuse it to interpret new context, thus effectively solving task B. But if not, memorizing some surface maps that are bounded with task-specific symbols is less effective to solve task B.
Motivated by this example, design systematic framework for probing abstraction capability. This framework considers a set of probing tasks along with three procedures of experiments based on the transfer learning paradigm. The use of abstract concepts and task specific characteristics in probing tasks are separately controlled. To probe the abstraction capability of language models, this work mainly considers grammar as the abstract concept] The grammar of a formal language set of hidden rules behind concrete sentences and determines how terminals are combined into sentences that are valid to the syntax. We want to explore whether the model can be aware of the grammar, simply memorize some specific word combinations. We instantiate as a grammar probe that is constructed from the designed formal grammar and terminal sets. The probing results show strong evidence that two probed PLMs (specifically, T5-Base (Raffel et al.) 2020) and GPT2-Medium (Radford et al. 2019)) have the abstraction capability to learn abstract concepts from concrete instances, rather than just simply memorizing surface patterns.
framework
After probing the existence of abstraction capability, we further explore the following questions. RQ1: What the characteristic of the training dynamies on learning abstract concepts? RQ2: How are these learned abstract concepts distributed in the model? Concentrated in few modules or evenly distributed whole model? RQ3: How robust is the abstraction capability on tasks with mutated abstract concepts? RQ4: How would generic pre- pre-training and general factors influence abstraction? Here we outline some interesting findings from our in-depth investigations: (1) the training phase exhibits a themorize-then-abstrae our probes are gathered in a few middle-layer heads; (3) abstraction capability is more robust to suure--iidcloow-eved mutations than to target-side/high-leed ones; (4) generic pre-training is critical to the emergence abstraction capability, and larger model size and data scale are beneficial.
two-stage process; (2) the abstract concepts learned in
Contributions 1) We propose systematic probing framework for abstraction capability, guiding the design of controlled tasks and procedures from a transferability perspective. 2) We instan- tiate this framework with concrete tasks and show strong evidence that two probed PLMs have the abstraction capability. 3) We further analyze this capability and provide insightful conclu- sions by investigating the above research questions. Our code and data publicly available hwtps///wwooocooooooooooeeeeeeee
We also prohed other abstract concepts such as operation semantics in Appendix
2

---

Published as a conference paper at ICLR 2023
2 RELATED WORK
Probing deep learning models. To explore whether deep learning models have certain capabilities, there has been much work examining these black- box models in some specially designed settings.
called probes (Petroni et al., 2019, Tenney et al. 2018 Warstadt et al. 2019; Lin et al. 2019 Hewitt & Manning 2019 Vulić et al. 2020). The key challenge in designing probes is to exclude superficial correlations. That is, the performance of the model in the probing setting should be highly correlated with the capability to be probed rather than other influencing factors. For instance, to probe whether the model encodes some snowledge/informatti data, a standard approach is to freeze the model parameters (Petroni et al. 2019 Tenney et al. 2018) ; to probe whether the model have compositionality rather than just memorize the label bution, previous work injected statistical bias into the data splits (Lake & Baroni 2018 Keysers et al. 2019
in the representation rather than just over-fr fit the
Kim & Linzen 2020). In this work, to explore whether models have abstraction capability rather than just memorize surface patterns, leverage the transferability of abstract concepts, which has been considered as one essential aspect of abstraction (Mitchell 2021; Kumar et al. 2022) and explored
from a cognitive science perspective on neural networks (Dienes al. [999] Geiger et al. 2022).
Abstraction capability. Abstraction has been studied for long term in cognitive psychology and behavioral sciences (Hull, 1920 Gentner& & Medina 1998, Barsalou et al.,
1999
Burgoon et al.
2013: Wang 2015; Shivhare & Kumar 6 et al. 2017 | 2011)] Konidaris 20199 and
has attracted attention in the artificial intelligence field (Guunchiglia Walsh, 1992 Richardson et al.
2020 Clark et al.] |020 Talmor et al. [220 Mitchell 2021 Zadrozzy] 2021 Millhouse et al. 2021
Kumar et al. 2022). ). The abstraction capability of DNN models has been explored in many tasks such as visual reasoning (Johnson et al. 2017, Barrett et al. 2018 Chollet, 2019; Kumar et al. (022). grounded language understanding (Ruis et al. 2020), and game playing (Tiividis et al.] (021). As our work focuses language models, another closely related topic is compositional generalization (Lake) & Baroni 2018 Keysers et al.] [219 Kim & Linzen 2020), which explored whether neural models could learn high-level grammars from specially designed training examples and apply the learned grammars through compositions. These works concluded that general-propose neural models (such as LSTM and Transformer) could not learn the full grammar with biased observations and demonstrated the importance of symbolic mechanisms for abstraction (Liu et al. 22200] Chen et al. [2020] Liu et al. 2021a). Some other previous work also explored the abstraction of language models in their specially designed tasks (Chollet) 2019, Mitchell| [2021] Zadrozny (021).
Most previous explorations of DNN abstraction capabilities did not consider to explicitly avoid and check the influence from task- specific characteristics, thus leaving potential risks that the model may perform well in terms of surface patterns over-fitted to task-specific designs (e.g.. patterns in candidate answers (Zhang et al. 2019) rather than abstract concepts. Some implicit strategies have been leveraged to alleviate such potential influence through indirect ways: some previous work considered using biased task task-specific designs in training and test data separately (Kim & Linzen 2020 Barrett et al. 2018): some have attempted to fix the observed problems in existing probes on an ad hoc basis (Hu et al. 2021 Benny al. 202T); some considered inject great task diversity, which implicitly increases difficulty of learning practical shortcut (Chollet) (019).. In this work, rather than implicitly alleviating this potential risks, we consider to explicitly check whether there is performance leakage from surface patterns by leveraging the transferability of abstraction capability and comparing performance among a set of controlled experiments.
PROBING FRAMEWORK
As mentioned in Section I abstraction is the capability to induce abstract concepts from concrete instances in a certain learning context and flexibly generalize these concepts beyond the context. A key difference between a surface pattern and an abstract concept is their different cross-task transfcrability, as the former is always bounded with some task-specific characteristics (c.g.. a certain vocabulary) while the latter is transferable across tasks. We define this property as following.
Property: Transferability of Abstract Concepts. Consider two machine learning tasks A and B that do not share any common instances between their task-specific characteristics spaces, but have essentially the same set of abstract concepts behind them, the transferability of abstract concepts means that learning A can help better learn B.
3

---

Published as a conference paper at ICLR 2023
Task Design
Procedures of Experiments
Expectations
Pretrin
Fivetune
Contral Exp B
B
Concepts
Arring
Prodinn
bak
Contreat
Main Exp A
B
Other Factor Retannel
Tack-Specffi Chanacteristics Dfffrentt
Bnd Partormancce C
Figure 2: The illustration of the probing framework.
Notations: We denote two learning paradigms: A B as procedure that the model is first trained on and then fine-tuned on B, and f as a procedure of directly training on B without the pre-training A. The transferability
be evaluated with the performance gain from procedure A + B compared to f B, denoted as (A + B).
verify the learning of abstract concepts by checking whether the transferability exhibited (i.e., assessing A(A = B)). In the following, we design a framework for probing the learning of abstract concepts in systematic manner and illustrate it in Figure 2
Based this property,
Aiming
This framework examines whether probed model could learn abstract concepts C, from the aiming task A with a train set A.
Task Design
Probing task B with the transfer set 13 and test set contains the abstract concepts C that is required to be the same as CA. The task-specific characteristies used to construct BUB do not overlap with that of A. In addition, the examples in B are restricted to contain insufficient information for the probed model to learn Cy perfectly. Thus, the gain from the abstraction in task would be noticeable.
Contrast task C with the contrast set T aims to further confirm that the performance in task B is principally correlated with abstractions rather than other factors. The abstract concepts Co is constructed by greatly breaking (changing) CA. thus compared with task A, the abstraction on task C' is less effective for solving task B. The task-specific characteristics and other latent factors in constructing C are kept the same with that in A.
Procedures of Experiments (abbreviated Exp)
Control Exp 1 B: only train the model on B and test on B.
Main Exp = B: train the model on A, then fine-tune on B, finally test on B.
Contrast Exp C + B: train the model on C, then fine-tune on B, finally test on B.
Hypothesis and Expectations
Hypothesis: the probed model can learn abstract concepts CA from A. Expectation 1: A(A + B) is significantly high, i.e., A = B brings considerable gain compared with f B.
Expectation 2: A(C = B) is significantly lower than A(A B) (or close to zero), i.e., Expectation is highly correlated with the learning of abstract concepts rather than other factors.
4 PROBING ABSTRACTION CAPABILITY OF LANGUAGE MODELS
The abstract concepts mainly considered this work is grammar, set of syntactic rules hidden behind concrete sentences that determine how terminals are combined into sentences that are valid to
4

---

Published as a conference paper at ICLR 2023
Table 1: Part of G2 and Gr Rules in the last row are allowed iterate up to times.
Source Grammaar G,
Target Side Gramnmrr G2
Type
verb S, sub direst-obj/ indirect-nhi -* S., conj -o S.
PREDICATE Sp
AGENT /HEME// RECIPIENT Sg
T-Production
CONCAT -n Sc
senience -M subj verb
CLAUSE PREDICATEE AGENT) )
sentence subj verb direct-obj indirect-obj
CLAUSE PREDICATE/ AGENT, THEME, RECIPIENT) CLAUSE CLAUSE CONCAT CLAUSE
N N-Production Rule
sentence - sentence conj sentence
the syntax. To design grammar probe, we instantiate the framework with formal language translation (FLT) tasks. We assume that the generative grammar of the source and target languages contain the abstract concepts of FLT tasks, and that the surface patterns (e.g.. familiar bigrams) are bounded with task-specific terminal sets. We give a more specific definition of abstraction based on FLT tasks:
Considering an FLT task T L. + L. that translate the source language Ls (with grammar G, and terminals S.) to the target language L. (with grammar G, and terminals S,). and set of concrete pairs T {C) - ++))* in which 12 and Iž are sentences from L. and La respectively, the abstraction capability learning the map from G, to G2 during training on rather than just simply memorizing terminal-specific patterns that are bounded with S, and S...
Definition:
Our FLT tasks are majorly derived from the synthetie semantic parsing task COGS (Kim & Linzen 2020) and the Probabilistic Context-Free Free Grammar (PCFG) it used. We directly take the source grammar G2 in COGS which mimics the English natural language grammar, and reconstruct the target grammar G2 in COGS to be chain-structured (dctailed in Appendix K.1)). The map from G, to G2 a homomorphism (partly shown in Table T.. Terminals can be divided into three groups: the verbs S, in G, (aand the PREDICATES Sp in Gt), the nouns S, (the ENTITYS Sg) and the conjunctions S. (the CONCATs Sc). The production rules can be categorized as T-Production rules (only containing terminals at the right side) and N-Production rules.
We assign to the tasks A and B the same set of production rules while different terminals. It means that task A and B share the same abstract concepts while having no overlap between the task-specific characteristic spaces. For constructing task C, we completely change the production rules for A while preserving the terminal sets, thus task A and C' do not share abstract concepts while could have similar task-specific characteristics. We describe the instantiation of different sets in detail as follows. Examples in these sets are contained in Appendix [.
Train set A. To generate examples in A, we derive G. and G. by only one-to-on toc-to-one replacing the terminals in G and G, with new o New terminals are sampled from the Wordlist Corpora in NLTK (Bird et al. 2009). Additionally, as the original S. (also Sc) only contains single terminal, we add 31 additional terminals into the new S. (and Sc) to increase the diversity. The terminal diversity will be further discussed in Section G.I
Transfer set B and Test set B. We take the train set in COGS as B. and take the sentential complement (Com.) set and subject modification (Mod.) set as B for two sub-probes, The B only contains examples with up to 2 recursions and object modifications, while the B contains up to 12 recursions and subject modifications. It has been proved that training on / is not enough for a DNN model to learn the full grammars of COGS for handling the test cases in B (Kim & Linzen 2020
Contrast set C. Compared with A, C is generated with the same source grammar Gj. but the target grammar is totally changed as G for each rule of G its right-side word order is reversed Except for the generative grammar, all other factors are kept the same with A during generating C.
EXPERIMENTAL SETUP AND MAIN RESULTS
We probe two pre-trained language models: T5- and GPT2-Medium. Our experiments based on the Huggingface Transformer models (Wolf et al. 2020). For both (continuc) pre- trainiiiing and fine-tuning, we take Adam (Loshchilov & Hutter 2018) with lc-5 learning rate and 0.01 weight
"Some non non-semantic terminals are kept the same, such as the period in Toore and parentheses in Legt- Some hasic rules are preserved (c.g., the order of the preceding and following parentheses).
5

---

Published as a conference paper at ICLR 2023
Table 2: The main results of our probe. A(A + B) and A + B) are in brackets. The evaluation metric is exact match accuracy (%). Com. and Mod. represent the sentential complement and subject modification sets for B. These results are in line with our two expectations.
Control Exp
Exp
Comtrast Exp
Control Exep
Main Exp
Contrast Exp
Model
Sub-Probe
1 B
A mo B
C =
Model
Sub-Probe
B
A mb 13
C =p B
18.7
71,9
(:2.7
Avg.
8.0
47.9 (+39.8)
8.1 (+0.1)
T5
Com.
23.1
88.2
15.4
GPT2
Com,
1.9
48.2
Mod.
14.3
55.6
16.5
Mod.
14.1
47.6
13.6
decay. Batch size is 8 and max training step is 100k. We generate 3 groups of new terminals, repeat the experiments on each group with 2 random seeds, and finally average 6 results. The early-stopping strategy is applied to avoid catastrophic forgetting. Detailed settings are listed in Appendix [K]
Table 2 shows the main results of our probe. For both two sub-probes, the performances of two probed models are in line with two Expectations set in Section R] First, the results of ft B are very low, and A + B can bring significant improvement, which is in line with Expectation 1. Second, the results of C + B are much lower than A + B (and are even just comparable with f B), which is in line with Expectation 2. As two expectations are experimentally examined, we can draw preliminary conclusion: our probing results provide strong evidence that two probed PLMs have the abstraction capability to learn abstract concepts from concrete instances rather than just memorize surface patterns, and to transfer the learned abstract concepts beyond specific tasks.
ANALYSIS
Based our designed prohe and results above, we further analyze the abstraction capability of PLMM to answer the RQs mentioned in Section | All experiments below are derived from Com. sub-probe, and are mainly conducted with T5-Base model except that are explicitly mentioned.
6.1 LEARNING PROCESS OF ABSTRACT CONCEPTS
To investigate the learning process of abstract concepts, we save checkpoints for every 1,000 steps during training A. Each checkpoint is further fine-tuned / and tested on B. For comparison, we also investigate the process of memorizing surface patterns by directly examining each checkpoint on the held-out dev set in task A. Figure shows the performance curves of two learning processes.
The training phase exhibits a Figure 3 there is obvious phase difference (48k training steps) between two time points that two learning processes achieve their 90% relative performance, respectively. Such a phase difference means that when the model has already performed well task A in an early training phase, the learning of desired abstract concepts is still on-going. In other words, the in-task performance in an early training phase comes mainly from the effects of some task-specific surface patterns rather than general abstract concepts. With extending the training phase, the abstract concepts can be further extracted/enhanced. This phase difference also suggests that the pre-training process should be continued even if the model has already achieved a good in in-pre-training performance.
two-stage process. As shown in
The learning of abstract concepts is accelerated after in-task examples are well learned. After the model reaches 90% in-task relative performance (i.e. right side of the red dashed line), the learning curve of abstract concepts (i.e., the blue curve) rises more rapidly.
The learning of abstract concepts is not stable in the early training phase. The curve of in in-kask performance is much smoother than the cross-task one. This suggests that the learning and transfer of abstract concepts is not stable. Nevertheless, the large fluctuations occur mainly in the early phases of training. With increasing training steps, this instability gradually decreases.
6.2 ABSTRACT ATTENTION HEADS
To investigate how the learned abstract concepts are distributed in the model, first conduct prelim- inary experiments by separately freezing parameters in each layer and sub-layer during fine-tunin tee-ttning. and find that the parameters in attention sub-layers play important roles (detailed in Appendix E To

---

Published as a conference paper at ICLR 2023
raw results
Relative Performance (%)
min-max nermalization
100
100
w NW Gooreap
90
80
80
Accuracy
60-
60
50
40
40
Task Test Acc
20
30
Task Dev Acc
Stops
20
0
20
60 Training Steps (k)
40
80
100
20
40
60
80
100
Training Steps (k)
Figure 3: Two learning process. Blue curves represent the learning performance abstract concepts
and red curves represent the learning performance of in-task examples.
Ditbeeree Pit Ohrggs
Er codedr Sell-Altant Heass Decodor ScliAaentann Heads Decoder Crov--ttettion leads
Figure 4: DPC of each pruned head. The heads sorted from left to right are located from the first to the last layer in the model.
further determine the contribution of each attention head, we consider measuring the performance degradation after excluding the effect of each head. Specifically, we evaluate the change in perplexity (PPL) of examples in B after pruning the normalized wight of each head as follows,
Dg Dnnn(1) (h)
[IPL((C,O - PPL(((C):
T3
in which h represents a certain head, @ is the full set of parameters in PLM after fine-tuning on B, 6-h means pruming the h head, and (1g,, 11) is the input-output pair in B. Note that a higher PPL means a lower performance. Considering that some heads may store the task-specific knowledge learned from fine-tuned data B, pruning these heads may also lead to performance changes. Therefore, we also evaluate baseline PPL change Ag,8(M) on fine-tuned examples in B and measure the difference in PPL PPL. changes (DPC)=
4.. D The DPC of each head shown in Figure
Abstract concepts are largely contained in a few heads, not evenly distributed in all heads. Note that there are totally 432 attention heads in T5-Base. Figure I that among hundreds of heads, only dozen of them are highly correlated with storing abstract concepts in our probe.
These abstract attention heads are gathered in middle layers in T5. A larger index in Figure 4 means that the corresponding head is more away from the input side and closer to the output side. shows that the abstract attention heads (i.e., heads with high DPC) are mainly located in the middle layers of T5 Base model, i.e., the last encoder layers and first decoder layers.
We further explore whether abstract concepts modularized in the model. A module is part of parameters that can individually perform a specific target functionality (Csordás et al., 2020). To investigate modularity, we take the method of freezing certain parameters during fine-tuning to examine whether the update of these parameters can be independent. We consider the top 36 heads with the highest DPC (which contain some redundant heads) abstract heads. For comparison, separately experiment with freezing 36 random heads. Table shows that freezing abstract heads takes effect while freezing random heads does not. We further explore the modularity in Appendix E
Table 3: Freeze abstract heads.
Table 4: PLMMs performance with fuzzy abstract concepts.
Method
A = B
Probe
Model
Control Exp
Main Exp
Contrast Exp
A
Baseline
92.8
+Frezze Abstract Heads
96.6 (+3.8)
24.0
35.1 (+111)
26.2 (42.2)
Fuzzy Grammar
+Frezze Random Heads
92.9
GPT2
16.4
(+4.)
11.6 (-4.8)

---

Published as a conference paper at ICLR 2023
Original
Original
Redundant
Coarse
LocalR
LocalR
Nest
Commited Exp
Nest 20
Conmited Fab
20
Accuracy (%)
60
120
Accuracy (%)
60
80
100
(a) Source-Side Mutations
(b) Target-Side Mutations
Figure Performance with different derivations of (a) source and (b) target grammar.
6.3 ROBUSTNESS OF ABSTRACTION CAPABILITY
We explore the robustness of the probed abstraction capability when the abstract concepts in designed probes are mutated. Different from the contrast task C' which the target grammar is totally changed, here consider partially injecting mutations into source/target-side grammar. According to the formal grammar in Table 1 we consider injecting mutations at different abstract levels: changing T-Production rules can be regarded and the whole sentence structure is kept; changing non-iterative N-Production rules can be regarded as mid-level mutation, since the local structure will be mutated but the whole recursive structure preserved; changing iterative N-Production rules can be regarded as a high-level mutation, since the whole recursive structure will be reconstructed. Based on the grammar used in formal language task A, we design three derivations G; by injecting mutations into the original G in different levels. Examples of different derivations are contained Appendix F.4
low-Ievel mutation, since only terminals will be influenced
Coarse G (low-level mutation): We omit the non-terminals AGENT, THEME, RECIPIENT, and their corresponding terminals. In other words, the second T-Production rule in Table T Tis removed. Compared with G, (also G,), Coarse G; does not contain detailed arguments of PREDICATE.
Local Reverse (LocalR) Gi (mid-level mutation): The local word order in a sentence is reversed. Specifically, we reverse the right-side word orders of the N-Production rules, except for the rule in the last row of Table which is an iterative one. It means that the order of CLAUSEs (determined by the last rule) remains the same, while the terminals in each CLAUSE are locally reversed.
Nested G; (high-level mutation): It is obtained by changing the iterative rule (i.e, the last rule in Table from the chain-structure to be nested. The new N-Production rule is "CLAUSE PREDICATE (AGENT, CONCAT CLAUSE)"
We also construct G from G with the same technique except for the language must contain enough information to generate targets. Thus, we design Redundant G which contains redundant terminals that are mapped into targets (detailed in Appendix F.3. We
one, the source
separately change the source and target grammars to derivations and show results in Figure 5
PLMs exhibit robustness against mutations in abstract concepts. Results of these derivations with mutations are higher than the Control Exp (and Contrast Exp). indicating that even though the learned abstract concepts are only partially matched with that in downstream tasks, the abstraction capability of PLMS can still leverage the similar parts in two sets of mutated abstract concepts.
Abstraction capability more robust to low-level mutations. Among three kinds of derivations, the low-level mutated ones (Coarse G; and Redundant G)) perform best, while the high-level mutated ones (Nested G and G.) perform worst. This trend implies that the robustness of the abstraction capability decreases as the mutation level of abstract concept rises. This also suggests that matching of high-level abstract concepts should prioritized when selecting pre-traine pre-training tasks.
A Abstraction capability is more robust to source-side mutations. Comparing the results in Figure and 5b) source-side mutations bring less affects to downstream performance than target-side robustly reuse source-side abstract concepts.
ones, indicating that PLMs
Redundant information barely affects abstraction. Surprisingly, the performance Redundant G. is nearly the same with that of the original G which means that injecting redundant information into inputs would hardly affect the learning of abstract concepts. indicates that the abstract capability of PLM naturally exclude the influence of irrlevant information.
8

---

Published as a conference paper at ICLR 2023
Carrat on 11
comniioa (1g)
90
Accuracy (36)
Accuracy (80)
Accuracy (%g
70
40
50-
20
30
TS-Base
0-
10a
10
10a
60M
220M
770M
117M
345M
762M
instances
(a) Model Sizes of T5
(b) Model Sizes of GPT2
(c) Data Scale
Figure 6: Performance with different model sizes, data scales and data diversity.
Table 5: Probing results for randomly initialized models. ( = B) and A(C = B) are in brackets.
Model
Probe
Control Exp
Main Exp
Contrast Exp
Model
Suh-Probe
Control Exp BB
Main Exp A 1
Contrast Exp
ft 1
-b 1?
C B
C m> B
Avg.
(+2.0)
(-1.1)
Avg.
1.5 (+1.7)
0.4 (-0.4)
Com..
0.1
1.0
0.1
Com.
0.1
12.3
Fuzzy abstract concepts can also be learned and transferred. Compared with the formal gram- mar discussed above, which can be concretely defined, fuzzy grammar is more free (such natural language grammar). To explore how would abstraction capability perform on fuzzy grammar, we take natural language sentences for experiments and design different sub-probe. Detailed designs are described in Appendix H. We report BLEU score in Table It shows that the performance of PLMs on learning fuzzy grammar is also in line with our two expectations.
by mimicking the Com.
6.4 GENERAL FACTORS & GENERIC PRE-TRAINIE
As explored in previous work, there are some general factors that influence the performance DNN models (Bommasani et al., 202 Wei et al. 2022 Henighan scale. We investigate how these general factors and the generic pre-training influence the learning of abstract concepts. More results and analysis can be found Appendix
2020). such model size and data
PLMs exhibit better abstraction with larger model sizes. Figure 6a and 6b show that for both T5 and GPT2 architectures, larger pre-trained language models have better abstraction capability than the smaller ones, as we can observe that the gains from the Control Exp to the Main Exp become greater with the model sizes increasing.
Larger data scale in pre-training helps better exhibit abstraction. Figure 6c shows 75-Base performance with different scales of the train set A. shows that performance increases rapidly from ~00 to <3.4K (with 550 absolute accuracy improvement) and improves marginally (and unstably) from <..4K to ~680K (with ~5% absolute accuracy improvement). Overall, the performance trend is going up with data scale increasing, indicating that the larger data scales benefit abstraction.
Generie pre-training is critical for the emergence of abstraction. We probe the abstraction capability of randomly initialized T5-Base and GPT2-Medium (i.e., without loading pre-trained checkpoints) and report the results in Table 5 The poor performance on A + reveals that without generic pre-training, these deep learning models can hardly extract transferable abstract concepts from task A, even though they can still achieve >98% dev set performance on task A by fitting some task-specific suffer patterns. The comparison of the results in Table 2 and Table 5 demonstrate that the broader background pre-training is critical for the emergence of abstraction capability.
CONCLUSIONS
In this paper, we introduce a systematic probing framework from a transferability perspective to guide the design of probes for abstraction capability. We instantiate this framework grammar probe and show strong evidence that two probed PLMs have the abstraction capability. We further analyze this probed capability by investigating several in-depth questions and provide insightful conclusions.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
sufficiently robust abstraction capability that can perfectly extract abstract concepts and exclude concrete information in any situation will help deep learning models avoid many potential risks of ethical issues such as social bias and privacy breaches. However, as investigated in this work, the abstraction capability of some commonly used deep learning models may be fragile and can be affected by their training situation. This suggests that the abstraction capabilities of these models are still not reliable enough to naturally avoid these potential ethical issues, and calls for future work to explore ways to strengthen the robustness of the abstraction capabilities of deep learning models.
ACKNOWLEDGMENTS
We thank all the anonymous reviewers for their valuable comments. This work was supported in part by NSFC under grant No. 62088102. We would like to thank Qian Liu for his valuable suggestions and feedback on this work.
REFERENCES
Alham Fikri Aji, Nikolay Bogoychev, Kenneth Heafield, and Rico Sennrich. In neural machine translation, what does transfer learning transfer? Association for Computational Linguistics, 2020.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4623 4637, 2020.
David Barrett, Felix Hill, Adam Santoro, Ari Morcos, and Timothy Lillicrap. Measuring abstract reasoning in neural networks. In International conference on machine learning, pp. 51 511-520.
PMLR, 2018.
Lawrence W Barsalou et al. Perceptual symbol systems. Behavioral and brain sciences, 22(4): 577-660, 1999.
Yaniv Benny, Niv Pekar, and Lior Wolf. Scale-localized abstract reasoning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12557-12565, 2021.
Steven Bird, Ewan Klein, and Edward Loper. Natural language processing with Python: analyzing text with the natural language toolkit. O' O'Reilly Media, Inc.". 2009.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv:2108 07258, 2021.
Erin M Burgoon, Marlone D Henderson, and Arthur B Markman. There are many ways to see the forest for the trees: A tour guide for abstraction. Perspectives Psychological Science, 8(5): 501-520, 2013.
Xudong Cao, David Wipf, Fang Wen, Genquan Duan, and Jian Sun. A practical transfer learning algorithm for face verification. In Proceedings of the IEEE international conference computer vision, pp. 3208-3215, 2013.
Xinyun Chen, Chen Liang, Adams Wei Yu, Dawn Song, and Denny Zhou. Compositional general- ization via neural-symbolic stack machines. Advances in Neural Information Processing Systems, 33:1690-1701 2020.
François Chollet. On the measure of intelligence. arXiv preprint xrXiv:11111.0111 2019.
Peter Clark, Oyvind Tafjord, and Kyle Richardson. Transformers as soft reasoners over language. arXiv preprint arXiv:2002.05887, 2020.
Róbert Csordás, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Are neural nets modular? inspect- ing functional modularity through differentiable weight masks. In International Conference on Learning Representations, 2020.
10

---

Published as a conference paper at ICLR 2023
Kahneman Daniel. Thinking, fast and slow, 2017.
Zoltán Dienes, Gerry TM Altmann, and Shi-Ji Gao. Mapping across domains without feedback: A neural network model of transfer of implicit knowledge. Cognitive Science, 23(1))53 82, 1999.
Mengnan Du, Fengxiang He, Na Zou, Dacheng Tao, and Xia Hu. Shortcut learning of large language models in natural language understanding: survey. arXiv preprint arXiv:2208.1 11857, 2022.
Atticus Geiger, Alexandra Carstensen, Michael C Frank, and Christopher Potts. Relational reasoning and generalization using nonsymbolic neural networks. Psychological Review, 2022.
Robert Geirhos, Jöm-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shorteut learning in deep neural networks. Nature Machine Intelligence, 2(1 (1):666-67777 2020.
Dedre Gentner and Jose Medina. Similarity and the development of rules. Cognition, 65(2-3): 263-297, 1998.
Fausto Giunchiglia and Toby Walsh. A theory of abstraction. Artificial intelligence, 57(2-3):323-389, 1992.
Ian Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empiri- cal investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv.1312.6211. 2013.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A Smith. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics:
Human Language Technologies, Volume 2 (Short Papers), pp. 107-112, 2018.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, al. Scaling laws for autoregressive generative modeling. arXiv preprint 2xXi:20010.14001 2020.
John Hewitt and Percy Liang. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (ENNNL- pp. 2733- 2743, 2019.
John Hewitt and Christopher D Manning. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers),
pp. 4129 4138, 2019.
Sheng Hu, Yuqing Ma, Xianglong Liu, Yanlu Wei, and Shihao Bai. Stratified rule-aware network for abstract visual reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 1567-1574, 2021.
Clark Hull. Quantitative aspects of evolution of concepts: An experimental study. Psychological monographs, 28(1):i, 1920.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: diagnostic dataset for compositional language and elementary visual reasoning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1988-1997. IEEE Computer Society, 2017.
Ronald Kemker, Mare McClure, Angelina Abitino, Tyler Hayes, and Christopher Kanan. Measuring catastrophic forgetting in neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, et al. Measuring composi- tional generalization: A comprehensive method on realistic data. In International Conference on Learning Representations, 2019.
11

---

Published as a conference paper at ICLR 2023
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. Unifiedqa: Crossing format boundaries with a single qa system. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1896-1907, 2020.
Najoung Kim and Tal Linzen. Cogs: A compositional generalization challenge based on seman- tic interpretation. In 2020 Conference Empirical Methods in Natural Language Processing, EMNLP 2020, pp. 9087-9105. Association for Computational Linguistics (ACL), 2020.
Philipp Koehn. Europarl: A parallel corpus for statistical machine translation. In Proceedings of machine translation summit x: papers, pp. 79-86, 2005.
George Konidaris. On the necessity of abstraction. Current opinion in behavioral sciences, 29:1-1 2019.
Sreejan Kumar, Ishita Dasgupta, Raja Marjieh, Nathaniel D Daw, Jonathan D Cohen, and Thomas L. Griffiths. Disentangling abstraction from statistical pattern matching in human and machine learning. arXiv preprint arXiv:2204.01437, 2022.
Yuxuan Lai, Chen Zhang, Yansong Feng, Quzhe Huang, and Dongyan Zhao. Why machine read- ing comprehension models learn shortcuts? In Findings of the Association for Computational Linguisties: ACL-IJCNLP 2021, pp. 989-1002, 2021.
Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of cerpuuennc-tosssqueeee recurrent networks. In International conference on machine learning, pp. 2873-2882. PMLR, 2018.
Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
Yongjie Lin, Yi Chern Tan, and Robert Frank. Open sesame: Getting inside bert's linguistic knowl- edge. In Proceedings of the 2019 ACL Workshop BlackboxNL.P: Analyzing and Interpreting Neural Networks for NLP, pp. 241-253, 2019.
Chenyao Liu, Shengnan An, Zeqi Lin, Qian Liu, Bei Chen, lian-Guang Lou, Lijie Wen, Nanning Zheng, and Dongmei Zhang. Learning algebraic recombination for compositional generalization. arXiv preprint 21Xi:22077.0616 2021a.
Qian Liu, Shengnan An, Jian-Guang Lou, Bei Chen, Zeqi Lin, Yan Gao, Bin Zhou, Nanning Zheng, and Dongmei Zhang. Compositional generalization by learning analytical expressions. Advances in Neural Information Processing Systems, 33:11416-11427, 2020.
Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, and Jian-Guang Lou. Tapex: Table pre-training via learning a neural sql executor. In International Conference on Learning Representations, 2021b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018.
Tyler Millhouse, Melanie Moses, and Melanie Mitchell. Foundations of intelligence in natural and artificial systems: workshop report. arXiv preprint arXiv:2105.02199 2021.
Melanie Mitchell. Abstraction and analogy-making in artificial intelligence. Annals of the New York Academy of Sciences, 550 2021.
Timothy Niven and Hung Yu Kao. Probing neural network comprehension of natural language arguments. In 57th Annual Meeting of the Association for Computational Linguistics, ACL 2019, pp. 4658 4664. Association for Computational Linguistics (ACL), 2020.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP.IICNIP)), pp. 2463-2473, 2019.
12

---

Published as a conference paper at ICLR 2023
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Gen- eralization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:220.002177, 2022.
Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang. Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel Bowman. Intermediate- transfer learning with pretrained language models: When and why does work? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pp. 5231-5247, 2020.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 18))9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67, 2020.
Vinay Venkatesh Ramasesh, Aitor Lewkowycz, and Ethan Dyer. Effect of scale catastrophic forgetting in neural networks. In International Conference on Learning Representations, 2021.
Ryokan Ri and Yoshimasa Tsuruoka. Pretraining with artificial language: Studying transferable knowledge in language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7302-7315, 2022.
Kyle Richardson, Hai Hu, Lawrence Moss, and Ashish Sabharwal. Probing natural language infer- ence models through semantic fragments. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 8713-8721, 2020.
Laura Ruis, Jacob Andreas, Marco Baroni, Diane Bouchacourt, and Brenden M Lake. A benchmark for systematic generalization in grounded language understanding. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 19861-19872. Curran Asso- ciates, Inc., 2020. URL htttps://prooceeinngg neurips cc cc/papprr/222//EE file/ pdd
e5as0182cc8 hat.cc/aeeeeeaeee766eeeeeeeeeee0
Priyanka Sen and Amir Saffari. What do models learn from question answering datasets? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing 2020.
(EMNLP), pp.
Radhika Shivhare and Ch Aswani Kumar. On the cognitive process of abstraction. Procedia Computer Science, 89:243-251 2016.
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint arXiv: 703.00810, 2017.
Alon Talmor, Oyvind Tafjord, Peter Clark, Yoav Goldberg, and Jonathan Berant. Leap-of- thought: Teaching pre-trained models to systematically reason over implicit knowledge. Advances in Neural Information Processing Systems, 33:20227-20233 2020.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang. Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. What do you learn from context? probing for sentence structure contextualized word representations. In International Conference on Learning Representations, 2018.
Pedro A Tsividis, Joao Loula, Jake Burga, Nathan Foss, Andres Campero, Thomas Pouncy, Samuel J Gershman, and Joshua B Tenenbaum. Human-level reinforcement learning through theory-based modeling, exploration, and planning. arXiv preprint arXiv:2107.12544 2021.
Ivan Vulié, Edoardo Maria Ponti, Robert Litschko. Goran Glavaš, and Anna Korhonen. Probing pretrained language models for lexical semantics. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pp. 7222-7240, 2020.
Yingxu Wang. Concept algebra: a denotational mathematics for formal knowledge representation and cognitive robot learning. Journal of Advanced Mathematics and Applications, 4(1):61-86, 2015.
13

---

Published as a conference paper at ICLR 2023
Alex Warstadt, Yu Cao, Ioana Grosu, Wei Peng, Hagen Blix, Yining Nie, Anna Alsop, Shikha Bordia, Haokun Liu, Alicia Parrish, et al. Investigating bert's knowledge of language: Five analysis methods with npis. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNI.-IJCNIP), pp. 2877-2887, 2019.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud. Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.0768) 2022.
Thomas Wolf, Julien Chaumond, Lysandre Debut, Victor Sanh, Clement Delangue, Anthony Moi, Pierric Cistac, Morgan Funtowicz, Joe Davison, Sam Shleifer, et al. Transformers: State-of-the-an art natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, 2020.
Wlodek W Zadrozny. Abstraction, reasoning and deep learning: study of the" look and say" sequence. arXiv preprint wtXiv:2109.12755, 2021.
Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song- Lang--hunn Zhu. Raven: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5317-5327, 2019,
14

---

Published as a conference paper at ICLR 2023
This is the Appendix of the paper Does Deep Learning Learn to Abstract? A Systematic Probing Framework.
A DISCUSSIONS
Below are more discussions about our work.
Potential factors that may hinder our probing. We consider the main factor that could block the of our probing framework is the catastrophic forgetting problem in deep learning (Goodfellow ct al., 2013 Kemker et al. 2018). Since our probing framework relies on the transferability property of abstract concepts, if catastrophic forgetting dominates the learning of downstream tasks, such transferability will hardly take effect and the probing results will fail to reveal the abstraction capabilities. Considering this problem, we utilize the early-stopping strategy (detailed in Appendix) to alleviate catastrophic forgetting. Moreover, our tested pre-trained models are naturally more robust to catastrophic forgetting (Ramasesh al. 202I).
Better understanding "why does transfer learning work". Recent success of transfer learning shows that pre-training (or continue pre-training) with similar source tasks can help better solve downstream target task (e.g., question answering (Khashabi et al.) 22000] Liu ct al. 2021b), face veri- fication (Cao et al. 2013), and general NLU tasks (Pruksachatkun al. 2020p). Some previous work in cross- lonsslinguullli transfer learning empirically observed that the model can transfer some knowledge
beyond vocabulary (Artetxe et al.) 2020 Ri & Tsuruoka (022). but they did not consider to exclude the influence from other potential factors. Our results can serve stronger evidence for the reason to the success of transfer learning, that in addition to transferring some surface patterns, the better target performance can also benefit from similar abstract concepts learned from source tasks.
Limitations and future work. The main limitations in this work are 1) we do not quantify the abstraction capability and 2) only test two large pre-trained models. We leave these two points to our future work. Another future direction to further explore the mechanisms behind abstractions.
B COMPARISONS WITH PREVIOUS FINDINGS ABOUT LEARNING DYNAMIC
Comparison with information bottleneck. Shwartz-Ziv & Tishby (0117) found two-phase learn- ing process from the view information flow in deep neural networks: empirical error minimization phase and representation compression phase. This process is different from the memorize-ther abstract process since they measure the training dynamics in quite different perspectives. The former focuses the compression of representation (and reduction of mutual information) while the latter portrays the learning of abstract concepts. The analogy between the two may lie in that the ex- traction of abstract concepts from concrete instances in some way have the effect of information compression.
Comparison with Grokking. Power al. 2022) revels that the improvement in generalization (on validation set) can happen well past the point of over-fitting (on train set). Both 'grokking' and mnemoriz-then--asttaet phenomenon indicate that some general patterns are always learned in later training stage. The difference is that the 'grokking" focuses generalization beyond over-fitting training data, while "memorize-then-abstee portrays the transfer of abstract concepts beyond task-specific characteristies.
C PRINCIPLES FOR DESIGNING PROBING TASKS
To verify whether the model could learn abstract concepts from task A by assessing A(A = B), we propose the following principles for designing task B:
1) The space specifc characteristics of B should be very different from that of so that the memorization of surface patterns in A is helpless to B.
2) The abstract concepts of B should be the same as that of A so that the abstraction on A could be reflected with a better performance on B.
3) The fine-tuning-on approach ft B should be not enough to lear task B perfectly; otherwise, the gain from the abstraction on A would not be noticeable.
15

---

Published as a conference paper at ICLR 2023
Furthermore, to verify that the performance gain on B is from the abstraction on A rather than other factors, we consider a contrast task of A:
4) The abstract concepts of C should be very different from A (also B), while other latent factors are kept the same such as data scale and pre-training steps.
The consideration of contrast task is similar selectivity Hewittt & Liang. 2019)
D AN OPERATION PROBE
The operations semantics (e.g., conjunction) in the Boolean algebra can be regarded as transition functions between the given Boolean variables and corresponding outputs. We want to examine whether the model can learn the meaning of operations from concrete logical expressions, or just learn superficial correlations from specific sketches in expressions.
In operation probe, we instantiate the framework with logical expression evaluation (LEE) tasks. We consider operation semantics in logical expressions as abstract concepts, and surface patterns (e.g., local string matching) are bounded with operation sketches.
Figure 7 shows two kind of sketches: chain sketch and tree sketch. The model trained on chain sketch may learned the meaning of operations (e.g., conjunction of two Boolean variables) or simply memorize some head or tail patterns of strings (e.g., the head of input string is "False AND (", the output is always "False"). Learning operation semantics can more generally help understand other expressions with different sketches, but memorizing head or tail patterns in chain sketches is helpless even harmful to understand tree sketches, since these surface patterns lead to wrong results in different sketches. We give a more specific definition of abstraction based on LEE tasks:
LEE task L : Eg + B3 that the source logical expressions E, (with operation semantics P. and sketch K,) are evaluated as Boolean values B,, and set of input- output pairs I. = {(e b in which et, is logical expression sampled from E, and b; (True, False) } is the evaluation result of e,. the abstraction capability is learning the meanings of operations IP, during training on T. rather than memorizing sketch-specific patterns that are bounded with K,
Definition 2: Considering
Our LLE tasks probe the learning of four logical operations: P = -CConjunction (Conj.), Discon- junction (Disc.), Alternative Denial (Alt.), Joint Denial (Joi.)). Figure 7b) illustrates the transition functions of these operations.. For generating data, each operations in P is constantly aligned with one operator (i.e.. a concrete string) in S,, Examples of these sets are contained in Appendix F2
Train set A. We synthesize the data in A with P+ and chain sketch. Each expression contains eight operators which are sampled from S..
Transfer set ß and Test set B. We synthesize the data in B U B with Pt and tree sketch. Each expression contains eight operators which sampled from S., When probing a certain operation P. E Ptt we limit that the expression in B does not contain p., while each expression in B must contain P To make the model familiar with operators S, or not forget them during further fine- tuning, we supplement Ô with 100 two-operator expressions which cover the full S,, Empirically, DNN models can be easily influenced by the statistical bias in label distributions, we balance the 'True' and 'False' examples during sampling.
Contrast set C. The operators and sketch in C are kept the same with A, but each operator in S. aligned with another set of logical operations P-- = Maaterial Implication, Converse Implication, Material Non- coniimliication, Converse Non- con-implication) ). The transition functions of these operations are listed in Appendix F2
Results of operation probe shown in Table 6
16

---

Published as a conference paper at ICLR 2023
Altemative Denial
oint Denial
Choin Sketch
Conjunction
Disjunction
OP OP: OP: OP4
Tree Sketch
OP:
1
0
OP
OP: OPs
Right
(a) Two kind sketches.
(b) Transitions of operations.
Figure 7: Four operations and two sketches in our operation probe. (a) shows the chain sketch used in task A and C, and the tree sketch used in B U B. Each 'OP" represents one operation. (b) shows the transition results of four operations with different left and right Boolean variables.
Table 6: Results our operation probe.
Control Exp
Main Exp
Contrast Exp
Probe
Model
Sub Probe
C H
Avg.
70.1
(+.9.8)
71.2 (+1.1)
Conj.
63.3
90.8
65.3
TS
Dise:
58.9
84.8
72.8
Alt.
79.1
90.7
72.6
Operation
Joi.
79.2
89.2
74.3
Avg.
64.7
76.3 (+1116)
64.0 ( (-0.7)
Conj.
78.6
61.1
GPT2
Disc.
59.8
72.5
64.1
Alt.
78.6
65.4
Joi.
75.3
65.3
E ABSTRACT CONCEPTS ARE MODULARIZED IN PLMS
To supplement our analysis on abstract attention heads, here we provide our detailed explorations to identify the modulars in PLMs that store abstract concepts in our probes. Our explorations based on the following property and assumption.
Forgetting of Abstract Concepts. Consider the Main Exp A = B that task A and B share the abstract concepts but do not share task-specific knowledge. After fully fine-tuning on task B, the model's parameters will somehow over-fit the task-specific knowledge in task B and the abstract concepts stored in these parameters will be partially forgotten.
Property:
Assumption: Identifying the Modularization of Abstract Concepts by Freezing Parameters. If module in the model individually store a part of abstract concepts, these parameters can be directly reused in tasks without further fine-tuning. Furthermore, considering the property above, freezing this abstract module can avoid the forgetting of abstract concepts, resulting in the performance improvement Main Exp + B.
Based on this assumption, to identify whether abstract concepts are modularized in some parameters, we partially freeze the model in a coarse-to to-fiee manner. The following experiments are conducted on one of the three pre-training terminal sets. First, we freeze each layer in the model, showing in Figure 8 We find that the last layer encoder and the first layer in decoder modularize part of abstract concepts. Furthermore, we freeze these two layers in our Contrast Exp C' + B and find no improvements Figure Da indicating that the improvement of freezing these two layers comes from keep abstract concepts.
In next step, we try to identify the abstract concepts are stored attention layers or FF layers. We separately freeze the attention sub-layer and FF sub-layer in the last encoder layer and first decoder layer. Figure (huurre9bhshae shows that attention layers take more responsibility for storing abstract concepts.
Then, analyze whether these abstract concepts are modularized in some attention heads averaged in the whole attention layers. Our investigation in Section 6.2 finds that the abstract concepts are centralized in some middle-layer attention heads. Based the results in Figure 4 freeze the top
17

---

The full sets are contained our Published as a conference paper at ICLR 2023
36 heads to further verify that they responsible for storing abstract concepts. Results in Table indicate that abstract concepts are modularized in this small part of attention heads.
baseline
Accuracy (%)
Frozen Layer
Figure Freeze 24 layers separately in A + B.
Accuracy (%)
Accuracy (%)
Frozen Layer
Frozen Layer
(a) Freeze layers C' B.
Separately freeze attention and FF B.
Figure 9: Further explore the modularity in the middle two layers.
Table Supplement to Table
Freeze set of abstract heads and fine-tune.
Fine- Fine-Tuning
Baseline A
B
92.8
+Freeze Abstract Heads
96.6 (+3.8)
+Freeze Random Heads
92.9
-Prune Abstract Concepts
13.6
Baseline C + B
13.9
+Freeze Abstract Heads
14.4
F EXAMPLES IN OUR DESIGNED TASKS
Here, we list some examples in our designed probing tasks. Supplementary Material.
F.1 GRAMMAR PROBE
Table g and 9 shows examples in Com, and Mod. sub-probes in grammar probe, respectively.
F.2 OPERATION PROBE
Table I0 lists the operation behind each symbol, and Table II shows examples in Conj. sub-probe in operation probe. Figure I0 illustrates the transitions operations used in contrast task in operation probe.
F.3
REDUNDANT DESIGNS
For Redundant G we supplement the second T-Production rule in Table [ as "sub direct-obj / indirect-obj Sadj S.". Terminals in Sadj regarded as adjectives for S., These terminals
18

---

Published as a conference paper at ICLR 2023
Table 8: Examples in different sets in Com. sub-probe in grammar probe. The terminal NONE in target side is omitted to more clearly show the structure of the target example.
Set
Side
Example
Train Set
Source
Soke incurve huave the soon upon huave ban bibb huave acetum goladar INCURVE SOKE) ) LG UPON (SOON) LG BAN LG GOLADAR (ACETUM) )
Target
Transfer Set
Source
Emara liked that girl
Target
LIKE ( FMMA ) CCOMP SFT ( GIRI.)
Test Set
Source
admired that Daniel ADMIRE ( EMMA CCOMP LIKE DANIEL) CCOMP MEAN ( (JAMES) CCOMP FREEZE ( (LION)
Target
Source
Soke incurve huave the soon upon huave han hihh huave acetum goladar
Contrast Set
Target
ACETUM GOLADAR LG BAN BIBB LG SOON UPON LG SOKE INCURVE
Table 9: Examples in different sets in mod. sub-probe in grammar probe.
Sct
Side
Example
Source
Safe above the poddy cord the soon pial.
Train Set
Target
CORD (ABOVE SAFE, PODDY ) PIAL, SOON
Source
Emma ate the ring beside a bed EAT EMMA. BESIDE RING BED) ), NONE) )
Transfer Set
Target
Source
The baby
in the house screamed.
Test Set
Target
SCREAM ON BABY IN TRAY HOUSE)) NONEE
Source
Safe above the poddy cord the soon pial
Contrast Set
Target
(SOON. PIAL, PODDY, SAFE ABOVE) CORD
only contained in the source side and no terminals in target side aligned with them. Table 13 shows an example in Redundant G
F.4 DERIVATIONS
Here show examples of different derivations of original grammars. Table 12 shows examples in different G7. The terminal 'NONE' is omitted.
G MORE EXPERIMENTAL RESULTS
We present additional experimental results to supplement our probing and analysis.
G.1 DATA SCALE AND TERMINAL DIVERSITY
Figure 11a shows the effects from data scale for different model sizes. It shows that the performance improves marginally and unstably when the data scale increases from 1.7K to 680K instances. Moreover, it seems that the performance gap between models with different sizes is still considerable when the data scale is enough large.
Figure I shows the effects from data diversity for different model sizes. Here, we consider the terminal diversity as perspective of data diversity, i.e., the number of terminals of the grammar. Following Section 4 we only change the number of terminals in S. and Sc, increasing from 1 to 128. The overall trend is that the performance improves marginally and unstably when the diversity increases. Interestingly, we observe that for all three models, their performances achieve the peak before rising to 128 terminals and then keep oscillating. We speculate that their performances are bounded by the limited data scale, as we control the data scale as 34K instances when increasing conduct another experiment T5-Base that
the terminal diversity. To examine our speculation, pre- training on 680K instances with 128 terminals, achieving an average accuracy rate of 93.5%. This performance is higher than the result after pre-training (89.2%) and higher than the best average accuracy of T5- Base in Figure [ (88.4%), suggesting that higher data diversity should be equipped with larger data scale.
680K instances with 32 terminals
19

---

Published as a conference paper at ICLR 2023
Table 10: Symbols and operations in different tasks in operation probe.
Symbol
Operation in A and B
Operation in C'
al
Conjunction
Material Non- Kon-implication
Alternative Denial
Material Implication
c3
Disjunction Joint Denial
Converse Non-implication
d4
Converse Implication
Table 11: Examples in different sets in operation probe.
Set
Side
Example
Train Set
Source
False c3 (( False al (( False b2 ( ("True d4 True d4 True > d4 Trre ) d4 False ) b2 False )
Target
True
Source
(( False c3 ( False b2 False b2 True ) d4 ("Tuue b2 (( False b2 False b2 Falsc
c3 True
Transfer Set
Target
True
Source
(Tru b2 (( Falsc al False ) d4 False ) al Falls c3 ((( Truc al True ) c3 True ) al True
Test Sct
Target
False
Source
False c3 (( False al (( False b2 (TTue d4 True d4 True d4 True ) d4 False b2 False
Contrast Set
Target
False
G.2 MULTI-GRAMMAR PRE-TRAINING
Before this section, we consider the setup in which we only see one pair of input and output grammars during pre-training. This section explores whether multi-gramma, pre-training would influence the model to exhibit abstraction. Here, we consider two cases: can or can not access the golden grammar. The golden grammar is the grammar used in the downstream task. For the nulli-grammar pre-training, assemble different target grammars in Section 6.3 while keep the source grammar. During generating pre-training data with more than one target grammar, prefix (chosen from original, coarse, localreverse, tokens, guiding the model which target grammar should use. Table [4 shows the results with different ensemble grammars.
each instance, we add and reverse) at the beginning of the source
First, consider that we have no access to the golden grammar. We take the target grammar that performs the best in Section 6.3] Coarse G;, as the single-gramman baseline. Table 14 shows that augmenting the Coarse G with other target grammars can always perform better than the single grammar. Even augmenting with the Reverse G from the contrast task can bring slight gain (1.4% accuracy). It indicates that even though the model has no access to the golden abstract concepts, increasing the diversity of abstract concepts can make the model better aware of the existence of abstract concepts. This awareness can be regarded
higher level of abstraction capibility.
Then, considering that the model has access the golden grammar, the downstream task performance is lower than only pre-training on the golden grammar (accuracy 88.2 88.2%). Therefore, augmenting other similar abstract concepts would confuse the model and make hard to choose which concepts should be used for the downstream task.
G.3 INCREASE TOTAL NUMBER OF TERMINAL.S
We increase the total number of terminals to 1,500 in our probing tasks and report the results with T5-Base in Table 15 These results are similar to the original results in Table 2 and are still in line with our two expectations.
H FUZZY GRAMMAR
The abstract concepts discussed in grammar probe in Section 4 can be concretely defined, but in many application scenarios, abstract concepts can be fuzzy (e.g.. natural language grammar). Here, want to examine whether models can learn fuzzy grammar just can recognize the concrete one.
20

---

Published as a conference paper at ICLR 2023
Material Non-Implication
Mattrrial Implication
Converse Non-implication
Converse Implication
1
Lett
1
1
1
Right
Figure 10: Transitions of operations used in contrast task operation probe.
Table 12: Examples in different G7.
Derivation
Target Example
Coarse
CLAN OG INCURVE () LG UPON
LocalR
BAN ) CLAN LG SOKE INCURVE LG BIBB, GOALDER, SOON UPON
Nested
CLAN [BAN, LG INCURVE (SKKE, LG UPON SOON, GOALDER, BIBB )
We take natural language sentences for experiments and design different sets by mimicking the Com. sub-probe in our grammar probe.
Data in our fuzzy grammar probe is taken from Europarl v7 Koehn (2005): large parallel corpus for machine translation. For the probing on natural language data, we can not guarantee to satisfy the requirements in our framework perfectly, the grammar of the natural language is hard to be controllable as the formal language. We describe the instantiation of different sets as following.
Train set A. We take the German-to-Freench (De-Fr) sentence paris A.
Transfer set B and Test set B. We take English-to-Romaniie (En-Ro) as the probing task B. As both German and English are belong to the West Germanic language branch while both French and Romanian are belong to Italic branch, the abstract grammars used in De-Fr and En-Ro have some similarities. To satisfy that the f B performs poorly, we limit the B with only short sentences (15.7/13.8 words in En/Ro sentences in average) while B with only long sentences (78.0/74.4 words in En/Ro sentences in average). It means that the model can learn most of the lexicons from B but can not be aware of the grammars of long sentences.
Contrast set C. Mimicking the construction of C in the formal language task. also reverse the word order the target language of A.
Table 4 shows the performance of two models natural language data. These results indicate that fuzzy grammar in natural language data can also be learned and transferred by the two PLMs. In addition, this setting can also be regarded length generalization problem, the low A(C B) further confirm that our probing results benefit from learning abstract concepts rather than surface patterns (i.e., length distribution).
I TRY TO MEASURE ABSTRACTION CAPABILITY
As mentioned in Section that one limitation in our probing is the lack of metric that can quantita- tively measure the abstraction capibility. Thus, we not compare the abstraction capibility of two models with different architectures. Here, we try to design such a metric to compare the abstraction capibility of T5- T5-Base, with 220M parameters, and
with 345M parameters.
In the beginning, we want to clarify that this metric is just for primary exploration, as it is based on a strong assumption that not be satisfied in all situations.
Assumption: We assume that the performance score of a certain task (such as accuracy and BLEU score) can linearly reflect the ability of the model to solve this task.
means that, for instance, improving the accuracy from 90% to 100% is not harder than improving from 40% 50%. Apparently, this assumption does not suit all tasks and performance scores (even
21

---

Published as a conference paper at ICLR 2023
Table 13: Examples in Redundant Cg. Gray terminals are redundant that would not be mapped to targets.
Derivation
Source Example
Redundant
A angry pial was ozophen to zogo by odd dermestes.
100
Accuracy (%)
80
Accuracy
60
40
20
15.5mal
20
0
10
10
100
10
10a
Number of Instances
Number of Terminals
(a) Data Scale
(b) Data Diversity
Figure 11: More results for different data scales and data diversity.
the tasks and scores in probing). But it does not influence the comparison between T5-Base and GPT2-Medium. The reason will be discussed later.
Intuitively, we consider the contribution of abstract concepts overall performance measure of abstraction capibility, that is,
MoA scorea
(2)
scorer
in which the MoA means the Metric of Abstraction, score; means the full performance score on certain task without limiting the training data, and score, means the part of the score contributed by the abstract concepts. Following our probing framework, we consider the score, as the relative gain from ft B to A + B. Furthermore, considering the influence of other factors which is reflected by + B, we design the score, as:
score, = score(A B) max[score((1 B), score(C + B)],
(3)
in which score() represents the performance score of a certain procedure, and max(score(( B), score(C' = B)] means to choose the maximum performance score between ft B and C = B. For the full performance score in the denominator in Equation 2 we evaluate the model performance on B after (only) fine-tuning on the full set B, which sampled in the same distribution B (rather than limited distribution of B). We denote this procedure as 1 B. Thus, the metric in Equation can be formalized as:
MoA score = B) maxx(soree(f) scoser = B)]
score(1 B)
Table 16 shows MoA for two models on grammar probe and fuzzy grammar probe, and lists the scores required calculate MoA. On each task, MoA T5-Base is higher than that of GPT2-Medium. Furthermore, during calculating MoA, the baseline score max[score(11 B), score B)] of T5- Base is always higher than that of GPT2-Medium. As it is harder for the model to improve the accuracy and BLEU scores these tasks from relatively higher baseline, MoA just under- estimate the abstract ability of T5-Base. Therefore, we can roughly conclude that the abstraction capibility of TS-Base higher than GPT2-Medium.
COMPARISON WITH PREVIOUS NEGATIVE RESULTS
Some previous work demonstrated that neural models could not learn abstract concepts (Liu et al. 2020: Chen et al. 2020 Liu al.) 2021a| Chollet 2019: Mitchell, 2021 Zadrozny 2202 Our probing results shed some light that neural models, especially PLMs, exhibit abstraction capibility to some extent. Compared with previous work, two points could lead to different conclusions.
22

---

Published as a conference paper at ICLR 2023
Table 14: Downstream performance under different multi grammar settings.
Grammars
Accurncy
Reverse
Nest
LocalR
Coarse
41.6
43.0
Without Golden Grammar
46.4
54.6
55.3
With Golden Grammar
74.0
Table 15: Increase total number of terminals.
Model
Sub-Probe
Control Exp ft B
Main Exp A = B
Contrast Exp
C B
Avg.
18.1
69.8 (+51.7)
15.9 (-2.2)
T5
Com,
21.9
83.9
15.2
Mod.
14.2
55.7
16.5
The first point is the probing methodology. In all works (including ours), the basic idea of probing abstraction is to separate it with memorization. To implement this idea, previous work has almost always involved designing special probing task in which memorization of the train set is helpless to solve the test set. However, such an implementation constraints the generation of the train set, which could bring some biases or limitations in training data. To overcome these biases or limitations, the model should have some other abilities more than abstraction, such as reasoning and systematic generalizability. Therefore, the previous disappointing results may have been caused by the lack of other abilities rather than abstraction.
The second point is the test model. Some previous work probed the vanilla Transformer or LSTM while we take the pre-trained language models. We suppose that the model may acquire better abstraction capibility from the pre-training corpus, and can better exhibit this ability with larger model sizes.
K DETAILS OF EXPERIMENTS
K.1
DATA
We show more details about the sets described in Section including data scales, average input lengths and average output lengths.
For the target side grammar of our formal language tasks, mentioned in Section that change the original target grammar of COGS to be chain-structured. In Table 18 we list some examples with the original target grammar and the new chain-structurre grammar. First, to distinguish the input and output tokens, we capitalize all output tokens (e.g.. from "rose" to "ROSE"). Second, we replace the variables (e.g., "x 1") in the original grammar with its corresponding terminals (e.g.. "ROSE"). Then, we group the terminals of AGEN'T (e.g.. "DOG"), THEME (e.g.. "ROSE") and RECIPIENT with their corresponding terminal of PREDICATE (e.g., "HELP") and combine this group of terminals in a function format, i.e., "PREDICATE (AGENT, THEME, RECIPIENT)" )" If the predicate is not equipped with an agent, theme or recipient in the original grammar, the corresponding new cen-terminals (i.e., AGENT, THEME and RECIPIENT, respectively) in the function format above will be filled with the terminal NONE (e.g.. "HELP ( DOG, ROSE, NONE )*)) For simplicity, we omitted NONE in [able[]] Table & and Table [1], Such function format is the minimum unit of CLAUSE Finally, each CLAUSE concatenated with another CLAUSE by the terminal CCOMP (e.g., "HOPE (LIAM, NONE, NONE CCOMP PREFER (DOG, NONE, NONE )").
K.2 PROCEDURE
Training Each pre- pre-trainin takes 100,000 steps, and the final-step checkpoint is used for fine-tuning. Each fine-tuning takes 100,000 steps, and the checkpoints for every 10,000 steps are saved.
23

---

Published as a conference paper at ICLR 2023
Table 16: MoA of two models both grammar probe and fuzzy grammar probe and the scores required to calculate MoA. T5 and GPT2 are T5-Base Base and GPT2-Medium respectively.
Prohe
Score
Model
Scurcs of Procodurcs
MoA
18
C
1
T5
88.2
23.1
15.4
95.7
0.68
Grammar
Accuracy (%)
GPT2
48.2
1.9
26
93.2
0.49
Fuzzy Grammar
BLEU Score
GPT2
21.0
16.4
11.6
42.2
0.11
Table 17: Data scales, average input lengths, and average output lengths of different sets in probing.
Das Scale
Ang Lapa
Ave Ocaps
Daaa Scal:
Ava lopai
Avy Oapps
Dea Seale
Ava lapei
Avg Caaper
Dea Seale
Avg Seper
Arg Chapet
36.315
16.
36.9
34,155
9.5
1,53
314
30,15
168
Opcsabs:
100000
163
30000
1800
144
198,03
16X
Evaluation We take an early-stopping strategy in our evaluation to avoid catastrophic forgetting. First, each checkpoint saved during fine-tuning is evaluated on the held-out dev set. We choose the first checkpoint that achieves the best dev score testing. For formal language tasks, we utilize constraint decoding strategy that the model can only generate the words in the vocabulary.
Compute and Resources We majorly use Tesla-V V100--100001000 GPUs for training and evaluation, except for the experiments T5-Large GargeLarge, which require Tesla- Tesla-V1004 V100-32GB GPUs. On average, one pre-training takes 15 GPU hours, one fine-tuning takes 155 GPU hours (including saving checkpoints), and testing takes ~2 GPU hours (as test
very long).
K.3 HYPERPARAMETERS
Hyperparameters used for training and testing are listed in Table 19
K.4 DEFINITION OF PERPLEXITY (PPL)
Th following equation explain how to calculate PPL Equation !
PPI.(E))(: expl-
log po (1/2 1 12)))
K.5 RESULTS
We list the detailed results that are plotted in the figures (i.e., Figure 5 and Figure 6 including the average scores, minimum scores, maximum scores, and standard deviations for all replicate experiments.
24

---

Published as a conference paper at ICLR 2023
Table 18: Examples with the original grammar and the chain- structured grammar
Original Tageet Greanes
Cleie Stuctard Target Grarate
IAND 3. captain Loaa. ogent 1) liges 4o: oegee 1.1mm)) Ahph commp (s 1AAN puikor agert
dog
ROSE YONE)
EAT CAPTION, NONE, NONE
LEAM, NONE, NOVE, EOCOM (1OO, NONI, NONE)
Table 19: Hyperparameters for training and testing.
Grammar Probe
Operation Probe
T5
GPT
T5
GPT2
Learning Rate
le-5
le-5
le.4
le-4
Weight Decay
0.01
0.01
0.01
0.01
Lahel Smooth
0.1
0.1
0.1
0.1
Max Input
1024
1024
Output Len
1024
Max Total Len
1024
1024
Beam Size
5
Table 20: Detailed results for Figure
Input Grummar LocalR
Output Grammar LocalR
Original
Redundant
Nest
Reverse
Original
Coarse
Nest
Reverse
Avg
88.2
86.0
76.2
55.5
15.0
88.2
41.6
33.5
29.1
15.4
Min
72.8
68.3
70.4
49.1
13.0
72.8
34,5
19.8
27.7
12.1
Max
96.9
95.6
80.6
61.1
20.1
96.9
50.8
38.7
31.7
19.2
8.5
10.2
4.2
4.3
8.5
4.9
6.7
1.3
2.4
Table 21: Detailed results for Figure 6a and
Exp
T5-Large
GPT2
GPT2-Medium
GPT2-Large
15-Base
T5-Large
GPTZ-Medium
GPTZ-Lare
Avg
Min
72.8
23.9
47.2
3.0
22.6
1.1
Max
47.4
96.9
97.2
12.4
63.8
85.7
6.0
23.5
21.1
0.2
299
1.6
Sid
9.8
8.5
4,9
4.4
14.5
13.7
3.0
0.5
1.7
0.1
09
0.3
Table 22: Detailed results for Figure 6c
(a) T5 L5-Large
Data Scale (T5-Large)
1.7K
3.4K
6.8K
I7K
34K
68K
170K
680K
Avg
90.4
96.7
96.7
92.3
94.0
98.8
97.6
98.9
Min
73.5
96.0
94.2
73.6
83.2
98.0
92.4
96.9
98.1
98.7
99.7
Std
10.5
0.9
1.7
10.8
4.9
0.5
3.0
L.1
(b) T5-Basc
Data Scale
(TS-Base)
1.7K
3.4K
6.8K
17K
34K
68K
170K
680K
75.1
85.0
88.2
87.0
89.2
Min
61.5
74.0
69.2
79.3
72.8
74,8
82.5
76.9
Max
90.8
88.7
94.4
97.2
96.9
92.3
94.4
Sid
10.0
4.9
6.3
8.5
6.3
3.8
7.6
(c) T5-Small
Data Scale (TS Small)
680K
Avg
28.3
31.9
26.1
27.3
34.4
40.5
38.7
39.4
29.7
Max
29.2
50.5
36.2
33.7
47.4
62.9
50.9
56.0
9.5
25

---

Published as a conference paper at ICLR 2023
Table 23: Detailed results for Figure 11b
(a) T5- T5-Large
Terminal Diversity (TS-Lage)
32
64
128
Avg
90.5
93.3
96.4
97.9
98.
1
94.0
96.9
94.2
79.7
97.1
93.2
87.3
Max
97.9
97.4
98.1
98.9
99.1
97.2
98.1
99.0
Std
6.9
3.9
0.9
1.0
0.7
4.9
1.7
4.6
(b) 35-Basc
Terminal Diversity (TS-Base)
16
32
64
128
Avg
66.5
72.5
76.3
85.3
88.4
88.2
86.4
84.2
Min
53.0
61.4
69.2
71.6
75.7
72.8
54.6
71.1
Max
79.4
86.1
92.2
94.1
96.4
96.9
93.7
97.5
T5-Small
Terminal Diversity (TS-Small)
128
Avg
Min
3.3
12.6
10.2
16.1
21.0
20.7
22.6
24.7
34.5
38.6
41.0
36.1
47.4
32.4
47.5
10.1
7.5
9.2
4.9
4.3
26


Reviews:
[{"review_id": "xa9hZewW_x", "reviewer": "Reviewer_VyFa", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper empirically investigates whether pretrained large language models can have abstraction capability --  whether the model can be aware of the grammar instead of memorizing surface word patterns, taking from a transferability perspective between tasks of different characteristics. The paper also reports the effect of the training dynamics on the abstraction. Under this scheme, interesting results were found:  (1) there is a \"memorize-then-abstract\" two-stage process in the training time; (2) several middle-layer heads capture the abstract concepts, and (3) scaling does help.\n", "summary_of_the_review": "A nice addition to the recent literature of large-language models, helping to explain their recent great performance on different settings.", "strength_and_weaknesses": "Strength\n======\n- An extensive and well-designed study.\n- Results are revealing. Limitations are articulated and partly addressed.\n- In-depth analysis is provided. The difference from previous negative results on the generalization of neural networks is briefly discussed.\n\nWeaknesses\n==========\n- The number of terminals is a bit small compared to real languages.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dnUjEp0kqD", "reviewer": "Reviewer_1xjY", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies whether language-pretrained transformer models can abstract grammatical structures and thereby transfer them across superficially unrelated domains. They use carefully controlled experiments, and show some strong transfer effects of grammatical structures, which increase with model scale. ", "summary_of_the_review": "Post-revision update\n---------------\n\nThe authors have addressed the three main weaknesses I listed in their follow-up experiments, and I believe they have improved the clarity of the paper. I have improved my score accordingly.\n\nOriginal review\n--------------------\n\nI think this paper is a potentially-compelling demonstration of a fascinating transfer phenomenon. I think this is a very interesting work that should lead to lots of follow-on investigations. I have some lingering concerns and suggestions noted above, that I hope the authors will address, and that I expect would increase the impact of the paper.", "strength_and_weaknesses": "Strengths:\n* The idea is interesting, and the experiments are fairly compelling.\n* The thoroughness and design of the experiments is a major plus. Comparing to the contrast and control conditions makes the main experiment results much more compelling.\n* I also appreciated the detailed evaluation of different conditions, exploring mutated grammars, analyses of head pruning, scaling, etc. There are a lot of fascinating details in this paper!\n\nWeaknesses:\n* It would be great to have a comparison of a model that was not pretrained on natural language first (e.g. T5 initial checkpoint or randomly initialized). That would help clarify the role of that initial pretraining in perhaps teaching the relevant structures of language (and might relate to why these results are different from other transfer results).\n* The claim that model scale improves transfer seems very clear. However, the claims that data scale and diversity improve performance seem weakly supported at best. I think the authors should demonstrate that these effects are statistically significant (which appears not at all true from Fig. 6c-d), or remove these claims.\n* In fact, the weakness of the data scale effect actually makes me more skeptical of the main results. How can 1.7k pretraining sentences have such a strong effect? How few domain A sentences would be needed to actually see a decrement? How can the model be learning abstract structures from such a small amount of data? It would be useful to clarify this (and/or to see if it breaks down when the scale is reduced farther).\n\nComments / notes:\n* There is some older connectionist work on the idea that neural networks could transfer abstract grammar structures across domains with superficially different features [Dienes et al., 1999]; and some more recent work on generalization of relational properties [Geiger et al., 2022]. They are of course far simpler than the present experiments, but I think these would be a nice piece of motivating context to add from the cognitive side. \n\nReferences\n------ \n\nDienes, Z., Altmann, G. T., & Gao, S. J. (1999). Mapping across domains without feedback: A neural network model of transfer of implicit knowledge. Cognitive Science, 23(1), 53-82.\n\nGeiger, A., Carstensen, A., Frank, M. C., & Potts, C. (2022). Relational reasoning and generalization using nonsymbolic neural networks. Psychological Review. (https://arxiv.org/pdf/2006.07968.pdf) ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "dhNzQue-G8", "reviewer": "Reviewer_dxgs", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel benchmark for evaluating whether neural networks have learned abstract concepts. This benchmark is applied to pre-trained language models, revealing evidence that they indeed can learn abstract concepts. The paper also presents a range of analyses assessing the factors that contribute to the emergence of abstract concepts.", "summary_of_the_review": "This paper provides an interesting analysis of the emergence of abstract concepts in pre-trained language models, with a set of well-controlled experiments and extensive analyses. The only major missing element is an analysis of the extent to which pre-training is necessary for the emergence of abstract concepts.\n\nUpdate after rebuttal: the authors have added new experiments demonstrating the importance of pre-training. This addresses my only major concern with the work, and I have updated my score accordingly.", "strength_and_weaknesses": "## Strengths:\n- The design of the probing framework includes a number of careful control experiments.\n- The analyses cover a wide range of factors that might contribute to the emergence of abstract concepts.\n\n## Weaknesses:\n- A key claim is that *pre-trained* language models, according to these results, show evidence of being able to learn abstract concepts. But the paper does not directly evaluate the importance of this pre-training (i.e. the generic pre-training that occurs prior to the experiment-specific pre-training in their probing framework). This should be straightforward to test, by simply using the same architectures (T5 and GPT-2), but training them from scratch on the probing framework training sets. I strongly suspect that the broader background pre-training is very important for this capacity, but it would be good to demonstrate this.\n\n### Other comments:\n- In the related work, the authors make a distinction between previous 'in-task' evaluations of abstraction vs. the proposed (between-task) evaluation. Are these really fundamentally different? Is there even a principled way to demarcate what constitutes a 'task'? It would be good for the authors to discuss this issue a bit more.\n- The 'memorize-then-abstract' phenomenon is interesting, but since the 'memorization' component of this phenomenon involves performance on a heldout (in-task) test set, it seems like 'interpolate-then-abstract' may be a better term, since it isn't strictly speaking memorization.\n- In the analysis of modularization, the paper states that 'we consider the first 36 heads in Figure 4', which are also described as 'abstract heads'. Does 'first' mean the heads with the highest DPC? The way it is phrased makes it sound as though the heads with the lowest value along the X axis were chosen, which would seem to be heads that are not important for abstraction.\n- In Figures 6a and 6b, it would be helpful to include the arrow notation (i.e. $A \\Rightarrow B$) in the legend.\n- The memorize-then-abstract phenomenon seems potentially related to 'grokking' [1], in that both involve improvement in generalization after a model has already largely converged on the training data.\n\n[1] Power, A., Burda, Y., Edwards, H., Babuschkin, I., & Misra, V. (2022). Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "aOA-kWdfL8", "reviewer": "Reviewer_tkML", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a framework to evaluate the abstraction capability of deep models in terms of transferability. A number of empirical simulations are run on language models in order to evaluate the dynamics of abstraction.", "summary_of_the_review": "The paper offers a useful probing framework for assessing the learning capabilities of deep neural models. Results are interesting in terms of transfer learning, but their meaning in terms of abstraction may be debatable. Explaining this gap and relating to other literature on abstraction would strongly improve the contribution.", "strength_and_weaknesses": "The paper considers a very interesting and relevant problem, and it explores it using a sensible framework and experiments. However, the reviewer is uncertain about its conclusions conceptually and pragmatically. From a conceptual point of view, abstraction is never well defined; all the empirical simulations seem to me deal with transfer learning, thus leading to a sort of identification of abstraction = transferability. I assume (together with some of the authors referenced in the introduction) that abstraction is something more than transfer ability. The author should probably discuss this point. From a pragmatic point of view, even if the authors explored some different settings in Section 6.4, it seems that the cases considered are limited, and hence the generalizability of the results.\nFurthermore, I think it would be interesting if the authors were to engage with other relevant papers. For instance, the framework considered by the authors is built similarly to other common datasets (e.g.: ARC by Chollet); beyond modality, how are the two related? The \"memorize-and-abstract\" is also reminiscent of the learning dynamics in information bottleneck (Tishby); is there an analogy there?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster