Incorporating Syntactic Knowledge into Pre-trained Language Model using Optimization for Overcoming Catastrophic Forgetting
Authors: Ran Iwamoto
Issei Yoshida
Hiroshi Kanayama
Takuya Ohko
Masayasu Muraoka
Venue: EMNLP 2023 Findings (2023)
Keywords: syntax
BERT
language model
optimization
catastrophic forgetting
TL;DR: We explore additional training to incorporate syntactic knowledge to a language model while clearly avoiding catastrophic forgetting.

Abstract: Syntactic knowledge is invaluable information for many tasks which handle complex or long sentences, but typical pre-trained language models do not contain sufficient syntactic knowledge. Thus it results in failures in downstream tasks that require syntactic knowledge.
In this paper, we explore additional training to incorporate syntactic knowledge to a language model. We designed four pre-training tasks that learn different syntactic perspectives. 
For adding new syntactic knowledge and keeping a good balance between the original and additional knowledge, we addressed the problem of catastrophic forgetting that prevents the model from keeping semantic information when the model learns additional syntactic knowledge. We demonstrated that additional syntactic training produced consistent performance gains while clearly avoiding catastrophic forgetting.

Incorporating Syntactic Knowledge into Pre-trained Language Model using Optimization for Overcoming Catastrophic Forgetting
Ran Iwamoto 1.2 Issei Yoshida' Hiroshi Kanayama' Takuya Ohko' Masayasu Muraoka'
IBM Research Tokyo, Keio University
ran. (tt.otooooeemmm.oomoooo.eeeee hkana, ohkot, mmuraoka)@jj..... ibm. com
Abstract
Semantic pre-training
:Optimizations in syntactic pre-training:
Maxoked Language Modeling
SGD, Adamt
Syntactic knowledge invaluable informa- tion for many tasks which handle complex or long sentences, but typical pre-trained lan- guage models do not contain sufficient syntac- tic knowledge. Thus it results in failures in downstream tasks that require syntactic knowl- edge. In this paper, we explore additional training to incorporate syntactic knowledge to a language model. We designed four pre- training tasks that learn different syntactic per- spectives. For adding new syntactic knowl- edge and keeping good balance between the original and additional knowledge, ad- dressed the problem of catastrophic forgetting that prevents the model from keeping semantic information when the model learns additional syntactic knowledge. We demonstrated that additional syntactic training produced consis- tent performance gains while clearly avoiding catastrophic forgetting.
+
For preventing catastrophic forgetting
+ Gradient Surgery
Additional
Elestic Weight Consolidation
syntactic training
Deprel prediction
e /
Phrwse delection
Mainisubordisste desefitctionn
Psemantis
syyntarite
Coentinatin delection
catastrophic forgetting
0 Pestiia
Language model with both semantic and syntactic knowledge
perameters
Low error region semantic pre training
Low crror regton syntactic pra-traning
An overview of additional syntactic training. Syntactic knowledge is added to the language model while preserving original semantic knowledge in the model using proper optimization functions.
Figure 1:
the models empowered with syntax knowledge and retaining semantic knowledge can be easily upplied to downstream tasks and, the methodology is use- ful for injecting other knowledge into the language model to create knowledge-enhanced models such as domain-specini ones.
1 Introduction
Pre-trained language models are commonly used and improve the performance of a variety of appli- cation tasks. It has been shown that those models roughly capture syntactic knowledge (Chi et al., 2020), such as dependency labels, but they lack some syntactic knowledge such dependency distance and head token required for application tasks (Xu et al., 2021).
Figure depicts our approach and techniques obtain desirable language models. We start with a language model such as BERT (Devlin et al., 2019 9) which was pre-trained with a large amount of unlabeled text data and self-supervised tasks such as Masked Language Modeling. Such model contains semantic knowledge but does not have explicit syntactic information, thus we enhance the model with labeled data with syntactic information.
Recent studies have shown that incorporat- ing syntactic knowledge into the models fur- ther improves the performance of language un- derstanding (Zhang et al., 2020) and translation tasks (Bugliarello and Okazaki, 2020) by adding other modules to the core model in the training and application phases.
One major issue of additional syntactic training is that we need to find good balance between the original and additional knowledge for improving performance of downstream tasks. Most of the previous works have selected dependency structure predictions as the task to learn syntactic knowledge. However, there are many other specific syntactic notions, such as phrase structures and coordina-
In contrast, Tian al. (2022) and Yang and Wan (2022) take approaches to embed syntactic knowl- edge into a model with keeping model structures. this paper we follow this approach (henseforth we call it additional syntactic training), because

---

tions from which the additional training may bene- fit, and it is non-trivial to identify the best ones. Another big issue in balancing the model and additional knowledge is catastrophic forget- ting (French, 1999; Kemker et al., 2018), where additional training causes a significant loss of im- portant knowledge embedded in the original model. To tackle this problem, we exploit multiple opti- mizers and quantitatively observe the balance of capabilities to predict semantic and syntactic fea- tures. To the best of our knowledge, our study is the first to analyze how both the types of addi- tional syntactic training and its optimizations affect downstream tasks.
Text
Syntax
Syntax Label
deprel
phrase mainisub
coord
We
PRON
nsub]
noubj
math
other
still
advied
advmod
nain
other
have
VERB
rost
root
nain
other
det
6b3
traders
NOUN
on]
obj main
conj
and
CCONJ
ob]
nain
books
NOUN
00]
obj
matin
conj
that
PRON
10
ob]
obj
sub
other
you
nsubj
nsubj
10
provided
VERB
acl
sel
sub
other
last
ab1
week
NOUN
obl
obl
sub
other
PUNCT
punct
punct
Main
other
An example of additional syntactic training tasks. With Tian et al.'s (2022) system, the model learns Syntax Head and Label simultaneously. Gray areas denote phrases, main clauses, and coordination structures.
Table 1:
Our contributions in this paper as follows:
Designed additional syntactic training with four syntactic tasks that can help a language model solve downstream tasks from different syntactic perspectives.
We therefore develop four pre-traue pre-training tasks predict various syntactic items. Each task predicts specific syntactic information: 1) deprel prediction, 2) phrase detection, 3) main/subordinate classifica- tion, and 4) coordination detection. We investigate which task would be effective on applications.
Exploited optimization functions that prevent catastrophic forgetting during the additional syntactic training, and discovered the trade- off between retention of original knowledge and enhancement with additional knowledge.
2.1 Additional Syntactic Training System
Here, we describe our method for additional syn- tactic training. We customize Tian et al.'s (2022) system to learn syntactic information.
Demonstrated that the syntactically enhanced models improved performances on CoLA,, RTE, MRPC and key phrase extraction tasks, with qualitative and quantitative discussions to find the syntactic perspectives that contribute to each task.
The system is trained on two tasks in parallel: de- pendency masking (DM), which predicts whether there is dependency between any pair of words in a sentence, and masked dependency prediction (MDP), which predicts the type of dependency re- lationships. We replace the input for MDP with one of four pre-training tasks.
2 Additional Syntactic Training
2.2 Syntactic Tasks
additional syntactic training, models mainly learn syntactic relationships between two tokens. There have been proposed tasks such as predict- ing dependencies (Wang et al., 2021a), predicting parent-child, sibling, or cousin relationships (Tian al., 2022), and dependency distance. The de- pendency structure encompasses a variety word relationships, a number of which are important in application tasks.
Table 1 illustrates the four variations of our pre- training tasks. Syntax Head conveys the form of dependency tree, which is used for DM in the sys- tem described in Section 2.1 and fixed in this paper. As MDP, we vary the viewpoint of syntax classifi- cation with the following four labeling tasks.
Deprel prediction (deprel) The model predicts dependency labels the same way as UD (Nivre et al., 2020) parsing in the task. For additional syntactic training tasks, we position this task as a baseline.
For example, in noun phrase extraction (Gu et al., 2021) and sentiment analysis (Kanayama and Iwamoto, 2020), the relationship between main and subordinate clauses and parallel (coordinate) struc- tures are handled well. To reflect the syntactic knowledge missing and required for downstream tasks in pre-trained models, we need to perform more diverse additional syntactic training tasks.
Phrase detection (phrase) In this task, model predicts the relationship between phrases (the de- prel label of the head token in the phrase). It is in- spired by Basirat and Nivre's 's (2021) work, and we

---

follow their definition of phrase-like units. They de- fined a phrase as the block connected by seven UD functional relations, such as determiner and case marker. Their model achieved high performance in dependency parsing by phrase-level pre-training.
each other. After that, we sum up the resultant gradients oblain a single gradient vector. For example, given two tasks and B, if their gradient vectors 9 and 9B are in opposite directions (i.e., if the cosine similarity of 9 and 9g is negative), one gradient 9 is projected onto the orthogonal plane of the other gradient gB follows:
Main/subordinate classification (main/sub) The task predicts two labels to classify clauses into main and sub. We refer to Nikolaev and Pado's (2022) method to investigate whether pre-trained model can detect subordinate clauses.
9A 9B
9A = 9A
|gs 2 9B.
where | denotes L2-norm of æ.
Coordination detection (coord) aims to pre- dict parallel structures like "A and B" where A and B can be tokens, phrases, or sentences. Specifi- cally, it predicts the labels of A and B's head tokens (conj), their child tokens (child), parallel conjunc- tions such as "and" and "but" (cc), and other tokens (other).
3.2 Elastic Weight Consolidation (EWC)
Elastic Weight Consolidation is continuous learn- ing optimization function in which tasks are learned in sequence. Given two tasks, A and B, the model first searches for the optimal solution for task A, and then for the parameters that perform well in both tasks and B. In EWC, when the model is fine-tuned for B after A, the important parameters of the task A are updated as little as possible, while the less important parameters of task A are updated with larger weights. We use the Fisher information matrix F (Pascanu and Bengio, 2013), which is diagonal matrix, reduce the computational cost and to utilize characteristies of F (Pascanu and Bengio, 2013), though gives an approximation of the optimal parameters function of EWC ((0) is as follows:
3 Optimization Functions to Prevent Catastrophic Forgetting
This section describes two optimization functions that prevent catastrophic forgetting for keeping original semantic knowledge during additional syn- tactic training. When the model has already been trained for specific task and is then trained for another task, the performance of the previous task significantly decreases. This is called catastrophic forgetting (French, 1999; Kemker et al., 2018).
L(0) for task A. The loss
Although large-scale pre-trained models are known to be relatively resilient to catastrophic for-
LB()) Σ
A 3 Fi0:-
getting (Ramasesh et al., 2022), the problem re- mains when using small models such as BERT (De- vlin et al., 2019). Since BERT is still often used in the real world task due to computational resources, a variety of methods are developed to prevent catas- trophic forgetting even in pre-trained models such as decreasing the learning rate (Kar et al., 2022). Among these methods, we used Gradient Surgery (Yu et al., 2020), optimization function known in the domain of multi-task learning, and Elastic Weight Consolidation (Kirkpatrick al., 2017), an optimization function commonly used in continual learning.
where (B) is the task B's loss, À sets how impor- tant the task A is compared with the task B, and labels each parameter.
In the rest of this paper, we show our experimen- tal results of three different tasks; additional syn- tactic training (Section 4), the GLUE benchmark (Section 5), and key phrase extraction (Section 6) and explore how syntactic information and differ- ent optimization functions affect the pre-trained model itself and downstream tasks.
Preliminary Experiment: Additional Syntactic Training
3.1 Gradient Surgery (GS)
Here, we first show how well pre-trained models with semantic information can learn specific syn- tactic information by additional syntactic training tasks (See Section 2.2 for detail) using various op- timization functions while preserving semantic in- formation.
Gradient Surgery is an optimization function for solving gradient conflicts in multi-task learning. first computes the gradients for each task involved in the multi-task learning. We then discard adver- sarial elements of the gradients that conflict with

---

train
dev
test
syntactic training none deprel
optimization function
F1
PPL (1)
original
12543
2001
2077
after filtering
10271
1471
1485
29.10
AdamW
96.22
702.87
Table 2: Number of sentences in the UD-EWT corpus and after filtering for additional syntactic training.
SGD
96.13
32392.10
GS
95.43
10.44
EWC
95.13
25.63
4.1 Settings
phrase
AdamW
97.25
1360.61
SGD
95.62
2587.11
The data for syntactic training was gener- ated from UD-EWT Ver 2.10 1 The sentences sat- isfying any of the following three conditions were removed as noise: 1) sentences with less than five tokens, 2) sentences containing foreign languages tagged as X, and 3) sentences containing undefined dependency relations labeled as dep. The num- ber of sentences used in the experiment was the same across all tasks since these treatments were applied to all pre-training tasks. Table shows the statistics.
Data
GS
95.78
9.68
EWC
95.58
24.17
main/sub
Adam
98.35
720.49
SGD
97.74
1471.63
GS
94.70
11.16
EWC
95.29
23.59
coord
AdamW
94.68
2971.79
SGD
89.55
1412.65
GS
91.33
9.56
EWC
91.00
23.78
Table 3: Comparison with different syntactic tasks and optimization functions in additional syntactic training. FI score is the average between syntax head Fl and syntax label FI. The row "none" means the baseline model without additional syntactic training.
System We used the syntactic pre-training sys- tem described in Section 2. We adopted the pre-trained bert-base-cased model downloaded from the Hugging Face transformers library (Wolf et al., 2020) in the following experiments. The mod- els trained by each of the four additional syntactic training methods described in Section 2.2 were compared with the original bert-base-cased model. Multiple learning rates (le-4.ee-5.le-6) and multiple numbers of max epochs (50,70,100) were tested.
Text2 (Merity et al., 2017) for each syntactic train- ing step, in which MLM gradients computed. It may be worthwhile to include MLM pre- training using AdamW/SGD on the corpus used for additional syntactic training as comparisons. Al- though it is already known that AdamW (or Adam) and SGD can cause catastrophic forgetting in both continual learning and multi-task learning. Also in multi-task optimization, existing studies such GS, Gradient Vaccine (Wang et al., 2021b), and GradNorm (Chen al., 2018) showed that models using Adam or SGD do not perform well.
We experimented with four optimization func- tions: GS, EWC, AdamW, and SGD. GS and EWC prevent catastrophic forgetting as described in Section 3. We used GS implemented by its au- thors (Tseng, 2020) and EWC by ours, respectively, For comparison, AdamW (Loshchilov and Hutter, 2019) and SGD (Robbins and Monro, 1951) from the official PyTorc library (Paszke et al., 2019) were used.
Evaluation We measured precision, recall, and Fl score in the additional syntactic training tasks. To evaluate whether the model retains the origi- nal semantic knowledge, we used the perplexity (PPL) (Jelinek et al., 1977) score of the MLM.
Models using GS are trained with gradients com- puted from small amount of text data used for Masked Language Modeling (we call them MLM gradients afterward), while models using EWC only use MLM gradients for determining the im- portance parameters, for parameter updating. A and SGD do not consider MLM gradi- ent information. For training with GS and EWC, 100 sentences were randomly selected from Wiki-
4.2 Results
Table 3 reports the highest average Fl scores of Syntax Head and Syntax Label predictions. There was significant difference between different op- timization functions the PPL measure MLM. AdamW and SGD focus only on improving the per- formance the target task (in this case, syntactic
http://gggttuocccoa////raallpaar English-EWT

---

types of tasks: single sentence tasks (CoLA, SST- 2), similarity and paraphrase tasks (MRPC, QQP and STS-B) and natural language inference tasks (MNLI, QNLI, RTE). We conducted detailed ex- periments on three tasks for which syntactic infor- mation seems to be important in solving the tasks: CoLA (Warstadt et al., 2019), RTE (Dagan et al.. 2006), and MRPC (Dolan and Brockett, 2005). Ex- periments other tasks that did not required signif- icant syntactic knowledge are shown in Appendix. From experiments of the GLUE subset, it can be concluded that additional syntactic training ben- eficial for tasks involving syntactic knowledge, as opposed to the tasks mentioned in the Appendix A. which do not appear to require such knowledge.
Adam, Iie5,, macepnehta
1es4
65, 1d144 manspeci00
DWC, Kladk maoapochdO
3015
1e+2
1011
30x3
0.6
0.7
0.8
69
Figure 2: Relationships between syntactic pre-training Fl score and PPL. The checkpoints at every 10 epochs are plotted in syntactic pre-training phase. The dotted line represents PPL. bert-base without additional syn- tactic training.
5.1 Settings
training), thus they achieve high Fl scores in the target task with significantly degrading the previ- ous task, which was indicated by high PPL scores (catastrophic forgetting). Much lower PPL scores in GS and EWC indicate that they retain seman- tic knowledge gained before additional syntactic training.
We evaluated on the official data splits provided by GLUE. The details and evaluation metrics of each task are described in Appendix A. As the overall score for our methods, we took a macro-average of scores for the tasks. The number of epoch was set to 5, and other hyperparameters were set to the default values specified in the code provided by Hugging Face?.
All syntactic pre-trained models achieved over 91% average Fl scores except for the model trained with SGD on the coord task, indicating that most models have sufficiently learned syntactic informa- tion. Especially, there is a small difference in Fl scores between models trained with AdamW and other optimization functions in the deprel task.
We trained models for each setting (of four op- timizers and four syntactic tasks). The training is conducted over three learning rates and three max epochs described in Section 4.1, saving the checkpoints at every 10 epochs in syntactic pre- training phase. We then trained all of these models for the application tasks. Therefore, approximately 1000 syntactic pre-trained models: optimizers x syntactic tasks learning rates x 22 checkpoints..
We explain why we use GS and EWC in addi- tional training and downstream tasks. The relation- ships between syntactic pre-training F1 score and PPL shown in Figure 2. The figure displays the models that achieved the highest Fl score for each optimization function, as well as the model with the smallest learning rate using Adam for reference. We plot checkpoints at every 10 epochs in syntac- tic additional training phase. GS and EWC can keep their PPLs below the bert-base PPL through- out the learning process, whereas Adam and SGD have very high PPL during learning (catastrophic forgetting). From this fact, we conclude GS and EWC are suitable for retaining both semantic and syntactic knowledge.
trained
For each pair of pre-training task and optimizer, we selected the best model in terms of the aver- age score of each GLUE task. To minimize the computational cost, we adopted the optimal hyper- parameters used in previous study (Wang et al., 2018). The reason we chose this approach was determine how much knowledge from MLM and syntactic structures is needed to achieve good per- formance in application tasks.
Regarding evaluation, we considered that the evaluation on dev data is appropriate for this pur- pose to keep the test data unbiased to our tuning, following the policy by Suzuki al. (2023).
5 Experiment 1: GLUE
We investigate the effects of syntactic pre-trained models to multiple downstream tasks using GLUE benchmark (Wang et al., 2018). GLUE has three
i http:///ww....ooooooii/a//////nr blob/ htt.m///wwweeeeeooooo/////ttttta run jun_glue.p 'Every 10 epochs for max epochs of 50, 70, 100.

---

Syntactic Training
GLUE tasks
Pre-train info
CoLA
RTE
MRPC
Avg.
Ir
epoch
Task
Optimizer
(mc)
(acc)
(ace/ / [FI)
none
60.10
64.62
84.56 / 89.23
70.54
deprel
AdamW
57.60
66.43
78.19 85.85
68.68
lo-5
10/70
SGD
57.80
67.15
85.05 89.78
70.79
le-5
10/50
GS
59.07
67.51
86.02 89.98
71.58
le-4
20/100
EWC
60.57
66.07
71.45
le-4
30/70
phrase
A
55.98
67.15
86.21
68.53
le-5
20/70
SGD
58.30
67.87
84.31/89.35
71.00
le-5
10/50
GS
60.86
64.98
84.56/89.22
70.91
le-5
20/70
EWC
60.32
67.15
86.28 90.54
71.96
le-5
20/50
main/sub
Adam W
55.73
66.07
77.94
85.44
67.83
le-5
40/70
SGD
57.31
67.51
85.29 / 19.90
70.81
le-5
10/50
GS
59.10
67.51
85.29
19.83
71.39
lo-5
20/70
EWC
60.09
68.23
84.80 / 89.49
71.82
le-5
20/70
coord
A
58.29
63.90
77.45
85.16
67.83
le-5
60/70
SGD
59.31
66.07
83.82
88.81
70.56
le-5
20/70
GS
59.56
68.59
85.54
90.02
71.98
le-5
60/70
EWC
61.10
68.23
85.05
89.61
72.22
le-5
30/50
Table Performance comparison of three GLUE tasks based on additional syntactic training and optimizers. The highest for each additional syntactic training underlined, and those for each GLUU task are shown in bold. The row "none" means the baseline model without additional syntactic training.
Optimizer
Average
Syntactic Task deprel
Average
CoLA
RTE
MRPC
CoLA
RTE
MRPC
AdamW
56.90
65.88
78,06 85.67
58.76
66.43
83.82/ 89.02
SGD
58.18
67.15
84.62 89.46
phrase
58.87
66.79
83.46 88.83
GS
59.65
66.79
85.48
89.90
main/sub
58.06
67.33
83.33 88 38.66
EWC
60.52
67.42
85.42 / 89.89
coord 59.56
66.70
82.97 / 88.40
Comparison of model average for each opti- Table 5:
Comparison of model average for each addi- Table 6:
mizer in three GLUE tasks.
tional syntactic training in three GLUE tasks.
5.2 Results
Comparison of optimizers Table 5 shows the scores of each optimizer averaged over four syntac- tic training tasks in Table The model pre-trained with EWC achieves higher scores on two tasks, and the model pre-trained with GS has similar scores. That is, regardless of the type of syntactic train- ing task, learning syntax using GS and EWC is effective to enhance the model performance.
The results are shown in Table 4. The right block is the performance in the pre-training with Ir and epoch syntactic tasks and PPL on MLM task. The models using GS and EWC, which were introduced to prevent catastrophic forgetting, achieved higher scores, as expected, than those using AdamW and SGD CoLA and MRPC tasks. Among the 16 combinations of syntactic tasks and optimizers, coord with EWC achieved the high- est average score (72.22), and most of other cases showed scores above the baseline (70.54) except for those optimized with AdamW. In the following, we compare the performance of each optimizer and each pre- pre-training task by taking the average score on each feature.
As shown in the right block of Table 3 GS and EWC preserve semantic information with small Sacrifice of additional syntactic training scores. In the trade-off relationship between these two, Ta- ble 5 shows the strength of GS and EWC in down- stream tasks, models with a good balance between semantic information and syntactic information achieved high performance.

---

Syntactic Training Task
Key Phrase Extraction
Pre-train Info
Optimizer
precision
recall
F1
Ir
epoch
none
59.71
74.11
66.14
deprel
AdamW
62.41
75.22
68.22
le-5
10/50
SGD
62.50
74.78
68.09
le-5
60/100
GS
62.24
75.30
68.15
le-4
60/70
EWC
61.43
75.97
67.93
le-5
30/60
phrase
AdamW
62.52
75.07
68.22
le-5
60/100
SGD
62.09
76.64
68.60
le-4
10/70
GS
62.52
76.34
68.74
le-6
60/100
EWC
61.77
75.97
68.13
le-5
40/100
main/sub
AdamW
62.40
75.07
68.15
lc-6
30/100
SGD
61.58
75.60
67.8
le-4
60/70
GS
62.04
75.89
68.27
le-4
10/50
EWC
61.12
75,45
67.53
le-5
20/70
coord
A
61.68
76.41
68.26
le-5
10/70
SGD
62.89
75.52
68.63
le-4
10/70
GS
61.42
75.82
67.87
le-6
60/100
EWC
62.03
75.60
68.14
lo-5
40/100
Table 7: Performance comparison key phrase extraction task.
Comparison of additional syntactic training Table 6 shows the scores of each pre-training tasks averaged over the four optimizers in Table 4. A variety syntactic information contributed to the performance improvement. In the CoLA task, co- ord achieved the highest average score. We explain this using the data grammatical features. Warstadt and Bowman (2019) compared CoLA performance for sentences containing major features using dev data. They defined 13 grammatical major features and further divided into 59 minor features, and ordination is the most frequent minor feature in major feature. They found that in the BERT model, Matthew's ' correlation for sentences containing the major feature was the second lowest among all major features. This indicates that the CoLA task performance was improved because models cap- tured the coordination structures through additional additional syntactic training.
analysis task, adjectives are much more effective than other PoS. The deprel task helped the model learn more comprehensive syntactic information than the other tasks, which explains our result.
Experiment 2: Key Phrase Extraction
In addtion to binary classification tasks in GLUE, we conducted another experiment on the key phrase extraction task (Gu et al., 2021). It is a task to de- tect domain-specific phrases in sentences, and is considered good workbench test the ability to extract syntactically correct sequences appli- cation close real-world use cases.
6.1 Settings
We used Gu al.'s 's (2021) method in our exper- iments. They trained a phrase tagger using lan- guage models by generating silver labels from un- labeled text and using neither external knowledge base nor dictionaries. We modified to work with BERT since their code worked only with models using byte pair encoding tokenizers. The number of epochs was set to 100. Other hyperparameters followed the default settings in the code.
The RTE task contains many long sentences, and we consider that the main/sub pre-trained model handled long sentences better than the other models since the model can capture subordinate structures.
For the MRPC task, deprel achieved the highest average score. Perez et al. (2021) mentioned that MRPC requires information with various PoS tags via experiments in which models are trained by masking words of a specific PoS tag in each GLUE tasks. As a counter example, for SST-2, sentiment
In experiments by Gu al. (2021), datasets KP20k (Meng al., 2017) and KPTimes (Gallina et al., 2019) are used. Since we evaluated many models, we used a small version of the KP20k pro- vided in their code to reduce computational time.

---

Statistics
train
test
[Gold Answer]
document
10000
1000
In addition to recycling, use of a RFID system can also reduce business costs, by identifying the position of goods and picking carts.
sentence
81340
8144
words per sentence
21.50
21.48
[Baseline (any syntactic pretraining]]
In addition recycling. of system also reduce business
Table Statistics KP20k small datasets.
identifying position of goods
and picking carts.
Syntactie
Average
[Syntactic pretraining (phrase task, using GS)]
Task
prec
rec
F1
In addition to recycling, , use of a RFID system can also reduce business costs, by identifying the position of goods and picking carts.
deprel 62.15
75.32
68. 10
phrase
62.23
76.01
68.42
main/sub
61.79
75.50
67.96
Figure 3: Phrase extraction 1. The same phrase extrac- tions among gold answer and two methods are blue- underlined, different phrase extractions are written in bold yellow. Wrong output: "reduce business costs."
coord
62.01
75.84
68.22
Table 9: Comparison of average scores of syntactic pre- trained models in the key phrase extraction.
[Gold Answer]
A geometric- method for recognizing overlapping polygonal-shapes and semi-transparent particles in gray tone images..
KP20k is dataset created from titles, abstracts, and key phrases of scientific articles in computer science. The data statistics are shown in Table We selected the models in the same manner as we did Scction 5.1 for the three GLUE tasks.
[Baseline (any syntactic pretraining)
grometrir based method recognizing overlapping polygonal-shaper and semi-transparent particles in gray tone images
[Syntactic pretraining (phrase task, using GS)]
6.2 Results
geometric- method recognizing overlapping polygonal-shaped semi-transparent particles in gray tone images.
The results shown in Table 7. As well in the experiments in Section 5, fine-tuned the syntactic pre-treined models for this task. All the syntactic models achieved higher Fl scores than the baseline by more than 1.4 points, regardless of the optimizers. This indicates that a variety of syntactic information contributes to improving key phrase extraction, and the effect is bigger than in the binary classification tasks in GLUE.
Figure 4: Phrase extraction 2. Wrong output: "recog- nizing overlapping."
as verb in this sentence. On the other hand, phrase pre-trained model successfully detected "business costs". In Figure 4, the baseline model incorrectly detected "recognizing overlapping" as a phrase. The phrase pre-trained model successfully avoided a phrase extraction error caused by a word whose grammatical behavior may be confused, in this case, between gerund and present progressive.
Table 9 shows the average scores over for op- timizers. The results supports that phrase was the best in all four metries. As shown Table 1, phrase has an intermediate granuality between deprel and main/sub, and it is intuitive that this matches well the syntactic knowledge required in the key phrase extraction task.
Conclusion
This paper investigated the incorporation of syntac- tic knowledge into pre-trained BERT model by additional training. We designed four additional syntactic training tasks: deprel prediction, phrase detection, main/subordinate classification, and co- ordination detection. We utilized GS and EWC optimizations for preserving semantic knowledge. The experimental results showed improvements of the models empowered with syntactic knowl- edge in the downstream tasks, and supported our intuition that each task requires different syntactic perspectives for more accurate predictions.
6.3 Case Study
We present qualitative analysis for showing how added syntactic knowledge is reflected in the model extractions. We observe the difference between the baseline model without additional syntactic train- ing and pre-trained model with the phrase task. The phrase pre-trained model using GS achieved the best Fl score observed in Section 6.2.
In Figure 3, we can find that the baseline model incorrectly detected "reduce business costs" as a key phrase, because it failed to recognize "reduce"

---

Limitations
Normalization for Adaptive Loss Balancing in Deep Multitask Networks.
Our main limitation is that our method is evaluated on English-only datasets including GLUE bench- marks and key phrase extraction datasets. The ap- plicability of our method to other languages can be investigated by evaluating, for example, a multilin- gual version of the GLUE task (Liang al., 2020; Kurihara et al., 2022).
Ethan A. Chi, John Hewitt, and Christopher D. Man- ning. 2020. Finding universal grammatical rela- tions in multilingual BERT. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 5564-5577, Online. As- sociation for Computational Linguisties.
An additional limitation comes from the pre- trained model. We used a pre-trained BERT model with 10M parameters generated for general pur- pose tasks. The investigation methods in this paper can be applied to other language models such as RoBERTa (Liu et al., 2019), though they may show different trends through further experiments.
Ido Dagan, Oren Glickman, and Bernrdo Magnini. 2006. The PASCAL Recognising Textual Entail- ment Challenge. In Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment, pages 177-190.
Jacob Devlin. Ming-Wei Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume 1 (Long and Short Papers). pages 4171-4186, Minneapolis, Minnesota. Associ- ation for Computational Linguistics.
Ethics Statement
Our method belongs to additional language model pre-training. We used publicly available codes/datasets such as licensed under CC BY-SA 4.0 and MIT to ensure reproducibility of our ex- perimental results. Using an efficient optimization function rather than many parameter tuning imple- mentations to prevent catastrophic forgetting can help reduce the carbon footprint of model training. Our method uses pre-trained language models that may contain gender or abusive data bias. We need to be careful when using the model to prevent potential biases in application task outcomes.
William B. Dolan and Chris Brockett. 2005. Automati- cally constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
Robert M. French. 1999. Catastrophic forgetting connectionist networks. Trends in Cognitive Sci- ences, 3(4):
Ygor Gallina, Florian Boudin, and Beatrice Daille. 2019. KPTimes: A large-scale dataset for keyphrase generation on news documents. In Proceedings of the 12th International Conference Natural Lan- guage Generation, pages 130-135, Tokyo, Japan. Association for Computational Linguistics.
References
Ali Basirat and Joakim Nivre. 2021. Syntactic nuclei in dependency parsing- - a multilingual exploration. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guistics: Main Volume, pages 1376-1387, Online. Association for Computational Linguistics.
Xiaotao Gu, Zihan Wang, Zhenyu Bi, Yu Meng, Liyuan Liu, Jiawei Han, and Jingbo Shang. 2021. UCPhrase: Unsupervised Context-Aware Quality Phrase Tagging. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, page 478-486,
Emanuele Bugliarello and Naoaki Okazaki. 2020. En hancing machine translation with dependency-awand self-attention. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 1618-1627, Online. Association for Computational Linguistics.
Fred Jelinek, Robert Mercer, Lalit R Bahl, and James K Baker. 1977. Perplexity- measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America. 62(S1):S63- S63.
Daniel Cer, Mona Diab, Eneko Agirre, lnigo Lopez- Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and Proceedings
crosslingual focused evaluation. of the 11th International Workshop on Semantic Evaluation (SemEval-201 pages 1-14, Vancouver, Canada. Association for Computational Linguisties.
Hiroshi Kanayama and Ran Iwamoto. 2020. How uni- versal are Universal Dependencies? exploiting syn- tax for multilingual clause-level sentiment detection. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4063-4073, Mar- seille, France. European Language Resources Asso- ciation.
Zhao Chen, Vijay Badrinarayanan, Chen-Yu Yu Lee, and Andrew Rabinovich. 2018. GradNorm: Gradient

---

Sudipta Kar, Giuseppe Castellucci, Simone Filice. Shervin Malmasi, and Oleg Rokhlenko. 2022. Pre- venting Catastrophic Forgetting in Continual Learn- ing of New Natural Language Tasks. In Proceedings of the 28th ACM SIGKDD Conference on Knowl- edge Discovery and Data Mining, page 3137-3145.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer Sentinel Mixture Models. In International Conference on Learning Representations.
Dmitry Nikolaev and Sebastian Pado. 2022. Word- order typology in multilingual BERT: A case study in subordinate-claus detection. In Proceedings of the 4th Workshop on Research in Computational Lin- guistic Typology and Multilingual NLP, pages 11- 21, Seattle, Washington. Association for Computa- tional Linguistics.
Ronald Kemker, Mare McClure, Angelina Abitino, Tyler L. Hayes, and Christopher Kanan. 2018. Measuring Catastrophic Forgetting in Neural Net- works. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirfi- eth Innovative Applications of Artificial Intelli- gence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAT IS/IAAT 18/EAAT 18. AAAT Press.
Joakim Nivre, Marie-Catherine de Marneffe, Filip Gin- ter, Jan Hajié. Christopher D. Manning. Sampo Pyysalo, Sehastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal Dependencies v2: An evergrowing multilingual treebank collection. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4034-4043, Mar- scille, France. European Language Resources Asso- ciation.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Ag- nieszka Bariskaaa Demis Hassabis. Clau- dia Clopath, Dharshan Kumaran, and Raia Hadsell. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521-352
Razvan Pascanu and Yoshua Bengio. 2013. Revisiting Natural Gradient for Deep Networks.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary De Vito, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, Hrigh-Performacce Deep Learn ing Library. In Advances in Neural Information Pro- cessing Systems 32, pages 8024-8035. Curran Asso- ciates, Inc.
Kentaro Kurihara, Daisuke Kawahara, and Tomohide Shibata. 2022. JGLUE: Japanese general lan- guage understanding evaluation. In Proceedings of the Thirteenth Language Rexources and Evaluation Conference, pages 2957-2966, Marseille, France. European Language Resources Association.
Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fen- fei Guo, Weizhen Qi. Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao. Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Win- nie Wu, Shuguang Liu, Fan Yang. Daniel Campos, Rangan Majumder, and Ming Zhou. 2020. XGLUE: new benchmark dataset for rross-lingual pre- training, understanding and generation. In Proceed- ings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages (008-6018 Online. Association for Computational Linguistics.
Ethan Perez, Douwe Kicla, and Kyunghyun Cho. 2021. Rissanen data analysis: Examining dataset character- istics via description length. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Re- search, pages 8500-8513, PMLR
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: Robustly Optimized BERT Pretrain- ing Approach. CoRR, abss/190.11666
Vinay Venkatesh Ramasesh, Aitor Lewkowyez, and Ethan Dyer. 2022. Effect of scale on catastrophic forgetting in neural networks. In International Con- ference on Learning Representations.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Con- ference Learning Representations.
Herbert Robbins and Sutton Monro. 1951, A Stochas- tic Approximation Method. The Annals of Mathe- matical Statistics, 22(3):400- 407.
Rui Meng, Sanqiang Zhao, Shuguang Han, Daqing He. Peter Brusilovsky, and Yu Chi. 2017. Deep keyphrase generation. In Proceedings of the 55th Annual Meeting of the Association for Computa- tional Linguistics (Volume I: Long Papers), pages 582-592, Vancouver, Canada. Association for Com- putational Linguistics.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment tree- bank. In Proceedings of the 2013 Conference on

---

Empirical Methods in Natural Language Processing, pages 1631-1642, Seattle, Washington, USA. Asso- ciation for Computational Linguistics.
Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State- the-art art natural language process- ing. In Proceedings of the 2020 Conference on Em- pirical Methods Natural Language Processing: System Demonstrations, pages 38-45, Online. Asso- ciation for Computational Linguisties.
Jun Suzuki, Heiga Zen, and Hideto Kazawa, 2023.
Extracting representative subset from extensive text data for training pre-trainee trained language models. Infor- mation Processing and Management, 60(3):103249.
7enan Xu, Daya Guo, Duyu Tang, Qinliang Su, Linjun Shou, Ming Gong, Wanjun Zhong, Xiaojun Quan. Daxin Jiang. and Nan Duan. 2021. Syntax-enhanced pre-trained model. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing, pages 5412- 5422.
Yuanhe Tian, Yan Song, and Fei Xia. 2022. Improv- ing relation extraction through syntax-induced pre- training with dependency masking. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1875-1886.
Wei-Cheng Tseng, 2020. WeiChengTTeeng/PPyorchh PCGrad.
Zhixian Yang and Xiaojun Wan. 2022. Dependency- based mixture language models. In Proceedings of the 60th Anmual Meeting of the Association for Com- putational Linguistics, pages 1758-7775
Alex Wang. Amanpreet Singh, Julian Michael, Fe- lix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. In Pro-
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. 2020. Gradient Surgery for Multi-Task Learning. In Ad- vances Neural Information Processing Systems, volume 33, pages 5824-5836. Curran Associates, Inc.
ceedings of the 2018 EMNLP Workshop Black- boxNLP: Analyzing and Interpreting Neural Net- works for NLP, pages 353-355, Brussels, Belgium. Association for Computational Linguistics.
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, and Ming Zhou. 2021a. K-Adapter: Infusing Knowledge into Pre- Trained Models with Adapters. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1405-1418.
Zhuosheng Zhang, Yuwei Wu, Hai Zhao, Zuchao Li. Shuailiang Zhang. Xi Zhou, and Xiang Zhou. 2020. Semantics-Aware BERT for Language Understand- ing. Proceedings of the AAAI Conference Artifi- cial Intelligence, (45)
Zirui Wang. Yulia Tsvetkov, Orhan Firat, and Yuan Cao. 2021b. Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models. In International Conference on Learning Representations.
Alex Warstadt and Samuel Bowman, 2019, Gram- matical Analysis of Pretrained Sentence Encoders with Acceptability Judgments. arXiv preprint arXiv:1901.03438.
Alex Warstadt, Amanpreet Singh, and Samuel R. Bow- man. 2019. Neural Network Acceptability Judg- ments. Transactions of the Association for Compu- tational Linguistics,
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies. Volume 1 (Long Pupers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguis- tics.
Thomas Wolf, Lysandre Debut, Victor Sanh. Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison. Sam Shleifer. Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame,

---

A GLUE task explanation
classification scheme, distinguishing between pos- itive and negative sentiments, and utilize only the labels assigned the sentence level.
We describe the details of dataset in our exper- iments GLUE (Wang et al.. 2018), the General Language Understanding Evaluation benchmark. GLUE is the common benchmark for language comprehension. The licence under CC-BY-4.0.
QQP
The Quora Question Pairs2 dataset (QQP) 4 consists of a set of question pairs gathered from the Quora website, where users ask and answer questions. The objective is to determine whether given pair of questions are semantically equivalent. Similar MRPC, the distribution of classes in the QQP dataset is imbalanced, with majority of 63% negative instances. Hence, we report both accuracy and Fl score for evaluation.
A.1 Syntactic-related Tasks
Here we explain three GLUB tasks in which addi- tional syntactic training would be highly useful.
CoLA The Corpus of Linguistic Acceptability (CoLA) (Warstadt et al., 2019) composed of En- glish sentences extracted from books and journal articles focused on linguistic theory. Each sentence is annotated to indicate whether it is grammatically acceptable in English. To evaluate performance, we utilize the Matthews correlation (mc), metric for evaluating binary classification performance, which accounts for class imbalance and ranges from 11 to 1, with 0 representing random guess- ing. For evaluation purposes, we utilize the stan- dard test set and obtained private labels from the original authors same as official GLUE evaluation.
STS-B The Semantic Textual Similarity Bench- mark (Cer et al., 2017) consists of various sentence pairs obtained from news headlines, video and im- age captions, and natural language inference data. Each pair has been assessed by humans and as- signed a similarity score ranging from 1 to The objective of the task is to predict these scores. our analysis, we evaluate the performance using Pearson correlation coefficients (pc).
RTE The Recognizing Textual Entailment (RTE) (Dagan et al., 2006) datasets come from series of annual textual entailment challenges. The examples in the datasets are created using news articles and text from Wikipedia. To ensure uniformity, all datasets are splitted into a two-class. In three-class datasets, we combine the "neutral" and "contradiction" classes into the category of "not entailment."
The Multi-Genre Natural Language Infer- ence dataset (MNLI) (Williams et al., 2018) is collection of sentence pairs of textual entailment. The task involves predicting whether the premise sentence entails, contradicts, or is neutral to the hypothesis sentence. The sentences are extracted from transcribed speech, fiction, and government reports and so on. Evaluation is conducted on both the matched (in-domain, and mismatched (cross- domain, mm)) sections.
MNLI
MRPC The Microsoft Research Paraphrase Cor- pus (MRPC) (Dolan and Brockett, 2005) consists of pairs of sentences extracted from online news sources. Each pair has been annotated by humans to indicate whether the sentences are semantically equivalent. Due to the imbalance in class distribu- tion (68% positive), we report both accuracy and Fl score (standard practice).
QNLI The Stanford Question Answering Dataset (QNLI) (Rajpurkar al., 2016) is collection of parsgranp pairs designed for question- answering tasks. The dataset includes questions generated by annotators and paragraphs sourced from Wikipedia, with the answer to each question located within one of the paragraph's sentences. To transform the task into sentence pair classification, the pairs created between each question and every sentence within the corresponding context. The objective is to determine whether the context sentence contains the answer to the question. We evaluate accuracy on this task.
A.2 Syntactic-unrelater Tasks
The results of those tasks shown in appendix.
SST-2 The Stanford Sentiment Treebank-2 (SST- 2) (Socher et al., 2013), developed by Socher et al. in 2013, comprises sentences extracted from movie reviews along with human annotations indicating their sentiment. The objective to determine the sentiment of a given sentence. We employ a binary
First- First-Quora- ttt.g/urr..coopoooileei/iit---e.
hittps://www..r htttps://www.uuura.co quora.

---

Syntactic Training
GLUE Tasks
Task
Optimizer
SST-2
QQP
STS-B
MNLI m/mm
QNLI
(acc)
(ace / FI)
(pc)
m/mm (acc)
(acc)
none
92.66
90.98 / 87.78
89.08
83.71 83.34
90.55
deprel
A W
91.74
90.88 / 87.66
88.00
83.29 / 83.80
90.94
SGD
92.20
91.05 / 87.89
89.27
83.52 / 84.18
90.87
GS
92.32
90.85 / 87.62
89.36
83.28 / 83.70
90.68
EWC
91.51
90.88 / 87.65
89.21
83.12 / 83.69
90.55
phrase
A
91.17
90.48 / 87.18
85.49
82.36 / 83.01
90.17
SGD
91.86
90.72 / 87.44
87.17
83.11
83.45
90.81
GS
92.43
90.82 / 87.55
88.56
83.53
83.86
90.26
EWC
92.09
90.91 / 87.71
89.26
82.93 / 83.57
90.37
main/sub
AdamW W
89.68
90.10 86.65
85.42
81.71 / 82.45
89.11
SGD
91.97
90.84 / 87.59
87.57
83.31 / 83.83
90.19
GS
92.09
90.94 87.73
89.35
83.61 / 83.73
90.37
EWC
91.97
90.90 87.68
89.15
83.47
84.06
90.65
coord
Adam W
90.60
89.80 86.24
82.67
83.11/ 83.45
87.99
SGD
92.20
90.92 / 87.71
88.15
83.67
84.19
90.61
GS
91.86
90.84 / 87.60
89.27
83.45 / 83.55
90.32
EWC
91.86
90.98 / 87.79
89.13
83.43 / 83.66
90.26
Table 10: Performance comparison of five GLUE tasks based on additional syntactic training and optimizers. The highest scores each additional syntactic training underlined, and those each GLUE task are shown bold. The row "none" means the baseline model without additional syntactic training.
B GLUE results
STS-B Models pre-trained on the GS tend have higher accuracy. To explain this, we show an example of the data (news headline). For exam- ple, a pair of similar sentences "A plane is taking off." / "An air plane is taking off." has the similar syntactic structure and difference is only an adjec- tive. This is because believe that the model retaining semantic knowledge performed better.
In the downstream tasks listed below, the mod- els without syntactic information performed higher than the models with it. Our assumption was sup- ported by the results Table 10, where most of additional syntactic training did not contribute to the scores as we have seen for our focused tasks in Table 4.
MNLI For the matched class (two sentences are sampled from the same source), the models without additional syntactic training achieved the highest accuracy, while for the mismatched class (two sen- tences are from different sources), most models with additional syntactic training achieved better performance than those without additional syntac- tic training. We could not identify the reason for the difference in accuracy between different classes.
SST-2 We found that the model without addi- tional syntactic training showed the highest score. It can be argued from these results that syntactic knowledge rather handicap for sentiment anal- ysis of movie reviews. Among the models with additional syntactic training, the model with GS scores higher since it does not lose semantics.
QQP QQP (question pair dataset) contains more sentence pairs that paraphrase or have completely different meaning than sentence pairs that have been syntactically rewritten. Therefore, there is lit- tle difference between the models with and without additional syntactic training, and the models with additional syntactic training score slightly lower. This result suggests that syntactic knowledge is not necessary to solve this task.
The models with additional syntactic train- ing achieved higher accuracy than those without additional syntactic training, with trend towards higher accuracy on the deprel task. This indicates that the deprel task effective capturing the over- all structure relatively long answer sentences. We could not observe any differences in scores between the different optimizers.
QNLI


Reviews:
[{"review_id": "XBdEmOzeLq", "reviewer": "Reviewer_TqHP", "summary": "", "questions": "a)  Is the EWC procedure something first designed in this work?  \n", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your highly informative and very detailed comments and questions.\nDiscussions with you will be helpful to take our paper in a better direction.\n\n\nFirst, we would like to share the importance of research that embeds syntactic information in BERT and evaluates its performance in application tasks.\nSmall language models such as BERT are still widely used in real applications especially where computational resource is limited. Hence we started with BERT to see if our approach works for models of this size. As you pointed out, there are several studies that have examined whether models such as GPT have linguistic information and can answer those questions, and the newer models have improved their performance. https://aclanthology.org/2022.blackboxnlp-1.24.pdf . It is an interesting research direction to verify the effectiveness of our approach for such larger models, but we believe that our paper is significant in that we have revealed what kind of syntactic information embedded in the models improves performance in which application tasks. To the best of our knowledge, our study is the first to address this research question.\n\n\nRegarding your second reason to reject regarding baselines: our objective is to embed syntactic knowledge into a model without any modification of its model structure because, when we apply any method that changes the model structure such as adapters, it would be difficult to see whether the difference in performance comes from the change in structure or additional syntax training itself. Given a model structure, our objective is to confirm that the model benefits from the additional syntax training. \nConsidering the purpose of this discussion, we did show the baselines as \"none\" column in Table 4 and Table 7, without any syntactic additional pre-training.  Thus other rows clarify the effects of multiple types of syntactic features and optimization techniques. \nWe would like to update our paper to emphasize this in the camera ready version.\n\n\nRegarding your 3rd and 4th reasons to reject regarding design choices and motivation: thank you for your insightful comments.\nTo measure the ability of our approach to adjust the balance of sensitivity to the syntax and semantics of BERT, the metric should be task-agnostic. That is why we chose PPL as such a metric to observe the degree of holding semantics. We agree that there are other metrics better for our purpose and it is interesting to see how domain drift affects the model performance.\n\n\nWe would also like to mention the importance of our simultaneous work on both optimization and syntactic pre-training. As we stated earlier, it was not clear what syntactic structures are important for what tasks since there has been no experiment that properly incorporated each syntactic item into the model. (This includes the task you commented on regarding key phrase extraction.) Therefore, it is worthwhile to combine an appropriate optimization method with pre-training of each syntactic structure.\n\n\nFinally, let us reply to your comment on the choice of GS and EWC. As we described in the paper, one major objective of our study is to analyze the (positive) impact of optimization methods to catastrophic forgetting, not to try any possible combination of an optimization method and a task and see which is the best. We agree that choosing different downstream tasks and/or optimization methods may give a slightly different result, but we believe that our findings will shed some light on future progress in incorporation of syntax into LLMs.\n\n\nWe would be happy to reflect these discussions to the camera ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Both methods for catastrophic forgetting achieve respectable improvements in performance on GLUE in tasks requiring syntactic knowledge.\n\n- Good experimental rigor\n-- Many comparisons\n-- Many tasks\n-- Each comparison computed across multiple runs/hyperparameter settings", "reasons_to_reject": "- The PLMs used (BERT) are by current standards, quite old, and quite small.  As work in scaling PLMs up to sizes orders of magnitude greater, performance on syntactic tasks has shown to improve naturally (along with many other useful emergent forms of knowledge).  Some comparison to larger models / application of this method to such models, is necessary to ensure that the method has any practical purpose.\n\n- There are also no baselines from existing work.  There are other forms of fine-tuning, such as adapters, which seek to add additional knowledge to PLMs with less chance of catastrophic forgetting.  The authors even cite one of these papers.  This is also a confusing oversight, because the many appropriate inline citations which contrast various decisions in this work to decisions in existing work demonstrate a great familiarity with the literation, so lacking any comparison to any existing methods is an odd oversight.\n\n- Some questionable design choices.  Perplexity is used as a measure of the model retaining semantic information after fine-tuning, and while that does relate to the original task, there are also aspects of domain drift which are possible and separate from catastrophic forgetting.  How are such factors controlled?\n\n- Related, there is questionable motivation.  Often when talking about catastrophic forgetting, both the original training and the new task are both relevant.  This is clear in the context of robotics, where learning a new behavior should not result in hindering the robot from performing existing behaviors.  Is this true in this case?  PPL is almost always not a valuable end goal, and the entire PLM/LLM paradigm is built around this notion of pre-training in whatever way leads to learning useful linguistic representations, before fine-tuning, aligning, or few-shotting the model towards the task the user actually cares about.  If users never care about both tasks in approximately equal measure, than what good is retaining the original model weights which were not pertinent to the target task?\n\n- There's arguably too much going on here.  The focus of the paper aims to be about catastrophic forgetting, but secondary to that, is also the problem of matching the right syntactic fine-tuning task to the right problem.  This is not entirely known a priori, so all possible pairings are explored, but realistically a good guess can be made (as it would likely be if pursued in a more practical setting).  For instance, it is no surprise that the phrase syntax task helps with key phrase identification.  The disadvantage of the exhaustive approach is that it has both distracted from the main takeaway points while cutting into the space available for supporting the main hypothesis.\n\n- No inclusion of baselines from existing work / SOTA on performance tables\n\n- Key extraction F1 results are better than standard optimizers, but negligibly so.\n\n- No discussion of GC vs EWC.  When a priori would you choose which method?  If the paper included only one such method, traditional optimizers would be the best choice in most situations.", "questions_to_authors": "a)  Is the EWC procedure something first designed in this work?  \n", "paper_topic_and_main_contributions": "The authors argue that PLMs lack \"sufficient\" syntactic knowledge, and seek to add this knowledge via fine-tuning. They propose four different syntactic tasks to use during fine-tuning, however, if done naively, fine-tuning on a task which differs from the original pre-training brings the risk of catastrophic forgetting.  Four different methods for optimization (including two which are designed to prevent catastrophic forgetting) are compared on a number of benchmarks from GLUE, and a separate key phrase identification task.  At least one of the catastrophic forgetting-based optimizations typically offer the highest performance on each of the presented GLUE task, and better retain PPL performance.\n", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Cite GS when discussing it.\n\n- L345: Missing appendix reference"}, "extra_text": {}}, {"review_id": "QDZx6l6qD9", "reviewer": "Reviewer_9i8A", "summary": "", "questions": "(A) What about the other GLUE tasks? Did you just do experiments on the subset presented in the paper?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and pointing out the missing reference. \n\nMany studies (Tian et al., etc.) that embed syntactic structures in language models use relation extraction or similar tasks as the applied task.\nWe select GLUE and key phrase extraction as the application tasks because we would like to investigate whether syntactic structures are useful in more general tasks.\nWe also include results for the other five GLUE tasks in Appendix, showing that some tasks benefit from the syntactic structure while others do not.\nWe think that our paper is worthwhile in that it analyzes in which tasks models with which syntactic structures achieve good performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The method chosen by the authors seems to be fairly general and the results look good", "reasons_to_reject": "Given that the main contribution consists in reinforcing the syntactic knowledge in a transformer model, the authors could discuss a bit more of the substantial body of literature that tries to do exactly that - on different tasks, or using techniques that are much more cumbersome, etc. - as a reader it isn't clear to me whether relation extraction or some other syntax-heavy task would profit even more from this.", "questions_to_authors": "(A) What about the other GLUE tasks? Did you just do experiments on the subset presented in the paper?", "paper_topic_and_main_contributions": "To improve the peformance of pretrained transformer (BERT) models on tasks that use syntactic knowledge, the authors examine methods to train the model on a syntax-correlated task (dependency relations, phrases, clause status, coordination) together with a specialized optimizer that is intended to mitigate the catastrophic forgetting problem, namely Gradient Surgery or Elastic Weight Consolidation.\nThe authors find that the syntactic knowledge incorporated in this way improves performance on a subset of the GLUE tasks that they examine as well as for key phrase extraction", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Bai et al. (ACL 2021) Syntax-BERT: Improving Pre-trained Transformers with Syntax Trees"}, "extra_text": {}}, {"review_id": "rt1cx0wQA6", "reviewer": "Reviewer_ahUV", "summary": "", "questions": "Can you please provide information regarding the average performance in GLUE and whether the new method demonstrates any improvement in this score?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your comments especially on the GLUE experiments of our paper.\n\nWe would like to emphasize the contributions of our paper.\nThe main contributions of our paper are twofold: incorporating syntactic knowledge into the model using EWC and GS without changing the structure of the model and analyzing what kind of syntactic knowledge works for what kind of tasks.\nWe are confident that these contributions will be also informative when using LLMs other than BERT in the future.\n\n\nFor the GLUE task results, the model using GS and EWC was higher than the model using AdamW and SGD for the average score across the eight GLUE tasks. In the comparison between the none model without syntactic pre-training and the model with syntactic pre-training, the effect of syntactic pre-training varied from task to task. This is an important contribution in that it shows that different tasks require different syntactic structures, and it also reveals what tasks require syntactic structures.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper extensively investigates the integration of syntactic knowledge into a language model through additional training. It introduces four pretraining tasks that effectively contribute to this goal. The clarity of the ideas presented and the overall ease of following the paper are commendable.\n2. The ablation study conducted on four optimizers and four syntactic tasks provides a comprehensive analysis. The inclusion of these experiments sufficiently covers and evaluates the impact of these components.", "reasons_to_reject": "1. The paper's novelty appears to be somewhat limited, as syntactic training has been widely used in previous methods, and the GS and EWC optimization functions are not entirely new.\n2. The study only explores encoder-only language models, and it may have been beneficial to expand the analysis to also include more commonly used decoder-only language models. For instance, it would have been useful to evaluate the proposed approach on models such as GPT.\n3. Despite leveraging external syntactic training, the study only demonstrates improvement on four of the tasks in GLUE, with no clear advancements observed in the remaining five tasks.", "questions_to_authors": "Can you please provide information regarding the average performance in GLUE and whether the new method demonstrates any improvement in this score?", "paper_topic_and_main_contributions": "The paper delves into the incorporation of syntactic knowledge into a language model through additional training. It introduces four pretraining tasks that focus on learning diverse syntactic perspectives. Additionally, the paper explores the utilization of gradient surgery and elastic weight consolidation methods as preventive measures against catastrophic forgetting. The main contributions of this research encompass: (1) the proposition of additional syntactic training incorporating four distinct syntactic tasks for language models, (2) the exploration of gradient surgery and elastic weight consolidation methods to mitigate catastrophic forgetting, and (3) the demonstration of the improved performance of the enhanced model across tasks such as CoLA, RTE, MRPC, and key phrase extraction.\n\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides solid experiments over a variety of settings, making a convincing case quantitatively for the claim of improvement over the baseline that does not include training with explicit syntactic knowledge. An issue brought up in each of the three reviews is proper placement of the proposed approach in the context of existing work on adding syntactic knowledge of language models. Although direct quantitative comparisons may not be straightforward, the authors can still present a qualitative case for the approach, contrasting it more directly with existing work in greater detail, since the reviewers identified this as a weakness of the paper. A second issue is whether the approach would be effective with other kinds of models, especially those in more current use, but this is beyond the scope of the current paper. Overall, the paper presents interesting results that add to the understanding of a kind of language model that may not be the most current, but was widely used in recent years and is still relevant.
Decision: Accept-Findings