Mixture-of-Linguistic-Experts Adapters for Improving and Interpreting Pre-trained Language Models
Authors: Raymond Li
Gabriel Murray
Giuseppe Carenini
Venue: EMNLP 2023 Findings (2023)
Keywords: adapters
graph neural networks
parameter-efficient fine-tuning
interpretability
dependency trees
TL;DR: In this work, we propose a method that selects adapter modules encoding linguistic-graphs based on the values from learned Gumbel-Softmax gates.

Abstract: In this work, we propose a method that combines two popular research areas by injecting linguistic structures into pre-trained language models in the parameter-efficient fine-tuning (PEFT) setting. In our approach, parallel adapter modules encoding different linguistic structures are combined using a novel Mixture-of-Linguistic-Experts architecture, where Gumbel-Softmax gates are used to determine the importance of these modules at each layer of the model. To reduce the number of parameters, we first train the model for a fixed small number of steps before pruning the experts based on their important scores. Our experiment results with three different pre-trained models show that our approach can outperform state-of-the-art PEFT methods with a comparable number of parameters. In addition, we provide additional analysis to examine the experts selected by each model at each layer to provide insights for future studies.

Mixture-of-Linguistic--EPprr Adapters for Improving and Interpreting Pre-trained Language Models
Raymond Lit, Gabriel Murray', Giuseppe Carenini University of British Columbia, Vancouver, BC, Canada 4 University of Fraser Valley, Abbotsford, BC, Canada {raymondl, carenini]@es .bc. merpiiellurrry@u.v.cc
Abstract
precise connection between the existence of struc- tures and their benefits to task performance is yet to be firmly established. On the other hand, while the conventional way of fine-tuning has found success in wide array of NLP tasks, its applicability has in- creasingly diminished due to the associated compu- tational expense with the recent shift towards larger and more complex models (Zhao et al., 2023).
this work, we propose a method that com- bines two popular research areas by inject- ing linguistic structures into pre-trained lan- guage models in the parameter-efficient fine- tuning (PEFT) setting. In our approach, parallel adapter modules encoding different linguistic structures are combined using novel Linxturr-o--Lnguuistt------ Expiitss architec- ture, where Cumbel-Softmax gates are used to determine the importance of these modules each layer of the model. To reduce the num- ber of parameters, we first train the model for a fixed small number of steps before pruning the experts based on their importance scores. Our experiment results with three different pre- trained models show that our approach can out- perform state-of-the-art PEFT methods with comparable number of parameters. In addition, we provide additional analysis to examine the experts selected by each model at each layer to provide insights for future studies.
While some have argued that pre-training unstructured text alone equips the model with suffi cient capacity comprehend the meaning of lan- guage, others (Bender and Koller, 2020; Prange et al., 2022) have asserted that mapping the model's behavior onto human- compreencompe structures offers more dependable evidence of its ability to tackle tasks beyond merely exploiting superficial cues. Specifically, studies in this area have yielded successful attempts to inject syntactic and seman- tic structures into the pre-trained language models (Bai al., 2021; Wu al., 2021; Yu al., 2022), with positive results reported on downstream tasks. However, despite recent efforts, no existing work has addressed the problem of where and how to effectively inject multiple different structures in an efficient manner.
1 Introduction
recent years, pre-trained language models have become the de facto instrument for the field of natu- ral language processing (NLP) (Devlin et al., 2019; Liu et al., 2019; Clark et al., 2020; He et al., 2021. 2023). This shift is largely due the emergence and success of transformer-basee models (Vaswani al., 2017) where large-scall pre-training helps the model to learn the syntactic and semantic struc- ture a language without explicit supervision. At the same time, there are good reasons to question whether these models can be said to understand a language in any meaningful and interpretable way (Trott et al., 2020; Merrill et al., 2021). To ad- dress this conundrum, probing studies have demon- strated, certain extent, that it possible infer linguistic structures from the representations within these models (Hewitt and Manning, 2019; Tenney et al., 20;9b; Maudslay et al., 2020). However, the
The conventional approach of fine-tuning pre- trained NLP models involves optimizing the full set of model parameters for each task. However, this results separate copy of fine-tuned model pa- rameters for each task and has become increasingly infeasible due to the recent trend pre-training larger and larger models. To address these con- cerns, a surge of recent work has been dedicated to the study of parameter-efficien fine-tuning (PEFT) methods (Ding et al., 2023), where only small portion of task-specifie trainable parameters are tuned while keeping the rest of the model frozen. While these studies have achieved impressive per- formance even comparable to the full fine-tuning, they have been mostly focused on either determin- ing the subset of model parameters for tuning (Lee

---

al., 2019; Ben Zaken et al., 2022) or finding the location to insert additional trainable parame- ters (Houlsby et al., 2019a: and Liang, 2021; Hu et al., 2022). No existing work has addressed the problem of whether linguistic structural priors can be incorporated into these trainable parameters under the PEFT setting.
2 Related Works
We organize this section based on the two research areas that our work seeks to align. In $2.1, we pro- vide an overview of techniques to inject linguistic structure, while $2.2 summarizes recent trends parameter-efficien fine-tuning.
In this work, we align the two research ar- of injecting linguistic structures and PEFT by proposing a strategy of effectively combining multiple linguistic structures into pre-trained NLP models in a parameter-cfficien fashion. To com- bine multiple linguistic structures, we propose a novel architecture inspired by Mixture-of- models (Shazeer et al., 2017), where Relational Graph Convolutional Networks (RGCN) modules (Schlichtkrull et al., 2018) encoded with differ- ent linguistic trees are aggregated using learnable Gumbel-Softmax (Jang al., 2017) gates, and in- serted between each layer of the pre-trained model. To reduce the number of parameters, we propose a pruning strategy where we first tune the full set of RGCN modules before pruning all but the top "experts" based on the importance score learned from the gates. To demonstrate the benefits of our approach, we perform experiments the GLUE benchmark with three different pre-trained NLP models and compare the results with state-of-the- art PEFT methods (Mao et al., 2022). Further, we perform additional analysis to understand which types of linguistic structures are kept at each layer of the model and provide insights for future work injecting knowledge through PEFT methods. In short, our contributions can be summarized as the following:
2.1 Injecting Linguistic Structures
Earlicr works on injecting linguistic structures into neural networks are often based on the recursive neural network architecture (Goller and Kuchler, 1996; Socher et al., 2011, 2012, 2013), where a compositional function recursively combines rep- resentations of child nodes following predefined tree structure. Following the same intuition, sub- sequent studies have extended their approach for composing hidden states into a variety of neu- ral architectures including recurrent neural net- works (RNNs) (Tai et al., 2015; Miwa and Bansal, 6016; Roth and Lapata, 2016; Kuncoro et al., 2017; Shen et al., 2019), graph neural networks (GNNs) (Marcheggiani and Titov, 2017; Bastings al., 2017: Zhang et al., 2018; Huang and Carley, 2019;
Wang al., 2020), and later, Transformers (Wu et al., 2018; Hao et al., 2019; Strubell al., 2018: Wang et al., 2019b,c). For instance, Strubell et al. (2018) 8) used the bi-affine operator (Dozat and Man- ning, 2017) to predict the affinity score between the token representations (key and query vector) based on the dependency tree, while (Wang et al.. 2019c) encouraged the attention heads to follow tree structures by applying constituent prior on the attention weights.
More recently, research in this area has shifted towards pre-trained language models (Devlin et al.. 2019; Liu et al., 2019; Clark et al., 2020; He et al., 2021, 2023). While prior studies probing (He- witt and Manning. 2019; Tenney al., 20190 9b; Maudslay et al., 2020; Newman et al., 2021: Arps et al., 2022) have shown that meaningful hierar- chical structures (e.g., syntactic trees) can be ex- tracted from pre-trained models without explicit supervision, has also been found that incorporat- ing linguistic structures can still be beneficial for downstream performance (Zhang et al., 2020; Kun- coro et al., 2020; Sachan al., 2021; Qian et al., 2021), even when the structures already exist in the model (Li et al., 2022). For example, Bai et al. (2021) explicitly masked the existing pre-trained attention weights based on the adjacency matri- ces defined by the syntactic trees. On the other
1. We propose a novel architecture to effec- tively combine and interpret multiple linguis- tic structures at different layers of the pre- trained model.
2. To improve efficiency, we adopt a pruning strategy by keeping only the top experts ac- cording to their importance scores.
3. Our experimental results with three different models demonstrate the benefits of our ap- proach by achieving the best overall perfor- mance on the GLUE benchmark.
4. We perform analysis on the experts selected by the model to providing valuable insights for future work.

---

hand, Wu al. (2021) used additional GNN layers incorporate semantic dependencies by append- ing them on top of the pre-trained encoder. Most similar to our work, Yu et al. (2022) extended the approach by Wu et al. (2021) and performed an empirical study on syntax and trivial graphs. How- ever, their method requires training new model for each graph, which is inefficient for studying their benefits at different layers of the model. To the best of our knowledge, no existing works have attempted to incorporate multiple different linguis- tic structures within the same model, as we do in this paper.
3 Model Architecture
In this section, we describe the architectures of our fixturee-f-LLingiiie adapters (Figure 1). We start by first introducing the Relational Graph Convolu- tional Network (RGCN) modules for incorporating linguistic structures ($3.1) before describing the method used for combining multiple RGCN ($3.2).
Finally, we discuss how the adapters are inserted into the pre-trained model ($3.3).
3.1 Modeling Dependency Structures
To model dependeney structures, adopt the method proposed by Wu et al. (2021), where RGCN (Schlichtkrull al., 2018) layers are used to propagate node representations according to the structure defined by the dependency tree.
2.2 Parameter Efficient Fine-tuning
While the standard paradigm of fine-tuning pre- trained language models has emerged as a common practice for NLP tasks (Min et al., 2021), it has become less applicable due to the computational cost associated with the increasingly large models (Brown al., 2020a; OpenAI, 2023). Parameter- efficient fine-tuning (PEFT) methods (Ding al.. 2023), on the other hand, present solution this problem by freezing most or all of the pre-trained weights and only fine-tuning a small set of parame- ters in proportion to the model size. PEFT methods can be roughly organized into two categories. The first category tunes subset of existing parameters with notable examples including freezing entire layers (Lee et al., 2019) or tuning only the bias terms (Ben Zaken et al., 2022). However, these approaches generally lead to worse performance and have only been shown to achieve comparable performance to full fine-tuning on low-resource tasks. Alternatively, the second category adds new trainable parameters while keeping the pre-trained weights frozen (Han et al., 2021; Karimi Mahabadi al., 2021; Lester al., 2021). For example, Houlsby al. (2019a) used trainable bottleneck layer after the feed-forward network in each layer the model, Li and Liang (2021) prepended train- able vectors to the input of multi-head attention, while Hu et al. (2022) combined the pre-trained attention weights with trainable low-rank matrices. Lastly, more recent studies (He et al., 2022; Mao et al., 2022) proposed a unified framework by com- bining different PEFT methods as sub-modules. While we use their approach as our baselines, no existing PEFT works have attempted incorporate interpretable structures as priors to the trainable modules, as we do in this paper.
WPh
W (E-1)
(0 = ReLU Σ
+ Wolk (C-1)
PER JEN:
(1)
Equation describe the propagation process for single RGCN layer, where the node representa- tion h2 is updated with learned composition func- tion based on the node's neighbors h, E NN (and itself) in the dependency graph. Specifically, we use the intermediate hidden states of the pre-trained model as input, where sub-word token vectors are pooan-pooled to create the node representation for the associated word. Since the number of parame- ters in RGCN linearly increases with the number of relation types, rather than associating each de- pendency relation with a separate set of weights W,, we only model the child and parent relations (RR) = 2) to reduce parameter count.
The graph convolution operation has compu- tational complexity O| d1 d2), where dj and d2 are respectively the number of input and out- put dimensions of the layer, and E is the total number of edges defined by the dependency graph, In addition, the self-loop operation in the RGCN layer adds complexity O([N) di d2), where N = E + is the total number of nodes or word tokens in the dependency graph. The self-loop op- eration has the same complexity as the standard linear layer.
3.2 Combining Different Modules
Inspired by the Mixture-of -f--Exerts architec- ture (Shazeer et al., 2017), we propose a strategy

---

Weighted Sum
Sampled Gates
Feed Forward Layer
Expert
Expert
Expert
Expert
Multi-head Self-Attention Layer
Figure I: Our proposed Mfxture-of- Linguistic- architecture for single transformer layer. In the four-expert configuration provided by the example, where the outputs from the expert modules are aggregated based on the weights sampled from the Gumbel-Softmax distribution.
determine the importance of different adapter modules by sampling gate values from Gumbel- Softmax distribution (Maddison et al., 2017; Jang et al., 2017). Specifically, we define a gate logit for each "expert" module E;, where the gate value 9: is sampled from the Gumbel-Softmn dis- tribution during training. The sampling method is defined
($3.2) and update only the adapters while keep- ing the pre-trained parameters frozen. We choose to insert modules following the suggestions by He et al. (2022), where they found that inserting adapters in parallel to the feed-forward networks (FFN) achieves the overall best performance.
halin MultiHeaddAttn(hh
(0
FFN + Adapter(h
9. softmax + c)/r
(2)
From Equation 3, our adapter module takes the hidden output hamm from the multi-head attention (MultiHeadAttt) sub-layer and uses an additive composition with the original FFN to create the final layer output (!)
where the stochasticity comes from the gumbel noise log(-) s.t. u Uniform(O, 1). and is the temperature to control the randomness the distribution. The value of the gate logit 2; can be interpreted as the contribution the respec- tive expert module when computing the aggregated representation from all experts.
Training Strategy
In contrast to the softmax gates used in the origi- nal MoE architecture (Shazeer et al., 2017), sam- pling gates from the Gumbel-Sofftax distribution provides more interpretability regarding its impor- tance since the inherent stochasticity introduces an exploratory characteristic and allows the model to consider a diverse set of potential outputs. Mean- while, the standard softmax operation assumes single correct answer at each layer, meaning the model could possibly overlook good combinations of modules by locally exploiting the module with the single highest probability.
While the architecture proposed section 3 lows us to aggregate multiple adapter modules at each pre-trained layer, it significantly decreases the efficiency due to the number of task-specific pa- rameters used during training and inference. To address this issue, we propose pruning strategy to reduce the number of experts.
In order to decide which expert to keep at each layer, we first fine-tune the full set of expert mod- ules using our Viittureof-Linguistti--e-e tecture (Figure 1) for fixed small number of steps. After the gates have converged, importance score from the gates can be used to determine which ex- perts keep. While iterative pruning strategy (Michel al., 2019; Behnke and Heafield, 2020: Tan and Motani, 2020) can also be used, it less ficient due to the requirement of more training steps. Finally, after the pruning process, we restart the
archi-
3.3 Adapters
Based on prior works on parameter efficient fine- tuning (Houlsby al., 2019b; Mao et al., 2022), we inject our Mixture- Linguistic-Experr layer be- tween the layers of pre-trained transformer model

---

training process and fine-tune the resulting model with one expert module per layer.
Edgeless Graphs In addition the three linguis- tic graphs, we also apply straight-forward nonlin- ear transformation using MLP layers. The intuition is that at some layers, injecting structures might be unhelpful (or even detrimental) to the task perfor- mance when the linguistic prior cannot be utilized based on the representation learned by that layer.
5 Experiments
We describe in detail the experimental settings and results in this section. We start by providing brief summary of the linguistic graphs ($5.1) before describing the datasets ($5.2) models ($5.3), and the hyperparameters settings ($5.4). Finally, we present the results in $5.5.
5.2 Datasets
Dataset
Task
Train
Dev
CoLA
Acceptability
1K
1.74
5.1 Linguistic Graphs
RTE
Entailment
2.5K
278
In our experiments, we use three different linguis- tic graphs to encode sentence-level structures, Fol- lowing prior studies (Wu al., 2021; Yu al., 2022), we infuse the semantic and syntactic depen- dency trees as well as sequential bidirectional graph into three separate RGCN adapter modules for each layer of the pre-trained model. In addition, to account for scenarios where structures are either not needed or harmful, we also use multi-layer perception (MLP) module to represent an edgeless graph, where no composition is performed.
MRPC
Paraphrase
2.7K
409
STS-B
Similarity
5.8K
1.5k
SST-2
Sentiment
67K
873
QNLI
Entailment
105k
5.5K
QQP
Entailment
363K
40K
MNLI
Entailment
392k
9.8K
Table 1: The statistics of the datasets in the GLUE benchmark, ordered by the size of the training set.
We conduct all our experiments on the GLUE benchmark (Wang et al., 2019a), consisting of a comprehensive suite of natural language under- standing tasks. The benchmark contains eight datasets for text classification, including linguis- tic acceptability (CoLA), sentiment analysis (SST- 2), similarity and paraphrase tasks (MRPC, STS- B, QQP), and natural language inference (MNLI, QNLI, RTE). For evaluation metric, we use Matthew's Correlation for CoLA, F1 for MRPC and QQP. Spearman's 's Rank-Order Correlation for STS-B, and Accuracy for SST-2, RTE, QNLI, and MNLI Following prior studies (Houlsby al., 2019h; He et al., 2022), we exclude the WNLI dataset from our experiments due to its limited cov- erage. The statistics of the datasets are presented in Table 1.
Syntactic Trees In syntactic parses, each word in the sentence is assigned syntactic head based on Universal Dependencies (UD) formalism (de Marn- effe et al., 2021). We the Hi-LSTM-based deep biaffine neural dependency parser (Dozat and Man- ning, 2017) trained on the English UD treebank from the Stanza library (Qi et al., 2020).
Semantic Trees Based on the DELPH-IN de- pendencies formalism (Ivanova al., 2012), se- mantic parses assign word dependencies based on predicate-argument relations. In contrast to syntac- tic graphs, words that do not contribute to the mean- ing representation of the sentence do not appear in the semantic graph. The graphs are extracted with a neural transition- bunsttion-based parser (Wang et al., 2018; Che et al., 2019) trained on the CoNLL200 shared task (Oepen et al., 2019).
5.3 Models
In our experiments, we apply our methods to three different pre-trained language models: BERT, RoBERTa, DeBERTaV3. RoBERTa (Liu et al., 2019) enhances BERT (Devlin et al., 2019) by in- corporating more training data and removing the next-sequence prediction objective, DeBERTa (He et al., 2021) introduced a disentangled attention mechanism for encoding relative positions at ev- cry layer, while DeBERTaV3 (He et al., 2021) im- proved upon the prior versions by adapting the
Sequential Bidirectional Graphs We also use a drrighttforwart sequential bidirectional graph that connects word tokens in sequential order. This allows the RGCN layers aggregate local information rather than potentially long dependen- cies, where has shown the ability to improve task performance when injected into pre-trained trans- former layers via fixed attention (Li et al., 2022).

---

Method
CoLA
RTE
MRPC
STS-B
SST-2
QNLI
QQP
MNLI
Average
BERT
Full Fine- Tuning
62.08
66.43
90.94
89.76
91.63
89.95
87.35
83.23
82.67
Adapter
61.31
71.84
89.86
88.63
91.86
90.55
86.78
83.14
83.02
Prefix-tuning
55.37
76.90
91.29
87.19
90.94
90.39
83.30
81.15
82.07
LoRA
60.47
71.48
90.03
85.65
91.51
89.93
85.98
82.51
82.20
UniPELT (AP)
61.15
71.84
90.28
88.86
91.86
90.77
86.74
83.41
83.12
UniPELT (APL)
61.53
73.65
90.94
88.93
91.51
90.50
87.12
83.89
83.51
Ours
61.49
70.36
90.43
88.71
92.66
93.03
87.82
84.30
83.60
RoBERTa
Full Fine-Tuning
68.0
86.6
90.9
92.4
96.4
94.7
92.2
90.2
88.9
UniPELT (APL)
61.91
74.31
91.74
90.26
93.92
92.00
87.68
87.23
84.88
Ours
62.20
72.32
92.77
90.34
94.27
92.53
88.29
87.83
85.07
DeBERTaV3
Full Fine- Tuning UniPELT (APL)
90.7
68.02
81.59
92.42
91.70
95.64
93.65
89.60
89.13
87.72
Ours
69.93
79.42
93.38
91.01
95.84
93.92
89.83
89.47
87.85
Table 2: Results on the GLUE benchmark for BERT, RoBERTa, and DeBERTaV3. For BERT, the full fine-tuning and PEFT baseline results are directly copied from Mao et al. (2022). For both RoBERTa and DeBERTaV3 V3, the UniPELT results obtained using the AdapterHub implementation (Pfeiffer et al., 2020), while the full fine-tuning results are copied from their original papers (RoBERTa only reported precision to the tenth decimal place, while DeBERTaV3 only reported the base model QQP). All our results are averages over three seeds, with statistically significant improvements (>99% confidence Bootstrap Test) over UniPELT highlighted in bold.
replaced token detection objective (Clark et al., 2020). For all models, we use the standard variant with 12 layers and 12 heads. For baselines, we use the unified framework for parfmeter-efficient language model tuning (UniPELT) proposed by (Mao et al., 2022). Since the results from the origi- nal paper demonstrated superior performance over other PEFT methods (Houlsby et al., 2019a; Li and Liang, 2021; Hu al., 2022), we only report the re- sults for these methods for BERT. For all tasks, we apply a classifier on the [CLS] token representation from the last hidden layer.
from 5 0.1 over 1000 steps. The intuition be- hind temperature annealing to allow the model to start with a more exploratory behavior before gradually becoming more exploitative. Lastly, we also scale the adapter output by a constant factor of 4 as proposed in the work by He et al. (2022).
5.5 Results
From the results in Table 2, we see that our ap- proach achieves the best overall performance on the GLUE benchmark. For individual tasks, al- though our method lags behind UniPELT in the low-resource tasks of RTE and STS-B, our method achieves consistent improvements in the four tasks with the highest number of training examples (Ta- ble 1): SST-2, QNLI, QQP and MNLI, where the improvements for SST-2 and QNLI are statistically significant for two out of the three models, and QQP and MNLI for all three models. This is con- sistent with the findings by prior work (Mao et al., 2022; Chen et al., 2022), where they found that while existing PEFT methods excel in low-resouree tasks, they still struggle to yield consistently com- petitive performance in medium and high-resource settings. However, we believe that learning to use the linguistic structures associated with the depen- dency trees requires more tuning and can outper-
5.4 Hyperparameters
Both MLP and RGCN adapter modules consist of two hidden layers with bottleneck dimension of 48. Since RGCN modules require 3x the number parameters as MLP modules, we only select the top-2 RGCN modules based on their gate values. Following the settings by Mao et al. (2022), we set the input length to 128, and train for a total of 50 epochs for with learning rate of 5e 4 and batch size of 16. During the initial steps of train- ing our Mixture-of-Linguistic Linguistic-Expert model, follow the suggestions from prior work (Huijben et al., 2022) and apply temperature annealing (Jang et al., 2017) to gradually decrease the temperature

---

BERT
RoBERTa
DeBERTa
GAT
MAPPC
DuL-
RTT
CHALL
BTSR
MALLI
DOP
SAT
MRPIC
Cui-A.
RTV
OM&L
MINLI
GCP
35T
Cul.A.
RTB
OMLI
VNNII
00P
12
OUT
Dyy
Seq
Sar
Sec
5pm
12
5U7
Sam
14.P
Sar
5p1
12
Gam
Spn
Oye
Syn
5m
Sec
Spm
9ym
18
80g
S9g
D0c
MIP
56
D4g
Emn
11
MPPP
M4P
S0n
960
5e2
MJP
13
5eg
Se0
8om
ML2
OUY
Sen
9k0
10
MILP
MLP
M8.
KUPP
MIP
VLP
MUP
10
MUP
D64
10.7
11LP
18.2
1ULP
MEP
VLP
MUP
18LP
VIL
MUP
MIPP
MUP
12
MIP
VLP
MUP
MPP
MLP
MP.P
64.
MLP
M8,9
MAP
MJF
MIP
VIPP
MIF
18.P
M12
M.P
MID
MLP
MIL
MUP
MILF
MUP
VLP
MUP
MUN
10.27
11P
18.27
MC2
11LP
14.
11.
11P
MIPP
M8PP
MIP
ULP
MUP
MIP
MLP
10
M1P
VLP
M,F
Layer
MIPP
MLPP
MEF
MPP
MUP
MPP
12.0
MUP
M.P
VLP
16.
M1P
USP
MUP
MUP
V0P
MIP MPP
10.
MIP
VLP
MPP
MPP
MLP
M.P
16.
01P
18.
MPP
MIP
VEP
MLP
18P
M12
M.P
V1P
MPP
MEP
10.7
10.
10.
KUP
HPP
0LP
MPP
HPP
9UP
10.7
10.7
117
10.7
10.
10.7
HUP
VEP
017
10.7
V1P
NUP
Figure 2: Theee heatmaps illustrate which of the four expert modules are used at cach layer for the three models. We use green, red, blue, and grey to represent the syntactic (Syn). semantic (Sem), sequential (Seq), and MLP expert. respectively.
form standard PEFT methods when there are more training data available. Lastly, it worth high- lighting that the application of our approach to the RoBERTa model resulted in a notable increase in performance (+0.19) over the baseline, surpassing the gains observed with BERT (+0.09) and De- BERTa (+0.13). Since RoBERTa is pre-trained on larger corpus than BERT, hypothesize that this discrepancy could be due to the fact that RoBERTa has learned more meaningful representation for understanding linguistic structures. Conversely, the advantage of the injected linguistic structures could be somewhat offset by the more sophisti- cated pre-training methodology employed by De- BERTa. Lastly, note that while Mao et al. (2022) reported that their method (UniPELT) achieved significantly better performance compared stan- dard fine-tuning', our experiments with RoBERTa yielded the opposite conclusion This consistent with the findings by Chen et al. (2022), where they find that PELT methods are highly unstable and can- not achieve consistently competitive performance compared to fine-tuning (especially in medium- to high-resource settings). Therefore, we hypothesize the discrepancy between our results and theirs is due to the likely extensive hyperparameter search conducted by (Mao et al., 2022), whereas we used the identical hy perraarameteer settings across all ex- periments as reported in subsection 5.4.
In Table 3, we report the number of trainable pa- rameters for each of the methods Table 2. While increasing the number of RGCN modules or hidden layer dimensions could improve the performance of our model, our hyperparameters settings (sub- section 5.4) are selected specifically to match the number of parameters used in UniPELT (Mao al., 2022). Additionally, it is worth mentioning that we elected not to incorporate the dependency relations since the number of parameters in RGCN layers in- creases lincarly with the number of relation types.
Method
Parameters
Fine-tuning
110M (100%)
Adapter
895K (0.81%)
Prefix-tuning
184K (0. 17%)
LORA
295K (0.27%)
UniPELT (AP)
1.1M (0.99%)
UniPELT (APL)
1.4M (1.26%)
Ours
1.2M (1.14%)
Table 3: Number of trainable parameters required for each parameter-efficient fine-tuning method.
6 Analysis
In this section, we provide an analysis of the model's behavior by first examining the linguistic expert used for each model ($6.1) before examin- ing the convergence rate of at different layers of the model ($6.2).
gates
SSine the original BERT paper (Devlin et al., 2019) does not report the GLUE development results, the full fine- tuning results for BERT in Table 2 are copied from Mao et (2022).
6.1 Gate Values
Figure 2 illustrates the experts used at each layer of the models. At first glance, we can clearly see that all models tend to favor RGCN modules at the upper layers, while the standard MLP adapter is used for lower layers. This could be due to
"The full fine-tuning results for RoBERTa and DeBER- TaV3 are copied for their original papers (Liu et al., 2019; He et al., 2021), where He et al. (2021) only reported the full set of GLUE results for their large variant. In both papers, the reported results are limited to three significant digits.

---

the fact that pre-trained language models are de- signed to learn hicrarchical representations of the input, where lower-level layers typically capture the surface knowledge required to understand high- order compositions (Tenney et al., 2019a; Niu et al., 2022). Since such knowledge are generally appli- cable to all downstream tasks with minimal mod- ification even during full fine-tuning (Zhou and Srikumar, 2022), augmenting their representations with compositional structures could be detrimen- tal the performance. Similar findings have also been reported by Rücklé et al. (2021), where drop- ping out adapters in the lower layers has the least amount of impact on model performance.
1ayo 12
t1 JoMe)
o1 1a88
Layer 9
Layor e
To gain a better understanding of how differ- ent linguistic structures are utilized, we provide a qualitative comparison linguistic experts used between models. From Figure 2, we can see that the experts used between BERT and RoBERTa are very similar, with 5 out of the 8 tasks being exactly the same. In contrast, DeBERTa tends semantic and syntactic experts, with no sequential experts selected on the top layer. We believe this is due to the disentangled attention mechanism used by the DeBERTa model (He et al., 2021), where the token positions are already encoded by an ad- ditional vector at each layer of the model. Addi- tionally, we see that semantic graphs are selected the least. This could be due to the fact that we do not model the relation types, which are necessary determine the nuanced semantic roles between concepts and ideas. Conversely, the relation types in syntactic trees (e.g., subject-verb) do not provide the full meaning of the sentence beyond grammati- cal structure, where prior studies have shown that syntax trees with no relations can still be beneficial for downstream performance (Bai et al., 2021).
Layer? 7
Layer
more
Layer 5
Layer 4
Layeer
Layer 1 LaYEr
250
500
750
1000
1250
1500
1750
2000
Steps
Figure 3: Average JS-Divergence between the gate value distribution, measured in 10-step intervals.
6.2 Gate Convergence
Next, we examine the convergence rate for the gate logits by measuring the changes the gate value between steps. For the purpose of analysis, we train the full set of experts for 2000 steps while keep- ing all hyperparameters the same. Figure 3 plots the JS-Divergence between the gate values' soft- max distribution in 10-step intervals. From the plot, we can that the gate values in the lower layers change rapidly in the early iterations before con- verging. This implies that the model can quickly learn to select the MLP module ($6.1), providing further evidence against injecting structural knowl-
edge at lower layers of pre-trained models. the upper layers, while it follows a similar trend where the gates change quickly before the change curve flattens out, we still see a moderate amount of oscil- lation even after 1000 steps. This can be interpreted as the best expert not having enough relative advan- tage over the others for the model to assign high importance score. Since the main purpose of our work is propose an architecture for selecting ex- perts, we leave the in-depth investigation regarding the trade-off between different linguistic experts as an interesting venue for future work. Finally,

---

we see that almost all gates have converged at the 250-step mark. For reference, this roughly 2% of the number of steps for a single epoch on the MNLI training set. This finding demonstrates that only a small number of steps are required to learn the importance of different adapter modules.
tuning (PEFT). To start, we introduce novel framework that combines multiple linguistic struc- tures an architecture inspired by the Mixture- of- of-exxssts model, where Gumbel-Softmax gates are used to learn the importance of these experts at different layers of the model in small fixed number of training steps. Finally, we reduce the pa- rameter count by pruning all but one expert at each layer such that the resulting number of trainable parameters is comparable methods. After running experiments with three dif- ferent pre-trained models on the GLUE benchmark, the results show that our method can achieve the best overall performance while significantly out- performing the baselines on high-resource tasks. Finally, we examine the experts selected by each model and the convergence rate of the Gumbel- Softmax gates to gain better understanding of the models' behavior and provide valuable insights for future studies on knowledge injection.
6.3 Ablation Study
We perform ablation experiments study the ef- fectiveness our expert selection method based on the importance scores (section 4). To ensure a fair comparison with the results in Table 2, we only use one expert per layer while using the same architecture. We manually design the ordering of experts based on the intuition of the traditional NLP pipeline (Tenney et al., 2019a), with surface features the bottom, syntactic features in the middle, and semantic features at the top (Jawahar et al., 2019). Specifically, we use sequential -grahh encoding position information at the lower four lay- ers, syntactic trees at the middle four layers, and semantic graph at the upper four layers. We also perform experience using only one expert for the entire model as baseline.
ofate-of-the-- PEFT
For future work, we plan to perform further ex- periments to determine the relative advantage of different linguistic knowledge and study how the quality of graphs affects model performance on downstream tasks. One significant challenge is to efficiently incorporate relation types of dependency trees, which will explore in future work. In ad- dition, we plan to further improve the efficiency of our approach by incorporating findings from other recent works, such as dropping adapters in the lower layers (Rücklé et al., 2021). Lastly, we plan to extend our approach to inject linguistic struc- tures (including discourse graphs) into decoder- only architectures (Radford et al., 2019; Brown et al., 2020b) and perform studies larger model variants (Touvron et al., 2023).
Method
SST-2
QNLI
Syntax-only
92.04
86.40
Semantic-Only
85.36
85.36
Positional-Omi
88.97
88.97
Manually-Designee
91.84
89.12
Ours
94.27
92.53
Table 4: Ablation results with RoBERTa with manually selected experts, averaged across seeds.
Table 4 shows the results for RoBERTa the two medium-resouree datasets (SST-2 and QNLI). From the results, we see that while our manually- designed approach achieved better performance than the single-expert models, they still signif- icantly trail behind our automatic selection ap- proach. This finding verifies our hypothesis that augmenting the representations of lower layers with compositional structures have unrecoverable effects on the upper-layer representations used for task prediction (subsection 6. 6.1), ultimately leading a significant deterioration performance.
Limitations
One limitation of our study is that our approach (excluding sequential graphs) requires high-quality parsers to construct gold-standard syntactic and semantic trees. While our approach is generally applicable all structures, our experiments focus on sentence-level linguistic graphs on the GLUE benchmark. Other structures such as discourse trees on multi-sentential tasks remain to be ex- plored in future studies. Additionally, all our exper- iments are performed on standard variants of pre- trained encoder models, different behavior could be observed larger differently structured models, such as the decoder-only architecture.
7 Conclusion and Future Work
this work, we introduce an approach that com- bines the two popular research areas of injecting linguistic structures and parfmeter-efficient fine-

---

References
Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter. Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Πya Sutskever, and Dario Amodei. 2020b. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
David Arps, Younes Samih, Laura Kallmeyer. and Has- san Sajjad. 2022. Probing for constituency structure neural language models. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2022. pages 6738 6757, Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
Jiangang Bai, Yujing Wang, Yiren Chen, Yaming Yang. Jing Bai, Jing Yu, and Yunhai Tong. 2021. Syntax- BERT: Improving pre-trained transformers with syn- tax trees. In Proceedings of the 16th Conference of the European Chapter of the Association for Compu- tational Linguistics: Main Volume, pages 3011 3011-3020, Online. Association for Computational Linguistics.
Wanxiang Che, Longxu Dou, Yang Xu, Yuxuan Wang, Yijia Liu, and Ting Liu. 2019. HIT-SCIR MRP 2019: A unified pipeline for meaning representa- tion parsing via efficient training and effective en- coding. In Proceedings of the Shared Task on Cross- Frumework Meaning Representation Pursing at the 2019 Conference on Natural Language Learning, pages 76 85, Hong Kong. Association for Computa- tional Linguistics.
Jasmijn Bastings, Ivan Titov, Wilker Aziz, Diego Marcheggiani, and Khalil Sima' an. 2017. Graph con- volutional encoders for syntax-aware neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Process- ing. pages 1957-1967, Copenhagen, Denmark. Asso- ciation for Computational Linguistics.
Guanzheng Chen, Fangyu Liu, Zaiqiao Meng, and Shangsong Liang. 2022. Revisiting parameter- efficient tuning: Are we really there yet? In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 261 2612-2626, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Maximiliana Behnke and Kenneth Heafield. 2020. Los- ing heads in the lottery: Pruning transformer attention in neural machine translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2664-2674, Online. Association for Computational Linguisties.
Kevin Clark, Minh- Thang Luong, Quoc V. Lc, and Christopher D. Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representa- tions.
Elad Ben Zaken, Yoav Goldherg, and Shauli Ravfogel. 2022. BitFit: Simple parameter-efficieet fine-tuning for transformer-based masked models.. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics ( (Volume 2: Short Papers), pages 1-9, 1- Dublin, Ireland. Associa- tion for Computational Linguisties.
de Marneffe, Christopher D. Man- ning, Joakim Nivre, and Daniel Zeman. 2021. Uni- versal Dependencies. Computational Linguistics, 47(2):255-308
Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: On meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 5185-5199 Online. Association for Computational Linguistics.
Jacob Devlin, Ming-W Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Pupers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan. Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu. Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020a. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Ning Ding, Yujia Qin, Guang Yang. Fuchao Wei. Zonghan Yang. Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. 2023, Parameter-efficient fine-tuning of large-scale pre- trained language models. Nature Machine Intelli- gence, 233)2200-335
Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency pars- ing. In International Conference on Learning Repre- sentations.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranay Shyam, Girish Sastry, Amanda
C. Goller and A. Kuchler. 1996. Learning task- dependent distributed representations by backprop-

---

agation through structure. In Proceedings of Inter- national Conference on Neural Networks (ICNN'96) 96), volume 1, pages 347-352. IEEE.
Edward J Hu. yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang. Lu Wang. and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
Wenjuan Han, Bo Pang. and Ying Nian Wu. 2021. Ro- bust transfer learning with pretrained language mod- els through adapters. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 2: Short Papers), pages 854 861, Online. Association for Computational Linguisties.
Binxuan Huang and Kathleen Carley. 2019. Syntax- aware aspect level sentiment classification with graph attention networks. In Proceedings of the 2019 Con- ference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP- IJCNLP), pages 5469-5477, Hong Kong, China. As- sociation for Computational Linguistics.
Jie Hao. Xing Wang, Shuming Shi, Jinfeng Zhang, and Zhaopeng Tu. 2019. Multi gulti-granularity self- attention for neural machine translation. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP),, pages 887-897, Hong Kong, China. Association for Computational Lin- guistics.
Iris AM Huijben, Wouter Kool, Max B Paulus, and Ruud JG Van Sloun. 2022. A review of the gumbel- max trick and its extensions for discrete stochastic- ity in machine learning. IEEE Transactions on Pat- tern Analysis and Machine Intelligence, 43(2):1353- 1371.
Angelina Ivanova, Stephan Oepen, Lilja Øvrelid, and Dan Flickinger. 2012. Who did what to whom? a contrastive study of pyntacto-semanti dependencies. In Proceedings of the Sixth Linguistic Annotation Workshop, pages 2-11, Jeju, Republic of Korea, As- sociation for Computational Linguistics.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig. 2022. Towards a unified view of parameter-efficient transfer learning. In International Conference on Learning Representa- tions.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023, DeBERTav3: Improving deBERTa using ELECTRA- embed-
Eric Jang. Shixiang Gu, and Ben Poole. 2017. Cate- gorical reparameterization with gumbel-softmax. International Conference on Learning Representa- tions.
style pre-training with ding sharing. In The Eleventh International Confer- ence on Learning Representations.
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meet- ing of the Association for Computational Länguistics, pages 1-3577, Florence, Italy. Association for Computational Linguistics.
Pengeheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. [DEBERTA]: I (DECODING)- (enhanced) [bert] [with| | (disentangled) ] In International Conference on Learning Representa- tions.
John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word represen- tations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Pupers), pages 4129-4138, Minneapolis, Minnesota. Association for Computational Linguistics.
Rabeeh Karimi Mahabadi. Sebastian Ruder, Mostafa Dehghani, and James Henderson. 2021. Parameter- efficient multi-task fine-tuning for transformers via shared hypernetworks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Pupers). pages 565-576, Online, Association for Computational Linguistics.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019a. Parameter efficieet transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2790-2792 PMLR.
Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, Graham Neubig, and Noah A. Smith. 2017. What do recurrent neural network gram- mars learn about syntax? In Proceedings of the 15th Conference of the European Chapter the Associa- tion for Computational Linguistics: Volume 1, Long Papers, pages 1249-1255 -258, Valencia. Spain. Associa- tion for Computational Linguistics.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019b. Parameter-efficient transfer learning for nip. In International Conference on Muchine Learning, pages 2790-2799. PMLR.
Adhiguna Kuncoro, Lingpeng Kong, Daniel Fried, Dani Yogatama, Laura Rimell, Chris Dyer, and Phil Blun- som. 2020. Syntactic structure distillation pretraining for bidirectional encoders. Transactions of the Asso- ciation for Computational Linguistics, 8:776-794.

---

Jaejun Lee, Raphael Tang, and Jimmy Lin. 2019. What would elsa do? freezing layers during transformer fine-tuning, arXiv preprint arXiv: 1eee19911.00999 1.03090.
William Merrill, Yoav Goldberg, Roy Schwartz, and Noah A. Smith. 2021. Provable limitations of acquir- ing meaning from ungrounded form: What will future language models understand? Transactions of the Association for Computational Linguistics, 9:1047- 1060.
Brian Lester, Rami AI-Rfou, and Noah Constant. 2021.
The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Paul Michel, Omer Levy, and Graham Neubig, 2019. Are sixteen heads really better than one? In Ad- vances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Bonan Min, Hayley Ross, Elior Sulem, Amir
Raymond Li, Wen Xiao, Linzi Xing, Lanjun Wang, Gabriel Murray, and Giuseppe Carenini. 2022. Hu- man guided exploitation of interpretable attention patterns in summarization and topic segmentation. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 10189-10204, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heinz, and Dan Roth. 2021. Re- cent advances in natural language processing via large pre-trained language models: A survey, arXiv preprint vrXiv:211..012
Makoto Miwa and Mohit Bansal. 2016. End-to-end re- lation extraction using LSTMs on sequences and tree structures, In Proceedings of the 54th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1105-1116, Berlin, Germany. Association for Computational Linguistics.
Xiang Lisa Li and Percy Liang. 2021. Trefix-tuning Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 4582- 4597, Online. Association for Computational Lin- guistics.
Benjamin Newman, Kai-Siang Ang, Julia Gong, and John Hewitt. 2021. Refining targeted syntactic evalu- ation of language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3710-3723, Online. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019.
Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 16992.
Jingcheng Niu, Wenjie Lu, and Gerald Penn. 2022. Does BERT rediscover a classical NLP pipeline? In Proceedings of the 29th International Conference on Computational Linguistics. pages 3143-3153, Gyeongju, Republic of Korea. International Com- mittee Computational Linguistics.
Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. 2017. The conerete distribution: A continuous relax- ation of discrete random variables. In International Conference on Learning Representations.
Stephan Oepen, Omri Abend, Jan Hajic. Daniel Her- shcovich, Marco Kuhlmann, Tim O' O'Gomaan, Nian- wen Xue, Jayeol Chun, Milan Straka, and Zdenka Cross- framework mean- ing representation parsing. In Proceedings of the Shared Task on Cross-Framewod sentation Parsing al the 2019 Conference Natural Language Learning, pages 1-27, Hong Kong. Asso- ciation for Computational Linguistics.
Yuning Mao, Lambert Mathias, Rui Hou, Amjad Alma- hairi, Hao Ma, Jiawei Han, Scott Yih, and Madian Khabsa. 2022. UniPELT: A unified framework for parameter-efficiiet language model tuning. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 6253-6264, Dublin, Ireland. Associa- tion for Computational Linguistics.
Uresova. 2019. MRP 2019
Meaning Repre-
OpenAL. 2023. GPT-4 technical report. CORR. abs/2303.08774,
Diego Marcheggiani and Ivan Titov. 2017. Encoding sentences with graph convolutional networks for se- mantic role labeling. In Proceedings of the 2017 Conference on Empirical Methods Natural Lan- guage Processing, pages 1506-1515, Copenhagen. Denmark. Association for Computational Linguis- tics.
Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020. AdapterHub: A framework for adapting transformers. In Proceedings of the 2020 Conference Empirical Methods in Nat- ural Language Processing: System Demonstrations, pages 46-54, Online. Association for Computational Linguistics.
Rowan Hall Maudslay, Josef Valvoda. Tiago Pimentel, Adina Williams, and Ryan Cotterell. 2020. A tale of probe and a parser. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7389-7395, Online. Association for Computational Linguistics.
Jakob Prange, Nathan Schneider, and Lingpeng Kong. 2022. Linguistic frameworks go toe-to-toe at neuro- symbolic language modeling. In Proceedings of the

---

2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 4375-4391, Seat-
Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. 2019. Ordered neurons: Integrating tree structures into recurrent neural networks. In In- rernational Conference on Learning Representations.
tle. United States, Association for Computational Linguistics.
Richard Socher, Brody Huval, Christopher D. Manning, and Andrew Y. Ng. 2012. Semantic compositional- ity through recursive matrix-vector spaces. In Pro- ceedings of the 2012 Joint Conference on Empiri- cal Methods in Natural Language Processing and Computational Natural Language Learning. pages 1201-1211, Jeju Island, Korea, Association for Com- putational Linguistics.
Peng Qi. Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics: System Demonstrations, pages 101-108, Online. As- sociation for Computational Linguistics.
Peng Qian, Tahira Naseem, Roger Levy, and Ramón Fernandez. Astudillo. 2021. Structural guidance for transformer language models. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol- ume 1: Long Pupers), pages 3735-3745, Online. As- sociation for Computational Linguistics.
Richard Socher, Cliff Chiung- Yu Lin, Andrew Y. Ng, and Christopher D. Manning. 2011. Parsing natural scenes and natural language with recursive neural networks. In Proceedings of the 28th international conference on machine learning (ICML-11), page 129-136.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang. Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference Empiri- cal Methods in Natural Language Processing, pages 1631- Seattle, Washington, USA. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog. 188):
Michael Roth and Mirella Lapata. 2016. Neural seman- tic role labeling with dependency path embeddings. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pages 1192-1221 1202, Berlin, Germany. Association for Computational Linguistics.
Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. 2018. Linguistically- informed self-attention for semantic role labeling. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 5027-5038, Brussels, Belgium. Association for Com- putational Linguistics.
Andreas Rücklé Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer. Nils Reimers, and Iryna Gurevych. 2021. AdapterDrop: On the efficiency of adapters transformers. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 7930-7946, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Kai Sheng Tai, Richard Socher, and Christopher D. Man- ning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of the 53rd Anmual Meeting of the As- sociation for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1556- 1566, Beijing, China. Association for Computational Linguistics.
Devendra Sachan, Yuhao Zhang, Peng Qi, and William L. Hamilton. 2021. Do syntax trees help pre-trained transformers extract information? In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guistics: Main Volume, pages 2647-2661, Online. Association for Computational Linguistics.
Chong Min John Tan and Mehul Motani. 2020. Dropnet: Reducing neural network complexity via iterative pruning. In International Conference Machine Learning, pages 19366.966..6 PMLR.
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolu- tional networks. In The Semantic Web: 15th Inter- national Conference, ESWC 2018, Heraklion, Crete, Greece, June 3-7, 2018, Proceedings 15, pages 593- 607. Springer.
lan Tenney, Dipanjan Das, and Ellie Pavlick. 2019a. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4593- 4601, Florence, Italy. Association for Computational Linguistics.
Noam Shazeer, *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis. Quoe Le. Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural net- works: The sparsely-gates mixture-of-exxe -eeeeeetsssssssssseeeeee layer. In International Conference on Learning Representa- tions.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang. Adam Poliak, R Thomas McCoy, Najoung Kim, Ben- jamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. 2019b. What do you learn from con- text? probing for sentence structure in contextualized

---

word representations. In International Conference on Learning Representations.
Zhaofeng Wu, Hao Peng, and Noah A. Smith. 2021. In- fusing finetuning with semantic dependencies. Trans- actions of the Association for Computational Linguis- rics, 9:226-242.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint WxXV:23302.13711.
Changlong Yu, Tianyi Xiao, Lingpeng Kong, Yangqiu Song, and Wilfred Ng. 2022. An empirical revisiting of linguistic knowledge fusion in language under- standing tasks. In Proceedings of the 2022 Confer- ence on Empirical Methods in Natural Language Processing, pages 1064-10070, Abu Dhabi, United Arab Emirates, Association for Computational Lin- guistics.
Sean Trott, Tiaoo Timponi Torent, Nancy Chang, and Nathan Schneider. 2020. (Re)construing meaning in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5170-5184, Online. Association for Computational Linguistics.
Yuhao Zhang, Peng Qi, and Christopher D. Manning. 2018. Graph convolution over pruned dependency trees improves relation extraction. In Proceedings of the 2018 Conference on Empirical Methods in Nat- ural Language Processing, pages 2205-2215, Brus- sels, Belgium. Association for Computational Lin- guistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Zhuosheng Zhang, Yuwei Wu, Hai Zhao, Zuchao Li, Shuailiang Zhang, Xi Zhou, and Xiang Zhou. 2020. Semantics-aware bert for language understanding. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):9628 9635.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. GLUE: multi-task benchmark and analysis plat- form for natural language understanding. In Interna- tional Conference Learning Representations.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang. Xiaolei Wang. Yupeng Hou, Yingqian Min, Beichen Zhang, Junjic Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303. 182223.
Kai Wang, Weizhou Shen, Yunyi Yang. Xiaojun Quan, and Rui Wang. 2020. Relational graph attention net- work for aspect-based sentiment analysis. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics. pages 3229- 3238, Online. Association for Computational Lin- guistics.
Yichu Zhou and Vivek Srikumar. 2022. closer look at how fine-tume tne-tuning changes BERT. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pages 106611661, Dublin, Ireland. Association for Computational Linguistics.
Xing Wang, Zhaopeng Tu, Longyue Wang, and Shum- ing Shi. 2019b. Self-attention with structural position representations. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) pages 1403-1409, Hong Kong, China. Association for Computational Linguistics.
Yaushian Wang, Hung-Yi Yi Lee, and Yun-Nung Chen. 2019c. Tree transformer: Integrating tree structures into self-attention. In Proceedings of the 2019 Con- ference Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP- IJCNLP), pages 1061-1070, Hong Kong, China. As- sociation for Computational Linguistics.
Yuxuan Wang, Wanxiang Che, Jiang Guo, and Ting Liu. 2018. A neural transition-based approach for semantic dependency graph parsing. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1).
Wei Wu, Houfeng Wang, Tianyu Liu, and Shuming Ma. 2018 Phrase-level attentinn networks for universal sentence encoding. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3729-3738, Brussels, Belgium. Association for Computational Linguistics.


Reviews:
[{"review_id": "qEqE1BjqyT", "reviewer": "Reviewer_y8mp", "summary": "", "questions": "- I see that Table 2 is averaged over 3 runs. However, what is the variance on the runs? Are the results statistically significant?\n- Table 3 shows the number of parameters for different methods. However, it is unclear to me how fast the different methods are. What is the computational complexity for the methods?\n- How does varying the temperature of the Gumbel-Softmax affect results?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and feedback on our paper. We appreciate your insights and suggestions. We provide responses below to the raised concerns and questions.\n\n---\n\n__What Improvements From Which Part__\n\nWe acknowledge the importance of determining exactly what type of knowledge can benefit which part of the model, pinpointing such contributions requires exhaustive analysis by training a new model from scratch for each combination of knowledge type and layer. This is precisely the problem that we tackle. By sampling the contribution from a stochastic gate, our method provides an estimation of the improvement gain in an efficient way. In other words, the results from Table 2 show the marginal improvements the model gets from replacing the two MLP adapters with the selected linguistic experts (Figure 2). We will include the results for the MLP-experts-only configuration as a baseline in our final version.\n\n---\n\n__Impact of Gumbel-Softmax__\n\nRegarding the influence of Gumbel-Softmax, we observed on the development sets that employing the standard softmax invariably defaults to the MLP adaptor across layers due to its ease of learning, resulting in an architecture similar to the standard Adapter baseline. In the revised paper we will clarify this point; namely, that the Gumbel-Softmax is critical for exploring a diverse combination of options, rather than defaulting to the local minima.\n\n---\n\n__Variance and Statistical Significance__\n\nWe report the standard deviation of our method with the BERT encoder in the table below. \n\n| Dataset           | CoLA | RTE  | MRPC | STS-B | SST-2    | QNLI     | QQP      | MNLI     |\n|-------------------|------|------|------|-------|----------|----------|----------|----------|\n| Training Set Size | 1K   | 2.5K | 2.7K | 5.8K  | 67K      | 105k     | 363K     | 392k     |\n| Standard deviaton | 1.32 | 2.43 | 0.45 | 0.22  | **0.32** | **0.47** | **0.06** | **0.18** |\n|                   |      |      |      |       |          |          |          |          |\n\nOn the four high-resource datasets (SST-2, QNLI, QQP, and MNLI), the results from our model were significantly better than UniPELT (APL) with a confidence level of 99% according to the Bootstrap Significance test. We will update Table 2 with the standard deviation row and reorder the columns of such table by the training set size so that it will be clearer that our methods deliver significant gains for higher-resource datasets. All this will much better support the results’ discussion starting on line 440.\n\nAs a side note, the UniPELT results for running the significance tests are obtained based on the [`adapter-transformers`](https://github.com/adapter-hub/adapter-transformers) library ([Pfeiffer et al., 2020](https://aclanthology.org/2020.emnlp-demos.7/)), since the original paper by [Mao et al., 2022](https://aclanthology.org/2022.acl-long.433/) did not report the standard deviation of results across random seeds.\n\n---\n\n__Computational Complexity__\n\nThe graph convolution operation has a computational complexity $\\mathcal{O}(|E| \\cdot d_1 \\cdot d_2) $, where $d_1$ and $d_2$ are respectively the input and output dimension of the layer, and $|E|$ is the number of edges between nodes. In addition, the complexity of self-loop adds $\\mathcal{O}(|N| \\cdot d_1 \\cdot d_2)$, where $|N|$ is the number of nodes. In our implementation based on the [`dgl`](https://github.com/dmlc/dgl) library ([Wang et al., 2019](https://arxiv.org/abs/1909.01315)), inference on GPUs is 2 times slower than the standard linear layer, which has the same complexity as self-loop. We will emphasize these points in our final draft.\n\n---\n\n__Effects on Varying Temperature__\n\nSince we employ temperature annealing, the start and end values of the temperature are selected based on the recommendation from the survey by [Huijben et. al, 2021](https://arxiv.org/abs/2110.01515), with the number of steps determined empirically based on changes in gate values (Sec 6.2). Nevertheless, we agree that the start/end value of the temperature hyperparameter and the decaying methods (e.g., linear, exponential) may have an effect on model behavior. Therefore, we are running some experiments to establish how sensitive the models are to these changes. Results should be available for inclusion in the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Gains over the majority of the baselines on a variety of tasks", "reasons_to_reject": "Despite some of the analysis, it is still a bit hard to discern what improvements the model is getting from what parts. For certain (but closely related) architectures, different linguistic structures appear to matter more. Likewise, how much does the Gumbel-Softmax impact this?", "questions_to_authors": "- I see that Table 2 is averaged over 3 runs. However, what is the variance on the runs? Are the results statistically significant?\n- Table 3 shows the number of parameters for different methods. However, it is unclear to me how fast the different methods are. What is the computational complexity for the methods?\n- How does varying the temperature of the Gumbel-Softmax affect results?", "paper_topic_and_main_contributions": "This paper combines injecting linguistic structure with Parameter Efficient Fine-Tuning (PEFT). The method relies on multiple different parts of the broader literature, such as Mixture-of-Experts (MoE) and the Gumball-Softmax trick. Overall, the method gets better results (in general) on a variety of tasks. Three linguistic graphs and non-linear transformations are applied to encode sentence-level structures. From Figure 2, it appears that the MLP is useful in most lower layers of the network, with various graph structures in the upper layers.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "014 “Important” … 105 “Importance” I think it is “importance”\n\nSection 4 is important, but brief. Could be made longer as it is central to the paper."}, "extra_text": {}}, {"review_id": "8OQxIcLXIt", "reviewer": "Reviewer_bbmx", "summary": "", "questions": "No questions, great work!\n", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thorough examination and the positive feedback! We appreciate your acknowledgment regarding the impact of our work, we hope our findings will motivate future studies on integrating various types of knowledge in the parameter-efficient setting.\n\n---\n\n__Lower-Resource Languages__\n\nWe also believe that as a next step, it is important to evaluate our method on other lower-resource languages (e.g., Tagalog, Bengali). One follow-up work we intend to do is to examine the degree to which the quality of the parses affects the model performance. This might provide some initial evidence of our method’s performance on low-resource languages with lower-quality parsers (e.g., [Şahin et al., 2018](https://aclanthology.org/D18-1545/))", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper innovatively combines two methods to adapt pretrained language models successfully, leading to an average improvement on results across tasks.\n\nThe injection of linguistic knowledge to along with adapters is important to share with the wider community.\n\nThe paper is well written, clear and easy to follow. \n", "reasons_to_reject": "The methods have only been tested on English. Despite the lack of availability of parses for lower-resource languages, it would be nice to see if this method works just as well for other languages.", "questions_to_authors": "No questions, great work!\n", "paper_topic_and_main_contributions": "This paper proposes combining parameter-efficient finetuning with injecting linguistic structures into large language models to create “Mixture-of-Linguistic Expert Adapters”. The adapters encoding different linguistic structures are inserted parallel to the feed-forward network in a transformer layer, with a Gumbel-softmax gate weighing the importance of each module. After the training, the model is pruned by removing all but one expert to reduce the parameter count. The resulting model is then analyzed to gain insight into the gate convergence rate and discover which expert is chosen at which layer. ", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dRJvts05EM", "reviewer": "Reviewer_otyM", "summary": "", "questions": "Please refer to the comments in the previous section.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and constructive criticisms. We would like to take this opportunity to address your concerns and questions.\n\n---\n\n__1. Experiment Results and Randomness__\n\nThe reviewer brings up a valid concern. However, we argue that the 0.5-2% improvements in the high-resource datasets (SST-2, QNLI, QQP, MNLI) are noteworthy, especially when compared with the smaller improvements obtained in prior work on integrating linguistic knowledge (e.g., [Bai et al., 2021](https://aclanthology.org/2021.eacl-main.262/), [Wu et al, 2021](https://aclanthology.org/2021.tacl-1.14/)). \n\nMore importantly, we agree with the reviewer that the standard deviation and statistical significance of these results should be reported; Table 2 will be revised to include those. More specifically, on the four high-resource datasets (SST-2, QNLI, QQP, and MNLI), the results from our model were significantly better than UniPELT (APL) with a confidence level of 99% according to the Bootstrap Significance test. We will update Table 2 with the standard deviation row (see below) and reorder the columns of such table by the training set size so that it will be clearer that our methods deliver significant gains for higher-resource datasets. All of this will much better support the results’ discussion starting on line 440.\n\n| Dataset           | CoLA | RTE  | MRPC | STS-B | SST-2    | QNLI     | QQP      | MNLI     |\n|-------------------|------|------|------|-------|----------|----------|----------|----------|\n| Training Set Size | 1K   | 2.5K | 2.7K | 5.8K  | 67K      | 105k     | 363K     | 392k     |\n| Standard deviaton | 1.32 | 2.43 | 0.45 | 0.22  | **0.32** | **0.47** | **0.06** | **0.18** |\n|                   |      |      |      |       |          |          |          |          |\n\nAs a side note, the UniPELT results for running the significance tests are obtained based on the [`adapter-transformers`](https://github.com/adapter-hub/adapter-transformers) library ([Pfeiffer et al., 2020](https://aclanthology.org/2020.emnlp-demos.7/)), since the original paper by [Mao et al., 2022](https://aclanthology.org/2022.acl-long.433/) did not report the standard deviation of results across random seeds.\n\n---\n\n__2. Results for Full Fine-tuning__\n\nWe thank the reviewer for suggesting to compare our proposed method with the results of fine-tuning the full PLM. This also led us to think about considering performing the standard fine-tuning with the full set of experts at each layer (without Gumbel-Softmax) to understand the benefits of linguistic experts in the standard fine-tuning setting. Even though we recognize that these experiments are computationally rather demanding, we will try to include corresponding results in the final version.\n\n---\n\n__3. Ablation Studies__\n\nSince there are a total of $4^{12}$ combinations for a 12-layer model (four choices of experts at each layer), exhaustive ablation experiments across all combinations would be computationally impractical, which is precisely the motivation of our selection-based pruning approach.  However, we agree that using a naive approach to selecting experts (e.g., only one expert type per model) could be useful for establishing a baseline and we are currently testing such an approach and will report the results in the final version. \n\n---\n\nWe will take the reviewer's suggestion and revise the conclusion section to make it more concise and distinct from the abstract.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea of incorporating linguistic knowledge into PLMs using adapter structure is well-motivated and novel.\n2. By involving an Edgeless Graph expert, the authors also study whether linguistic knowledge is necessary to PLMs. This is interesting as previous works usually assume linguistic knowledge is beneficial to PLMs - such assumption, however, is not verified. The analysis of expert selection partly shows that additional linguistic knowledge is only useful to higher layers in the PLM architecture.", "reasons_to_reject": "1. The experiment results are not strong enough to support the effectiveness of the proposed method. Though the authors argue that the proposed method perform considerably better in the four tasks with the highest number of training examples, I don't think improvement less than 1% can be viewed as \"considerably better\". For this claim, I think the authors should at least report the standard deviation in Table 2 to exclude the influence of randomness.\n2. About the experiment setup, I think the authors should compare the results of the proposed method with the results of fine-tuning the full PLM. Though I'm aware that the method falls in the PEFT setting, I think the paper should compare the full fine-tuning to see whether the additional linguistic knowledge is indeed beneficial to the PLM.\n3. The ablation study is missing. The paper should do ablation study to the involved linguistic information sources and the proposed expert pruning component.", "questions_to_authors": "Please refer to the comments in the previous section.", "paper_topic_and_main_contributions": "This paper proposes a method that combines linguistic structure injection and parameter-efficient fine-tuning (PEFT) of pre-trained language models. The method leverage a group of parallel adapter modules to encode different linguistic structures via Mixture-of-Linguistic-Experts architecture. To cut off the parameter budget, the method prunes the experts based on their important scores. Results on BERT, RoBERTa, DeBERTaV3 show that the proposed method outperforms baselines on high-resourced datasets. Besides, the paper analyzes the expert selection at each layer to provide some insights.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "No.", "Typos_Grammar_Style_and_Presentation_Improvements": "1. The abstract and conclusion have some sentences that are nearly the same. I would suggest making the conclusion section more concise."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agree that the method proposed in the paper is interesting and novel. They find the paper well written and easy to follow. Two of the reviewers seem to be concerned by the lack of ablation tests/clear indication of what information has an impact on the results. The authors' answer indicates that they can provide at least a preliminary analysis of this, should the paper be accepted. The reviewers ask a few other questions related to methodology/technical details, which are successfully answered by the authors. 

The scores of one of the reviewers seem to be rather low given the wording of their review. The questions and objections of the reviewer are addressed in the authors' rebuttal. The reviewer seems to be reasonably satisfied with the answer, so I assume the reviewer forgot to increase the scores. 

Overall, the reviews (and partially the scores) indicate that this is a sound paper which contains some novel ideas. The discussion during the rebuttal period will enable the authors to address some of the problems identified in the original version.
Decision: Accept-Findings