Knowledgeable Parameter Efficient Tuning Network for Commonsense Question Answering
Year: 2023

Abstract:
Commonsense question answering is important for making decisions about everyday matters. Although existing commonsense question answering works based on fully fine-tuned PLMs have achieved promising results, they suffer from prohibitive computation costs as well as poor interpretability. Some works improve the PLMs by incorporating knowledge to provide certain evidence, via elaborately designed GNN modules which require expertise. In this paper, we propose a simple knowledgeable parameter efficient tuning network to couple PLMs with external knowledge for commonsense question answering. Specifically, we design a trainable parameter-sharing adapter attached to a parameter-freezing PLM to incorporate knowledge at a small cost. The adapter is equipped with both entityand query-related knowledge via two auxiliary knowledge-related tasks (i.e., span masking and relation discrimination). To make the adapter focus on the relevant knowledge, we design gating and attention mechanisms to respectively filter and fuse the query information from the PLM. Extensive experiments on two benchmark datasets show that KPE is parameter-efficient and can effectively incorporate knowledge for improving commonsense question answering.

heading: 1 Introduction
text: Commonsense question answering is the process of combining observations and the basic knowledge that reflects our natural understanding of the world and human behaviors, to make presumptions about ordinary situations in our daily life (JohnsonLaird, 1980). It has emerged as an important task in natural language understanding.
Pre-trained language models (PLMs), which revolutionize many areas with superior performance, have been applied for the commonsense question answering task based on full fine-tuning
∗Corresponding author
as shown in Figure 1(a). For example, Lourie et al. (2021) fully fine-tuned the PLM Unicorn and achieved competitive performance on 8 commonsense benchmarks. However, they inevitably incur prohibitive computation costs as the scale of parameters increases, and are lacking in transparency and interpretability (Houlsby et al., 2019a; Lin et al., 2019).
Furthermore, some works couple the PLMs with knowledge to improve the interpretability of the reasoning process. As shown in Figure 1(b), they typically extract the relevant knowledge subgraphs about entities in the query and then elaborately design a graph neural network (GNN) module to perform reasoning (Lin et al., 2019; Feng et al., 2020; Yasunaga et al., 2021; Sun et al., 2022). Despite the fact that they provide certain evidence for the reasoning process, it requires expertise to design effective GNN modules. Additionally, they generally consider only the structured triple knowledge about the entities in the query, while ignoring the textual knowledge about the query itself.
In this work, we propose a simple Knowledgeable Parameter Efficient model (KPE) for commonsense question answering. In particular, we design a parameter-sharing adapter
9051
plugin for incorporating knowledge into the frozen PLM as shown in Figure 1, which largely reduces the scale of trainable parameters. Our adapter plugin integrates both the entity- and query-related knowledge (uniformly grounding to the unstructured commonsense knowledge base GenericsKB) through two auxiliary knowledgerelated tasks (i.e., span masking and relation discrimination). Additionally, to make the adapter focus on the relevant knowledge for commonsense question answering, we design gating and attention mechanisms to respectively filter and fuse the query information from the PLM. Overall, our main contributions can be summarized as follows:
• To the best of our knowledge, we are the first to propose a knowledgeable parameter efficient tuning network for commonsense question answering, which adopts a new parametersharing adapter to incorporate knowledge.
• Our designed adapter integrates both the entity- and query-related knowledge with two auxiliary knowledge-related tasks. Additionally, the gating and attention mechanisms are respectively employed to filter and fuse the query information from the PLM to make the adapter focus on relevant knowledge for commonsense question answering.
• Extensive experiments on two benchmark datasets have demonstrated that our proposed KPE can effectively incorporate knowledge for improving commonsense question answering, with a tiny computation cost.

heading: 2 Related Work
text: In this section, we review the related works on commonsense question answering and parameter efficient tuning.

heading: 2.1 Commonsense Question Answering
text: With the remarkable success of PLMs on various tasks (Liu et al., 2019; Raffel et al., 2020), some researchers propose to fully fine-tune PLMs on the commonsense question answering task. For example, Lourie et al. (2021) fully fine-tuned the PLM Unicorn on 8 commonsense benchmarks respectively, and achieved promising results. Khashabi et al. (2020) built a universal PLM for the question answering task and fully fine-tuned it on 10 factoid and commonsense QA datasets. Despite
the prevalence of PLMs, fine-tuning all the parameters brings prohibitive computation costs as the scale of PLM parameters grows. Moreover, due to the lack of modules explicitly modeling knowledge, the PLMs suffer from poor transparency and interpretability. In light of this, some methods improve the PLMs with well-designed GNN modules to integrate relevant knowledge from knowledge graphs (Feng et al., 2020; Sun et al., 2022; Wang et al., 2022). For example, MHGRN (Feng et al., 2020) combines PLMs with a graph relation network to perform multi-hop reasoning on knowledge subgraphs and provides certain evidence for the reasoning process. GreaseLM (Zhang et al., 2022) and JointLK (Sun et al., 2022) introduce GNN-based modules to perform joint reasoning over both the text and knowledge subgraphs for commonsense question answering. Nevertheless, they require expertise to design an effective GNN module for encoding the knowledge subgraph. Additionally, they only consider the entity-related structured knowledge, ignoring the query-related knowledge which could be in the form of text.
Differently, in this work, we present a simple knowledgeable parameter efficient tuning network which utilizes a parameter-sharing adapter to incorporate both entity- and query-related knowledge for improving commonsense question answering.

heading: 2.2 Parameter Efficient Tuning
text: Since fine-tuning all the parameters of PLMs causes prohibitively expensive costs, researchers propose to fine-tune a small part of the model parameters while freezing the rest. Adapter-tuning, firstly proposed by Houlsby et al. (2019a), is a prevalent parameter efficient tuning method which inserts trainable adapter modules between the layers of frozen PLMs to bootstrap PLMs (Mahabadi et al., 2021; Pfeiffer et al., 2021). Wang et al. (2021) adopted adapters to infuse knowledge into the large pre-trained language model. Inspired by the prompting methods, some researchers also exploit the prefix-tuning (Li and Liang, 2021) and prompt-tuning (Lester et al., 2021; Liu et al., 2022b). They preset a sequence of trainable prompt tokens to the input or intermediate layers and only update these tokens during training. Additionally, some works explore the low-rank adaptation method which injects and optimizes the low-rank matrices of attention weight in the frozen PLMs for parameter efficient tuning (Hu et al., 2022; Ma-
IsA
DerivedFrom
MadeOfIsA
HasARelatedTo
IsA
PartOf
candy
sugarfood
plant
cotton candy cotton
[cotton, IsA, plant]
[cotton candy, DerivedFrom, cotton]
[cotton candy, IsA, food] [cotton candy, MadeOf, sugar]
[cotton candy, IsA, candy]
[candy, HasA, sugar]
[sugar, RelatedTo, food]
[plant, PartOf, food]
Knowledgeable Adapter
(Trainable)
预训练 语言模型
(参数冻结)
参数共享
采样&掩码处理
Cotton is used for [MASK] [MASK].
分数预测 跨度掩码 关系判别 UsedFor
RelatedTo HasA
知识感知的适配器 (可训练)
实体相关的知识 问题相关的知识 Cotton candy is sometimes made out of cotton. yes
making (0.75) cooking (0.11) physical (0.04) …… clothing (0.46) food (0.29) fitness (0.15) ……
知识抽取
门控 单元
habadi et al., 2021).
In this work, we focus on the commonsense question answering task and propose a knowledgeable parameter efficient tuning network that effectively couples PLMs with external knowledge.

heading: 3 KPE Model
text: Following previous works (Feng et al., 2020; Yasunaga et al., 2021), we focus on the commonsense question answering task in the form of multiplechoice question answering. Formally, given a natural language query q and a se of candidate answers A = {a}, we will measure the plausibility score ρ(q, a) for each answer and choose the most plausible one a∗. To promote the commonsense question answering process, we resort to external knowledge bases to extract both entity- and query-related knowledge pieces K = {k} based on q and A.
As shown in Figure 2, our KPE couples the PLM with external knowledge via a parameter-sharing knowledgeable adapter attached to the frozen PLM. The PLM takes (q, a) as input and outputs the plausibility score ρ(q, a). The knowledgeable adapter aims to integrate the knowledge pieces k. In the following, we first introduce the knowledge extraction process. Then we describe the knowledgeable adapter that effectively integrates the extracted knowledge based on two auxiliary tasks (i.e., span masking and relation discrimination tasks), as well as gating and attention mechanisms for information interaction with the PLM.

heading: 3.1 Knowledge Extraction
text: A traditional source of commonsense knowledge is triple-based knowledge graphs such as ConceptNet (Speer et al., 2017). However, they encode limited types of the knowledge. Here, we use a corpus of generic sentences about commonsense facts, i.e., GenericsKB (Bhakthavatsalam et al., 2020) as the final knowledge source. The text can represent more complex commonsense knowledge, involving facts that relate three or more concepts. Next, we introduce how to extract entity- and query-related k owl ge from GenericsKB.
Entity-related Knowledge. For entity-related knowledge, we first recognize all the entities in the query and candidate answers, and ground them to triples in ConceptNet. Then, we serialize the triples to sentences and use them as the keys to retrieve knowledge pieces from GenericsKB.
Triple Grounding. Given the query q and candidate answers A = {a}, we first extract the entities e from them. Then, we ground all the triples in ConceptNet originating from e to obtain the triple set T = {h, r, t}. To condense and filter the extracted triples, we follow Xu et al. (2022) to score each triple:
pi = wi ∗ N
Nri , (1)
where pi denotes the score of the i-th triple (hi, ri, ti), wi is the triple weight provided by ConceptNet, N is the size of T and Nri is the number of triples with relation ri in T . If pi is higher than the predefined score threshold p∗, the triple
(hi, ri, ti) will be added to the selected triple set T ∗ ⊆ T .
Knowledge Retrieval. Now, we convert these triples in T ∗ into a series of sentences for knowledge retrieval from the unstructured commonsense knowledge base GenericsKB. Specifically, for each triple (hi, ri, ti), we employ a set of pre-defined relation templates (Ma et al., 2021) to generate a sentence si at first. For example, the triple (sweltering, RelatedTo, hot) can be serialized to the sentence "sweltering is related to hot". Then, we take si as a key to retrieve the related knowledge pieces (in the form of sentences) from GenericsKB. The knowledge pieces without the entity pair (hi, ti) are directly disregarded. Afterwards, we select the knowledge piece which is most relevant to the query to enhance the commonsense question answering. Particularly, we use the pre-trained SimCSE (Gao et al., 2021) to obtain sentence embeddings, based on which, we compute the cosine similarity between each retrieved knowledge piece ki and the query q as the knowledge relevance score. Finally, after processing all the triples, we choose top K (K = 5 in this work) retrieved knowledge pieces as entity-related knowledge KE = {kEi } according to the computed knowledge relevance scores.
Query-related Knowledge. Considering the rich semantic information contained in the query q, we also explore the query-related knowledge for improving the commonsense question answering task. Specifically, similar to the entity-related knowledge retrieval, we retrieve query-relevant knowledge pieces from GenericsKB by concatenating the query with all the candidate answers as the key for retrieval. We also compute the knowledge relevance scores and choose top K knowledge pieces with the highest scores as the query-related knowledge KQ = {kQi }.

heading: 3.2 Knowledgeable Adapter
text: In this subsection, we detail our knowledgeable adapter that effectively incorporates the above extracted knowledge for commonsense question answering.
Knowledgeable Adapter Layer. For parameterefficiency, we connect each PLM layer with a parameter-sharing adapter layer as shown in Figure 3. For the l-th (l ∈ [1, L]) adapter layer, the input H lA ∈ R(m+n)×d is formed by vertically concatenating the output features H̃ l−1A ∈ Rn×d of
Self-Attention
Add & Norm
Feed Forward
Add & Norm
Down
Up
!-th PLM-layer !-th Adapter-layer
"
concatenate
Self-Attention
#!"
#!#
#!$%&
#!' #!&
the (l-1)-th adapter layer and the output features H̃ lP ∈ Rm×d of the l-th PLM layer, where m and n respectively denote the length of PLM input sequence and knowledge piece, and d is the hidden size. Note that, a learnable gating function is applied to filter the PLM output features H̃ lP to obtain crucial information of the query. Formally,
H lA = [H̃ l P ⊙ σ(G); H̃ l−1A ], (2)
where G ∈ Rm×d is a trainable matrix and is learned in the training process, ⊙ denotes the element-wise multiplication.
Now, given the input H lA, the adapter layer first projects it down to r dimension with a linear projection layer. Then we apply a self-attention layer to better fuse the knowledge and the query information from the PLM. After that, another linear projection layer is applied to project it up to the original dimension d. Finally, we split the output features H ′ l A ∈ R(m+n)×d of the up projection layer into two parts: H̃ lR ∈ Rm×d for the residual connection layer of the PLM and H̃ lA ∈ Rn×d for the next adapter layer. To enhance the knowledge modeling ability of the adapter, we also design the following two knowledge-related tasks, which take the final output H̃LA of the adapter as input.
Span Masking Task. Mask prediction task can help promote the knowledge memorization of the adapter (Sun et al., 2021). For the entityrelated knowledge kEi corresponding to the triple (hi, ri, ti), we mask out the corresponding tokens of the tail entity mention and replace them with the same number of [MASK] to yield the corrupted sequence. Then, we fed the corrupted sequence into the adapter for forward reasoning. Based on the final adapter output H̃LA , we predict the masked tokens and calculate cross-entropy loss LMLM over them. For the query-related knowledge piece kQi ,
we mask 15% tokens in total at the span level and predict them in the same way as SpanBERT (Joshi et al., 2020).
Relation Discrimination Task. Relation discrimination task can facilitate the adapter to understand the intrinsic relational facts in text and improve the robustness of the learned representations through contrastive learning (Chen et al., 2022). This task applies only to entity-related knowledge pieces that include entity pairs. Given the entityrelated knowledge piece kEi and its corresponding triple (hi, ri, ti), we conduct mean pooling operation over the token embeddings (from the adapter output H̃LA ) of the entity mentions to obtain entity representations vHi and v T i . Then, we follow Qin et al. (2021) to concatenate vHi and v T i as the relation representation vRi = [v H i ,v T i ]. For improving the understanding of relational facts, we treat the relation ri as its positive sample and the rest relations as negative samples. Finally, we adopt the InfoNCE (van den Oord et al., 2018) loss to make the positive pair closer and push away the negative pairs:
LRD = − log exp (vRi · f(ri)/τ)∑|E| j=1 exp (v R i · f(rj)/τ) , (3) where τ is a temperature hyper-parameter, |E| is the number of relations ri in ConceptNet, and f(ri) denotes the lookup operation for the token id of the relation ri based on the PLM. If there are multiple tokens in ri, we will apply mean pooling.

heading: 3.3 Model Training
text: Given the query context q and a candidate choice a ∈ A, we leverage the output H̃LP of the final PLM layer to compute the plausibility score ρ(q, a)=MLP (H̃LP ) and maximize the plausibility score of the correct answer a∗ via a cross-entropy loss:
LQA = Eq,a∗,A [ − log exp (ρ(q, a
∗))∑ a∈A exp (ρ(q, a))
] .
(4) Overall, the whole training objective of KPE is
formulated as follows:
L = LQA + LMLM + LRD. (5)
During training, we will randomly sample one piece of knowledge from the entity- and queryrelated knowledge (KE and KQ) at each step. Note that for the query-related knowledge piece which
is not applicable to the relation discrimination task, we will ignore the corresponding loss LRD .

heading: 4 Experiments
text: In this section, we evaluate the effectiveness of our proposed KPE.

heading: 4.1 Datasets
text: We evaluate KPE on two benchmark datasets: OpenbookQA (Mihaylov et al., 2018) and CommonsenseQA 2.0 (Talmor et al., 2021).
OpenbookQA is a question answering dataset about elementary scientific knowledge and each question has four different options. This dataset contains 5, 957 questions in total and we utilize the official data splits from Mihaylov et al. (2018).
CommonsenseQA 2.0 (CSQA2) is a binary classification dataset including 14, 343 questions. Note that the test set of CSQA2 is not public, and we need to submit the model predictions to the official leaderboard to get the evaluation results.

heading: 4.2 Implementation Details
text: For knowledge retrieval, we first store GenericsKB via Elasticsearch1 and use the retrieval function of Elasticsearch based on BM25 for retrieval. We choose the parameter values that achieve the best results on the development set. Experimentally, we set the temperature hyper-parameter τ in the relation discrimination task to 0.1 and set the score threshold p∗ in triple grounding to 3.5. The down size in the adapter r is set to 256. Following previous works, the hidden dimension of the model d=1024, and the number of layers L=24. We use AdamW (Loshchilov and Hutter, 2018) optimizer in our experiments. For model training, we set the batch size to 32 and the learning rate to 2e-5. We implement the parameter efficient tuning baselines for commonsense question answering based on AdapterHub (Pfeiffer et al., 2020).

heading: 4.3 Baselines
text: We compare our proposed KPE with the following parameter efficient tuning based methods and existing strong commonsense question answering methods.
Parameter Efficient Tuning based Methods. We compare KPE with the following parameter efficient tuning based methods.
1https://github.com/elastic/elasticsearch
• Bottleneck Adapter (Houlsby et al., 2019b) is the first method to perform the adapterbased tuning in NLP.
• Prefix Tuning (Li and Liang, 2021) inserts a sequence of learnable prompts into the input or intermediate layers to decrease training costs.
• LoRA (Hu et al., 2022) presets trainable rank decomposition matrices in each layer of PLM for less trainable parameters.
• MAM Adapter (He et al., 2022) builds an effective adapter module that combines the advantages of adapter, prefix tuning and lowrank methods.
• Compacter (Mahabadi et al., 2021) is built on top of ideas from adapters, low-rank optimization, and parameterized hypercomplex multiplication layers, achieving a better tradeoff between task performance and the number of trainable parameters.
For fair comparison, we improve these baseline methods with our extracted knowledge by concatenating the extracted knowledge with the original inputs of these baselines.
Existing Commonsense Question Answering Methods. We also compare KPE with the existing strong commonsense question answering methods. For OpenbookQA dataset, we compare our model with the following baselines that enhance PLMs with knowledge via GNN modules: (1) RN (Santoro et al., 2017), (2) RGCN (Schlichtkrull et al., 2018), (3) GconAttn (Wang et al., 2019), (4) MHGRN (Feng et al., 2020), (5) QA-GNN (Yasunaga et al., 2021), (6) GSC (Wang et al., 2022), (7) JointLK (Sun et al., 2022). For fair comparison, we use the same PLM (i.e., RoBERTa-large (Liu et al., 2019)) in all the above baselines and our KPE
on OpenbookQA. For CSQA2 dataset, we employ the vanilla Unicorn-11B (Lourie et al., 2021) as the PLM model for KPE, and compare KPE with the following fully fine-tuned model from the official leaderboard2: (1) T5-large (Raffel et al., 2020), (2) Unicorn-large (Lourie et al., 2021), (3) T511B (Raffel et al., 2020), (4) Unicorn-11B (Lourie et al., 2021), (5) GKP+Unicorn-11B-ft (Liu et al., 2022a). Among these baselines, GKP+Unicorn11B-ft performs best. It handcrafts demonstration examples to guide GPT3 (Brown et al., 2020) to generate knowledge and integrates the knowledge via prompting for commonsense question answering.

heading: 4.4 Results and Analysis
text: Table 1 reports the results of our proposed KPE in comparison with the prevalent parameter efficient tuning based methods on both OpenbookQA and CSQA2 datasets. Note that, for a fair comparison, we concatenate our extracted commonsense knowledge with the original inputs of these baselines. Since the annotation of the CSQA2 test set is not released, we only report the comparison results on the dev set. From table 1, we can observe that: (1) KPE consistently outperforms all the baselines on both datasets. Compared to the best baseline method, KPE achieves around 12.5% and 1.3% improvements on CSQA2 dev set and OpenbookQA test set, respectively. We believe that KPE benefits from the designed knowledgeable adapter which is parameter-efficient and effectively incorporates the commonsense knowledge. Moreover, KPE achieves a much larger improvement on CSQA2 dataset (+12.5%) than OpenbookQA dataset (+1.3%). The reason could be that
2https://leaderboard.allenai.org/csqa2/submissions/public
the CSQA2 dataset is much more difficult, in which the knowledge is more needed. Thus, KPE achieves a greater improvement on CSQA2 dataset by effectively incorporating the external knowledge. (2) Compared to Bottleneck Adapter, Prefix Tuning and MAM Adapter, KPE introduces fewer parameters while achieving conspicuous improvements on both datasets. The reason is that our knowledgeable adapter employs an efficient parameter-sharing strategy and better integrates the knowledge via two auxiliary knowledge-related tasks. The gating and attention mechanisms also help the adapter to focus on useful knowledge for improving commonsense question answering. (3) The baseline methods Compacter and LoRA, although introducing fewer parameters, achieve much lower performance than KPE. Our method achieves a better trade-off between the number of trainable parameters and task performance.
Table 2 and 3 show the results of our model in comparison with the existing strong commonsense question answering methods on OpenbookQA dataset and CSQA2 dataset, respectively. As we can see from Table 2, our KPE outperforms all the GNN based methods and achieves the best performance. It demonstrates the effectiveness of our KPE with the knowledgeable adapter for incorporating knowledge to improve commonsense question answering. We believe that KPE could further benefit from the advancement of large language models and is of much value to the parameter efficient tuning research.
From Table 3, we can observe that our model KPE based on the PLM Unicorn-11B achieves comparable performance to the best fully fine-tuned models Unicorn-11B and GKP+Unicorn-11B-ft, through updating a much smaller amount of parameters (around 0.019% compared to their parameter
scale).

heading: 4.5 Ablation Study
text: To verify the importance of each module in our KPE, we compared it with the following variants: (1) KPE-w/o-E: A variant of KPE that removes the entity-related knowledge. (2) KPE-w/o-Q: A variant of KPE that removes the query-related knowledge. (3) KPE-w/o-E&Q: A variant of KPE that removes both entity- and query-related knowledge. Accordingly, the span masking and relation discrimination tasks are removed. (4) KPE-w/o-S: A variant of KPE that removes the span masking task. (5) KPE-w/o-R: A variant of KPE that removes the relation discrimination task. (6) KPEw/o-S&R: A variant of KPE that removes both span masking and relation discrimination tasks. (7) KPE-w/o-A: A variant of KPE that replaces the self-attention mechanism in the knowledgeable adapter with a conventional nonlinearity function. (8) KPE-w/o-G: A variant of KPE that replaces the learnable gating function in the knowledgeable adapter with direct concatenation.
Table 4 shows the results of the ablation study. We can obtain the following observations: (1) On both datasets, removing query-related knowledge results in a larger performance drop than removing entity-related knowledge. It demonstrates the importance of the query-related knowledge for com-
monsense question answering. When removing both entity- and query-related knowledge, the performance largely decreases (-2.8% and -5.05% on OpenbookQA and CSQA2, respectively). (2) Disabling any auxiliary knowledge-related task will result in performance degradation, which shows that both tasks enable the adapter to better capture the knowledge, thus improving the commonsense question answering. (3) KPE consistently outperforms KPE-w/o-G and KPE-w/o-A on both datasets, which verifies that both the gating and self-attention mechanisms promote the knowledge integration for improving commonsense question answering.

heading: 4.6 Impact of Down Size r in Adapter
text: To explore the impact of the down size r on model performance, we vary r from 16 to 1024, and report the results on two datasets in Figure 4. We can observe that the accuracy on both datasets generally first grows and reaches the highest value from 16 to 256, while it begins to drop when r is larger than 256. Overall, KPE achieves the best performance at
r=256 on both OpenbookQA and CSQA2 datasets.

heading: 4.7 Case Study
text: In order to intuitively understand how the external knowledge in KPE helps improve the commonsense question answering, we compare KPE with the variant KPE-w/o-E&Q. We visualize the predicted score distributions over the candidate choices using two examples from OpenbookQA and CSQA2 datasets. As can be seen from Figure 5(a), given the query “Desert environments are generally _”, KPE makes the right choice “sweltering” while KPE-w/o-E&Q assigns a higher score to the incorrect choice “arctic like”. We believe that the extracted knowledge (e.g., “some plants grow in the hot, dry desert”, “sweltering is related to hot.”) facilitates the commonsense question answering. In addition, we can observe from Figure 5(b) that although both KPE and KPE-w/o-E&Q correctly predict the answer, KPE is more confident with the prediction results by benefiting from the extracted knowledge.

heading: 5 Conclusion
text: In this work, we present a knowledgeable parameter efficient tuning network KPE to effectively incorporate both entity- and query-related knowledge for improving commonsense question answering. Particularly, we design a parameter-sharing knowledgeable adapter as the plugin attached to the frozen PLM to incorporate knowledge. Two auxiliary knowledge-related tasks are specifically designed for the adapter to better model and capture the knowledge. Moreover, to make the adapter integrate relevant knowledge, we introduce gating and attention mechanisms to respectively filter and fuse the query information from the PLM. Experiments on two benchmark datasets have demonstrated the effectiveness and parameter-efficiency of KPE for commonsense question answering. In future work, we will explore to integrate other parameter-efficient tuning tricks in KPE.
Limitations
The performance of KPE is also related to the used pre-trained language model (PLM), in addition to the proposed framework. KPE could suffer from unsatisfactory performance when the base PLM is not strong enough. Applying our proposed KPE to stronger PLMs, such as DeBERTa, may lead to further improvements.

heading: Acknowledgement
text: This work was supported by the National Key R&D Program of China (2020AAA0105200), the National Science Foundation of China (NSFC No. U19B2020, No. 62276029, No. 62106249), Beijing Academy of Artificial Intelligence (BAAI) and CCF-Zhipu.AI Large Model Fund (No. 202217).

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section Limitations
7 A2. Did you discuss any potential risks of your work? There are no potential risks in our paper.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Section 4.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 4.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In future work, we will explore to integrate other parameter-efficient tuning tricks in KPE. Limitations
The performance of KPE is also related to the used pre-trained language model (PLM), in addition to the proposed framework. KPE could suffer from unsatisfactory performance when the base PLM is not strong enough. Applying our proposed KPE to stronger PLMs, such as DeBERTa, may lead to further improvements