SlowGuess/NAACL_2022_OCR huggingface dataset:
As shown in the table, our models outperform all corresponding baselines. KAM-BERT-Base achieves an average GLUE score of 78.7, lifting 1.2 scores from standard BERT-Base model with only a few extra parameters introduced to the baseline model. Particularly, the improvements on CoLA datasets are fairly large, showing that our knowledge integration method has good generalization performance for natural language inference and understanding. ERNIE have also added external information such as entity and knowledge graph, but it needs much more time for a joint re-training. As for BERT-Large and its counterpart KAM-BERT-Large, the average improvement on GLUE benchmark is 0.9. We can see that the improvement becomes smaller when the model grows larger, because larger models often capture more semantic knowledge in the pre-training phrase. But incorporating explicit knowledge is still indispensable for achieving a superior performance.
4.3 Question Answering
We conduct experiments on two kinds of question answering tasks, i.e., commonsense QA and open-domain QA. Commonsense QA aims to answer questions with commonsense. We adopt CosmosQA for evaluation, which requires commonsense-based reading comprehension formulated as multiple answer selection. Open-domain QA aims to answer questions using external resources such as collections of documents and webpages. We consider two public datasets for this task, i.e., Quasar-T and SearchQA.
The results of CosmosQA are shown in Table 2. Compared with BERT-Large, KAM-BERT-Large achieves  $10.6\%$  improvement in accuracy. KAM-RoBERTa-Large further improves the accuracy of RoBERTa-Large by  $5.4\%$ , which indicates that our models has better knowledge inference ability. For open-domain QA, our model also achieves better results compared to corresponding baselines. This because that KAM-based models can make full use of the infused knowledge. At the same time, one can notice that KAM-based models have fewer parameters than K-Adaptor, demonstrating its effectiveness for knowledge infusion. WKLM (Xiong et al., 2019) forces the pre-trained language model to incorporate knowledge from a knowledge graph. This makes WKLM to achieve a better score on QA tasks, but KAM-BERT performs even better than WKLM.
<https://gluebenchmark.com
Table 1: Comparison of different model backbones on GLUE benchmark
Model	#Params	Avg	CoLA	SST-2	MRPC	STS-B	QQP	MNLI-m/-mm	QNLI	RTE	WNLI
BERT-Base	109.5M	77.5	50.1	92.6	88.7/84.3	85.7/84.6	71.0/88.9	83.6/83.2	89.4	67.9	65.1
ERNIE	114M	77.5	52.3	93.5	88.2/-	83.2/-	71.2/-	84.0/83.2	91.3	68.8	65.1
SenseBERT	133M	77.9	54.6	92.2	89.2/85.2	83.5/82.3	70.3/88.8	83.6/-	90.6	67.5	65.1
CorefBERT	110M	77.6	51.5	93.7	89.1/-	85.8/-	71.3/-	84.2/83.5	90.5	67.2	65.1
KAM-BERT-Base	112M	78.7	53.7	93.2	89.2/85.0	86.1/84.8	71.2/89.4	84.3/83.5	90.6	67.9	65.1
BERT-Large	345M	80.5	60.5	94.9	89.3/85.4	87.6/86.5	72.1/89.3	86.8/85.9	92.7	70.1	65.1
KAM-BERT-Large	350.4M	81.4	61.5	95.3	91.4/88.1	88.5/87.3	72.5/89.5	87.1/86.6	93.0	72.0	65.1
RoBERTa-Large	355M	83.9	63.8	96.3	91.0/89.4	89.4/87.9	72.7/90.1	89.5/89.7	94.2	84.2	65.1
KEPLER	360M	84.4	63.6	94.5	89.3/-	91.2/-	91.7/-	87.2/86.5	92.4	85.2	65.1
KAM-RoBERTa-Large	361M	84.6	65.1	96.6	92.4/90.7	90.2/88.4	73.2/90.5	89.8/89.7	94.6	85.4	65.1
Table 2: Results for Question Answering
Model	#Params	SearchQA	Quasar-T	CosmosQA
BERT-Large	345M	51.7/61.9	40.4/46.1	68.5
WKLM	348M	58.7/63.3	43.7/49.9	-
WKLM+Ranking	348M	61.7/66.7	45.8/52.2	-
KAM-BERT-Large (Ours)	347M	62.3/67.2	47.0/53.5	69.3
RoBERTa-Large	355M	59.0/65.6	40.8/48.8	80.6
RoBERTa + multitask	355M	59.9/66.7	44.6/51.2	81.2
K-Adapter	384M	62.0/67.3	46.3/53.0	81.8
KAM-RoBERTa-Large (Ours)	361M	64.4/68.6	46.6/53.4	81.9
4.4 LAMA Benchmark
To further verify whether KAM-BERT better integrate internal knowledge into pre-trained language models, we conduct experiments on LAMA, a widely used benchmark for knowledge probing. LAMA examines models' abilities on recalling relational facts by cloze-style questions. The first place micro-averaged accuracy is used as evaluation metrics. The evaluation results are shown in Table 3. KAM-BERT consistently outperforms corresponding baselines on all tasks. It indicates that KAM-BERT can generate better attention maps with semantic guidance.
4.5 Query-Ad Relevance
Query-ad relevance measures how relevant a search ad matches with a user's search query. Very often queries and ads have words with special meanings, which are not easily understood well by traditional NLP techniques but can benefit from the knowledge-assisted mechanism proposed in this work. Besides, user queries and ads text often contain noises, so evaluation on query-ad relevance task would test our model's robustness and resistance of noise. We compare BERT and KAMBERT on a large-scale internal dataset of a commercial search engine. As shown in Table 5, our model outperforms corresponding baselines by a large margin, which is statistically significant under  $95\%$  confidence interval. One thing to call out is that, although NER and syntax parsing results are nosier comparing to the ones in academic datasets, we still have good improvements on this
Table 3: Results for knowledge probing benchmark LAMA
Model	SQuAD	Google-RE	T-REx	ConceptNet
BERT-Large	14.1	9.8	31.1	15.6
KAM-BERT-Large	14.5	10.1	32.2	16.0
RoBERTa-Large	15.9	11.3	33.7	17.1
KAM-RoBERTa-Large	16.3	11.9	34.5	17.3
Table 4: Results for Ad Relevance
α	CoLA	SearchQA	Google-RE
0.0	51.2	59.6/63.8	8.2
0.1	53.0	60.1/64.9	8.6
0.2	53.7	60.7/65.3	8.7
0.4	53.5	60.4/65.0	8.6
Table 5: Hyper-parameter Analysis
Model	#Params	#FLOPs	AUC
BERT-Base	110M	6.3G	73.54
KAM-BERT-Base (Ours)	112M	6.8G	75.95
BERT-Large	345M	11.7G	81.77
KAM-BERT-Large (Ours)	348M	12.2G	83.97
RoBERTa-Base	125M	6.5G	85.87
KAM-RoBERTa-Base (Ours)	127M	6.9G	86.63
dataset. This indicates the way we combine those knowledge together makes our model more robust to noisy inputs.
4.6 Model Analysis
In this section, we explore the sensitivity of hyperparameter  $\alpha$ , and then conduct ablation experiments on three types of added knowledge.
Hyper-parameter Analysis The optimal  $\alpha$  value after grid search is 0.2, which means that the original attention maps still dominate the token relationships. We chose three tasks from different fields to do ablation study for  $\alpha$ . Our model is KAM-BERT-Base, and its performance is shown in Table 4. In three different tasks, setting  $\alpha$  to 0.2 achieves the best results. An intuitive understanding is that when  $\alpha$  is small, external knowledge plays an unimportant role and cannot participate in
Table 6: Ablation study
Model	Avg	CoLA	SST-2	MRPC	STS-B	QQP	MNLI-m/-mm	QNLI	RTE	WNLI
BERT-Base	77.5	50.1	92.6	88.7/84.3	85.7/84.6	71.0/88.9	83.6/83.2	89.4	67.9	65.1
KAM-BERT-Base	78.7	53.7	93.2	89.2/85.0	86.1/84.8	71.2/89.4	84.3/83.5	90.6	67.9	65.1
-KAM-BERT w/o PMI	78.0	52.1	93.0	89.0/84.9	83.5/81.7	71.0/89.1	84.1/83.4	90.2	67.9	65.1
-KAM-BERT w/o Phrase	78.1	53.0	93.1	89.0/84.6	83.2/81.5	71.1/89.1	84.2/83.3	90.5	67.9	65.1
-KAM-BERT w/o Entity	77.7	51.7	92.8	88.9/84.7	82.8/81.0	71.1/88.9	83.8/83.3	89.7	67.9	65.1
-KAM-BERT w/o Conv	78.0	53.1	93.0	89.1/84.9	83.1/81.2	70.8/88.9	84.2/83.3	90.2	67.9	65.1
the entire training process well. With the gradual increase of  $\alpha$ , the intervention of external knowledge on the attention map will increase, and the attention relationship in the original sequence will be gradually lost, resulting in the decline of model performance.
Ablation Study For a comprehensive understanding of our model design, we conduct ablation study with the following settings in Table 6. (1) KAM-BERT w/o PMI: the PMI-based attention maps are removed; (2) KAM-BERT w/o Phrase: the phrase-based attention maps are removed; (3) KAM-BERT w/o Entity: the entity-based attention maps are removed. (4) KAM-BERT w/o Convolution: the convolution layers for knowledge-assisted attention map integration are removed. Instead, we merge all the attention maps through average aggregation.
The average scores of all ablation experiments are better than BERT, but are relatively worse than KAM-BERT, demonstrating all the components are beneficial for the final performance. At the same time, we observe that after deleting the entity attention map, the score of KAM-BERT drops drastically from 78.7 to 77.7. This shows that the gain brought by entity information is the greatest. In addition, the convolution layer is indispensable for achieving a superior performance.
In Figure 3, we visualize an example of query-ad relevance, where the query is "buy glipizide" and ad text is "Fred Meyer Pharmacy Near Me". The darker color in the figure represents a higher attention score. Figure 3(a) is the attention map of vanilla BERT without adding explicit knowledge. When encountering rare words like "glipizide", the self-attention mechanism cannot do a good job to decide which terms should "glipizide" attend to. But in Figure 3(b), the attention map of KAMBERT uses term correlations to learn that "glipizide" is a medicine, so it focuses on the medicine-related tokens like "Pharmacy" and "antidiabetic".
(b) KAM-BERT
(c) BERT
Figure 3: Visualization of attention maps in BERT and KAM-BERT.
(d) KAM-BERT
Figure 3(c) and 3(d) are the attention maps for the sentence "The Academy of Fine Arts is located in Northern Maidan." We only visualize the key parts due to space limitation. Figure 3(c) shows the attention map for vanilla BERT, and 3(d) is the attention map for KAM-BERT. It can be observed that the terms in the same entity are highly correlated with each other in KAM-BERT, which is more reasonable than vanilla BERT.
In this paper, we proposed KAM-BERT, a flexible and efficient approach to inject knowledge into transformer-based pre-trained models. Extensive experiments on GLUE and LAMA benchmark show that our approach outperforms all BERT-Style baselines and achieves new SOTA on QA tasks, suggesting that our models indeed integrate knowledge in an effective manner and have good generalization ability. In future work, we hope to investigate more types of knowledge which can be effectively integrated in our framework.
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. 2016. A latent variable model approach to pmi-based word embeddings. TACL, 4:385-399.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In *NAACL-HLT* (1).
John Hewitt and Christopher D Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 NAACL-HLT(1), pages 4129-4138.
Sarthak Jain and Byron C Wallace. 2019. Attention is not explanation. In Proceedings of the 2019 NAACL-HLT(1), pages 3543-3556.
Ganesh Jawahar, Benoit Sagot, and Djame Seddah. 2019. What does bert learn about the structure of language? In Proceedings of the 57th Annual Meeting of the ACL, pages 3651-3657.
Anne Lauscher, Ivan Vulic, Edoardo Maria Ponti, Anna Korhonen, and Goran Glavaš. 2019. Informing unsupervised pretraining with external linguistic knowledge. arXiv preprint arXiv:1909.02339.
Yoav Levine, Barak Lenz, Or Dagan, Ori Ram, Dan Padnos, Or Sharir, Shai Shalev-Shwartz, Amnon Shashua, and Yoav Shoham. 2020. SenseBERT: Driving some sense into BERT. In Proceedings of the 58th Annual Meeting of the ACL.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
David Nadeau and Satoshi Sekine. 2007. A survey of named entity recognition and classification. Linguisticae Investigationes, 30(1):3-26.
Matthew E Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A Smith. 2019. Knowledge enhanced contextual word representations. In Proceedings of EMNLP-IJCNLP'19, pages 43-54.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the ACL: System Demonstrations.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30:5998-6008.
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Cuihong Cao, Daxin Jiang, Ming Zhou, et al. 2020. K-adapter: Infusing knowledge into pre-trained models with adapters. arXiv preprint arXiv:2002.01808.
Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhiyuan Liu, Juanzi Li, and Jian Tang. 2019. Kepler: A unified model for knowledge embedding and pretrained language representation. arXiv preprint arXiv:1911.06136.
Yujing Wang, Yaming Yang, Jiangang Bai, Mingliang Zhang, Jing Bai, Jing Yu, Ce Zhang, Gao Huang, and Yunhai Tong. 2021. Evolving attention with residual convolutions. arXiv preprint arXiv:2102.12895.
Wenhan Xiong, Jingfei Du, William Yang Wang, and Veselin Stoyanov. 2019. Pretrained encyclopedia: Weakly supervised knowledge-pretrained language model. In ICLR.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019 Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. Ernie: Enhanced language representation with informative entities. In Proceedings of the 57th Annual Meeting of the ACL, pages 1441-1451.