KEPLET: Knowledge-Enhanced Pretrained Language Model with Topic Entity Awareness
Authors: Yichuan Li
Jialong Han
Kyumin Lee
Chengyuan Ma
Benjamin Z. Yao
Xiaohu Liu
Venue: EMNLP 2023 Findings (2023)
Keywords: language model
knowledge enhanced language model

Abstract: In recent years, Pre-trained Language Models (PLMs) have shown their superiority by pre-training on unstructured text corpus and then fine-tuning on downstream tasks. On entity-rich textual resources like Wikipedia, Knowledge-Enhanced PLMs (KEPLMs) incorporate the interactions between tokens and mentioned entities in pre-training, and are thus more effective on entity-centric tasks such as entity linking and relation classification. Although exploiting Wikipedia's rich structures to some extent, conventional KEPLMs still neglect a unique layout of the corpus where each Wikipedia page is around a topic entity (identified by the page URL and shown in the page title). In this paper, we demonstrate that KEPLMs without incorporating the topic entities will lead to insufficient entity interaction and biased (relation) word semantics. We thus propose KEPLET, a novel {K}nowledge-{\'E}nhanced {P}re-trained {L}anguag{E} model with {T}opic entity awareness. In an end-to-end manner, KEPLET identifies where to add the topic entity's information in a Wikipedia sentence, fuses such information into token and mentioned entities representations, and supervises the network learning, through which it takes topic entities back into consideration. Experiments demonstrated the generality and superiority of KEPLET which was applied to two representative KEPLMs, achieving significant improvements on four entity-centric tasks.

Pre-trained language models (PLMs) (Radford al., 2018; Devlin al., 2019; Liu al., 2019) have shown their effectiveness many natural lan- guage understanding tasks. To exploit the rich syn- tactic and semantic information in the pre-training KÉPLET: Knowledge-Enhancel Pretrained Language Model with Topic Entity Awareness
Yichuan Li" Worcester Polytechnic Institute yli29 @ wpi.edu
Jialong Han Airbnb Inc.
Kyumin Lee
Worcester Polytechnic Institute
jialonghan @ gmail..co
kmlce @ wpi.edu
Chengyuan Ma and Benjamin Yao and Derek Liu
Amazon Alexa Al
maannyyunmmmaan @aaa@gmillcom, benjamy amazon.com, derecliu @ amazon.com
Abstract
https /tttps//ee.wwee wttp:///wwwwwwiiiiaaa.m/m org Mwwiki/B/ieeeeee
Title: Beyoncé
In recent years, Pre-trained Language Mod- els (PLMs) have shown their superiority by pre-training unstructured text corpus and then fine-tuning on downstream tasks. On entity-rich textual resources like Wikipedia, Knowledge-Enhancee PLMs (KEPLMs) incor- porate the interactions between tokens and mentioned entitiees in pre-training, and thus more effective on menttyccettrc tasks such as entity linking and relation classification. though exploiting Wikipedia's ss rich structures to some extent, conventional KEPLMs still ne- glect unique layout of the corpus where each Wikipedia page is around topic entity (iden- tified by the page URL and shown in the page title). In this paper, demonstrate that KT- PLMs without incorporating the topic entities will lead to insufficient entity interaction and biased (relation) word semantics. We thus propose KÉPLET, a novel Knowledge-Enhaneed Pre-trained LanguagE model with Topic entity awareness. In end-to-end manner, KÉPLET identifies where to add the topic entity's 's in- formation in a Wikipedia sentence, fuses such information into token and mentioned enti- ties representations, and supervises the net- work learning, through which it takes topic entities back into consideration. Experiments demonstrated the generality and superiority of representa-
Words
She
released
Crazy
Love
and
Mentioned Entitie
Crazy
Love
Baby
Boy
Words
She
released
Crazy
Love and
Baby
Boy
Tapic Enilyy
Beyancé
Mentioned Entilie
Cruzy
In
Love
Baby
Boy
released
Crazy
Love
Baby
Figure I: An illustration of Wikipedia page Beyoncé. There are three levels of interactions among words, mentioned entities and topic entity. We use to stand for the entity and words linkage, and - to express an entity interaction need be considered and word se- mantics need to be covered by modeling the topic entity.
data, PLMs are designed model the word co- occurrences shown the top of Fig. However, they usually fall short discovering factual knowl- edge (Logan et al,, 2019) and applying such knowl- edge in language understanding (Zhang et al., 2019). For example, in sentence "She released Crazy in Love" on the Wikipedia page of Bey- oncé, PLMs will try mask and predict words like "Crazy" and "in". not knowing that Crazy Love mentioned entity but tearing apart. To incor- porate entity knowledge into PLMs, knowledge- enhanced PLMs (KEPLMs; Zhang et al.; Yamada et al.: Qin et al.; Liu et al.) are proposed to work on not only word co-occurrences but also inter- actions between words and mentioned entities well among the latter. As shown in the middle of Fig. 1, most KEPLMS work entity-rich textual resources like Wikipedia, and consider hyperlinks of Wikipedia page as mentions of the target pages' entities to model the rich interactions. Auxiliary objective functions are also designed to enforce the
KÉPLET which was applied tive KEPLMS, achieving significant improve- ments on four entity-centric tasks!
1 Introduction
"Wok performed during internshup at Amazon. Work done when Jialong was with Amazon. 'Code is available at https//gihuh..co//ihhhee KEPLET.

---

model learn entity knowledge from interactions. In general, KEPLMS have achieved superior per- formance compared with PLMs in entity-centric tasks (Safavi and Koutra, 2021; Yin et al., 2022), like named entity recognition, entity typing etc.
To exploit mature KEPLMS its base while filling in the gap that topic entities do not have explicit positions, KÉPLET features topic entity fusion module. To integrate topic entities into KEPLMs, KÉPLET first identifies potential fusing positions in sentences where topic entities can clarify local am- biguity including but not limited to co-references, through gated neural network function (Hochre- iter and Schmidhuber, 1996). It then fuse the topic entity features into the hidden representations of those positions in a soft manner. Finally, KÉPLET trains the fusion module with the base KEPLM in an end-to-end manner, through a specially de- signed topic-entit--wwar contrastive loss. To val- idate the generality and effectiveness of KÉPLET, we conducted comprehensive experiments based two representative KEPLMs, i.e., LUKE (Ya- mada et al., 2020) and ERNIE (Zhang al., 2019), on entity-centric benchmarks. The results demon- strate that KÉPLET consistently improves the per- formance of these two KEPLMS across the tasks.
KEPLMs have gained their efficacy by modeling entities mentioned in Wikipedia-like textual cor- pus. Unfortunately, they still did not fully exploit the entity knowledge there, in the sense that they neglected another important role entities named, topic entities, which are page titles in the corpus. For example, in Fig. 1, those sentences are actually from the page of Beyoncé and around this celebrity topic. However, conventional KEPLMs sim- ply neglect such linkages, and treat those sentences independently, which will lead to both insufficient entity interaction and biased (relation) word se- mantics. Take the sentence "She released Crazy in Love and Baby Boy." for example, if ignoring its topic entity Beyoncé, KEPLMS can no longer rely on this sentence to capture the interaction be- tween Crazy in Love and Beyoncé. Moreover, the semantics of word "released" will be biased since is between the above pair of entities, not between Crazy in Love and common word "she".
We summarize our contributions as follows: 1) We identify the systematic neglect of topic entities in existing KEPLM efforss.. 2) We propose KÉPLET with a novel topic entity fusion module and topic-entity-aware loss, forcing existing KEPLMS to fully exploit entity knowledge in Wikipedia-like corpus. 3) KÉPLET achieves the best performance among existing KEPLMS on several entity-centric benchmarks.
Based on the above discussions, topic entities are indeed important to sentence semantics in Wikipedia-like textual corpus. However, they are non-trivial to model with simple revisions to exist- ing KEPLMs. Readers may wonder whether they be treated similarly as mentioned entities, e-g., assign them certain position embeddings. How- ever, we note that impractical since most topic entitics do not explicitly appear in the sentences like mentioned e Readers may also think of using a co-reference resolution model to replace words like "she" with the topic model's mention before feeding them to KEPLMs. However, em- bedding such model to KEPLMs will not only in- troduce resolution noise, but also be insufficient to cover cases where topic entity information can clar- ify the local semantics of non-pronoun positions, e.g., entity mentions with ambiguous 3
2 Preliminaries
In this section, we define two types of entities es- sential in Wikipedia-based KEPLM pre-training, ie., mentioned entities and topic entities. We also give an overview of how conventional KEPLMs enhance PLMs by incorporating mentioned enti- ties, but neglect topic entities.
2.1 Notations and Definitions
Mentioned Entities. On a Wikipedia page, each sentence S consists of a sequence of tokens W = [w;) and hyperlinks under some tokens, linking to other Wikipedia pages and forming mentioned en- tities. For example, in Fig. 1, the last sentence with tokens W [She, released, Crazy, in, Love] men- tions song entity Crazy in Love with the last three words. In this paper, we denote mentioned entities in a sentence by E = (e;), where each e; has infor- mation about both the entity (e.g.. Wikipedia URL of Crazy in Love) and the position of the mention
In this paper, we seek to fix the systematic neglect of topic entities in existing KEPLM ef- forts. We develop KÉPLET, general Knowledge- Énhanced Pre-trained LanguagE model with Topic entity awareness, which applies to most KEPLMs.
"In our initial analysis, for top-500K popular Wikipedia pages, only 6% of these pages mention the topic entity. "Crazy in Love" can refer to not only Beyoncé' 's song, but also an album of ltzy, a Korean girl group.

---

span in the sentence.
Neglect of Topic Entities. Although more ef- fective than PLMs in entity-centric tasks, con- ventional KEPLMs did not fully exploit the rich structure inside a Wikipedia corpora. By work- ing on only W and of a Wikipedia sentence S (e,, E), those KEPLMS neglect the link- age between S and its topic entity et, leading to two weaknesses follows: 1) Besides transformer parameters and initial word vectors, KEPLMS also have third type of parameters, i.e., initial entity vectors, to keep information of entities seen on the pre-training corpora for both topic and mentioned entities. Neglecting e, of every sentence will dete- riorate the learning of vectors for entities occurring more as topic entities but mentioned less. 2) In a Wikipedia corpus, it is crucial for words repre- senting relations, e-g-, "released" in "She released 'Crazy in Love' ", learn a good initial represen- tation for downstream tasks like relation classifi- cation. Neglecting e, of every sentence will cause relation words in W to interact only with E rather than both, losing their semantics that they charac- terize certain relations between e, and E.
Topic Entities. Besides mentioned entities, every Wikipedia sentence S is also associated with an- other important entity, the topic entity (denoted by ed), which is the entity of the page where the sen- tence is from. Topic entities usually identified by the page URLs and indicated by the pages titles. For example, for the aforementioned sentence in Fig. 1, the topic entity e, is Beyoncé. Although not necessarily mentioning the topic entity e, all sentences on e,'s page are usually around the topic of discussing all aspects of e;-
2.2 PLMs, KEPLMs, and Neglect of Topic Entities
Based the above notations, every Wikipedia sen- tence S is essentially tuple S = (e, W, E). On large-scale textual corpus, PLMs (Radford et al., 2018; Devlin et al., 2019; Liu al., 2019) work on token sequences W by training transformer-syle encoders via specially designed losses LPLM (e.g., masked language modeling).
KEPLMs. On top of PLMs. KEPLMs like LUKE (Yamada et al., 2020) and ERNIE (Zhang et al., 2019) make a further step by utilizing the mentioned entity information in E (Safavi and Koutra, 2021). Specifically, they extend the en- coders to also generate contextualized vectors for the entity mentions, and design entity disambigua- tion losses LED to make those vectors capable of predicating the topic entities. To avoid undermin- ing the lexical and synthetic information of the PLMs, KEPLMs pre-train by jointly optimizing its loss and the conventional PLM losses, i.e.,
In this paper, we improve KEPLMs by bringing such linkages back into the architecture and train- ing of LM in Eq. 3, through which we make up for the above two weaknesses.
KÉPLET: Integrating Topic Entities in KEPLMs
So far, have motivated the fusion of topic en- tities into KEPLMs. The task then boils down to finding proper places in KEPLM training fuse topic entities information. For mentioned entities, existing KEPLMS (Yamada et al., 2020; Zhang et al., 2019; Liu et al., 2020; Lu al., 2021; Xiong et al., 2020; Févry et al.. 2020) fuse their information by considering them as special tokens spanning multiple positions in sentence. How- ever, topic entities non-trivial to incorporate in similar manner, as they do not directly appear in the sentence. Naively assigning them artificial positions and doing insertionnsrreplaceents will (2) potentially break the semantics of the sentences,
LKEPLM = LPLM + LAUX.
(1)
From holistic point of view, KEPLMS aim to LKEPLM pre-train language model LM that can infer contextualized hidden representa- tions for both tokens W and mentioned entities E a Wikipedia-like sentence, i.e.,
LM: (en W) + (H, H.). CM:(W.L. - (H,,. H.).
(3) thus deteriorating the LM's training.
In this section, detail how KÉPLET integrates topic entities. KÉPLET features modules, fus- ing position identification and entity feature and we customize the transformer encoders to ac- commodate the two modules. Besides, it employs a iopic-entity-awee aware contrastive learning loss. Fig. 2
and fine-tune LM in entity-centric downstream tasks. Note that there is special hidden vector bicss in H., which is the conventional sentence representation and will be useful in various down- stream tasks as well as this work.

---

Topio-Enilty-Awar Contrastive Learning
Predict en
Predict Crazyy Love
Transformer th Layer
(ayynnai Seq, Baynnolj Swa, x
järoee Seq.
[Bross] Beq,
Topic Enity Fusion
Beyonos)
[Brooo]
Poslive Instancee
Transformer th Layer
Negativa Instants
and Entry Linkage
Fully Connected Layers
Pointiiee Operaton
(CLS)
She released [MASK] Love
[MASK] Baby Boy
Figure 2: Illustration of KÉPLET with input "She released Crazy in Love and Baby Boy". The mentioned entities Crazy in Love and Baby Boy are linked with the mentioning words. The middle left part is topic entity fusion module interleaved between transformer layers. The model is trained predict the masked words and entities in the top Tepic-ppiityy distance among sentences' representations under Beyoncé and hidden representations Beyoncé while enlarging the distance from sentences of other topic entities, e.g., Bruce.
contrastive learning is used to enforce the topic entity information by minimizing the
illustrates the architecture KÉPLET.
to potentially imperfect model. For each position i, we compute 8p e, to position i, i.e.
(n) indicating the necessity of fusing
3.1 Entity Fusing Position Identification
In conventional KEPLMs, the encoders consist of multiple layers of transformers, each of which de- pends on the contextualized hidden representations of words and mentioned entities from the previous layer. Let Hy- CONCAT(H,,, H.) be all hidden representations for a Wikipedia sentence output by layer (1- - 1), and let h be the i-th column of Hy. From the perspective of preserving the semantics of topic entities e,, we want to find proper positions i to alter the hidden vectors h with information of e, before they are sent to the next layer, so that Hş still carries e'ss semantics even if it is used without knowing what e; is.
i 8 = c(F (F))("")).
(4)
Here Fp is fully connected layer and or is the sigmoid activation function. To qualitatively eval- uate the effectiveness of this module, we did case study in $ 5.5.
3.2 Entity Feature Fusion
With fusing positions softly identified by gp the entity feature fusion module aims to inject iee, information of the topic entity e, the correspond- ing h Inspired by the idea of Adapters (Houlsby et al., 2019) interleaved between the transformer layers, we apply an adapter function on h() and e, to create a fused representation h We then follow the gating mechanism to softly combine O and the original h for an updated input h00 to the next layer I, i.e.,
One ideal type of such positions would be pronoun co-references referring to the topic en- tity, e.g., word "She" in "She released 'Crazy in Love'", They reasonable places to fuse e,'s 's in- formation, since the ambiguous semantics of those positions will be clarified after the fusion. How- ever, we note that co-references are not the only instances such positions. For example, as word mentions, "Crazy in Love" can refer to not only Be- yoncé's song. but also an album of Itzy, a Korean girl group. Fusing Beyoncé's 's information to the three words will thus benefit the representation of this ambiguous span. To this end, we resort to an end end approach of identifying fusing po- sitions instead to embed a co-reference resolution model (Ye et al., 2020) in KÉPLET, for the latter cannot cover all cases and is also error-prone due
f0 Adapter(h 1 e,),
(5)
10 LN((1 -00 h gp h())
10
(6)
Here * is multiplication and LN refers to layer normalization.
As for Adapter(-,-). had two implementa- tions: concatenation fusion and attention fusion. We will compare them in the experiment section. Concatenation Fusion. This fusion approach uses a fully connected layer F, to transform the topic en- tity vector e,, then concatenates with the hidden

---

representations h and feeds the result through another fully connected layer Fe as:
vectors Ar for different topic entity /' We then compute the contrastive learning loss as follow.
Adapter(h, e) = Fo(,ONCAT(h; F((e)))
(7)
Ls Σ log DEAs h'
biimthhh") )TT
Zieehtuu esimbhhi//
Although the concatenation fusion is quite param- eter efficient, assumes that the words' and men- tioned entities' hidden representation as well as topic entity vectors e, are from an unified feature space. This may be incorrect since the topic entity did not have the position information, while the words and mentioned entities had them.
(11)
Here sim is the cosine similarity between vectors and r a temperature hyperparameter. Finally, the overall objective function of KÉPLET is the sum of Ly across the corpora and LXEPLM in Eq. 1.
4 Experiment Settings
Attention Fusion. To discern among topic entity, words, and mentioned entities, we propose the tention fusion. For each hidden vector h, there is an attention fusion around it and the topic entity vector e, as follows:
We evaluate the effectiveness of KÉPLET on exten- sive entity-centric tasks: entity typing, relationship classification, named entity recognition and extrac- tive QA. To ensure fair comparison, we follow the experiment settings of previous work (Yamada et al., 2020; Zhang et al., 2019; Févry al., 2020).
(()))FF(H)))
Adapter(h, e) softmax
(.(H)
vd
4.1 Baseline Methods
(8)
We compare KÉPLET with the vanilla PLMs: I) BERT (Devlin et al., 2019), 2) ROBERTa (Liu et al., 2019), and KEPLMs: 3) KEPLER (Wang et al., 2021b), which utilizes the additional knowledge embedding loss to enhance the factual triplets from the knowledge graph for the PLM; K-Adapter (Wang al., 2021a), which did not explicitly model the entities and adopted the fac- tual knowledge into the external adapter. 5) ERNIE (Zhang et al., 2019), which injects the men- tioned entities' static embedding from KB and dis- tinct the mentioned entity from the negative sam- pled entities; 6) LUKE (Yamada et al., 2020), which contains separated entity embedding and word embedding, and utilizes the auxiliary masked en- tity prediction besides the masked token prediction to optimize the model.
H = CONCAT(T; e,).
(9)
Here Fg. Fs. Fv, are three fully-connected layers get the query, key and value in the self-attention and is the dimension size of hidden vectors h, Finally, we note that all fully connected layers in KÉPLET are layer-specific, i.e., they do not share pa- rameters across different layers in the transformer.
3.3
Contrastive Loss
To enforce KÉPLET to really fuse the topic entity in- formation into its output sentence and entity repre- sentations, we leverage the co-occurrences of sen- tences around the same e, as supervision signals. We design novel contrastive learning loss L, as shown in the right part of Fig. The positive pairs are sentences' and topic entity's ' representa- tion from the same topic entity, while the negative pairs are the sentences' and topic entities' represen- tation from other topic entities in the minibatch. Concretely, given output sentence representa- tion hiCIs] from token representations H, of Wikipedia sentence S and its topic entity's rep- resentation e,, we compute the per-sentence loss Ls as follows. We denote As = (hicLs), e,), and all such vectors of Wikipedia sentences on r by
4.2 Implementation Details
The pretraining Wikipedia corpus is the same as the LUKE (Yamada et al.. 2020) and we fol- low the same data preprocessing steps (Yamada et al., 2020) to extract entities from hyperlinks. For each entity, we assign unique entity ID. The pre-training of KÉPLET starts from LUKE (Yamada et al., 2020) and ERNIE (Zhang et al., 2019), and is optimized for 4.9K steps (1 epoch). The temper- ature r for contrastive learning is set to 0.07. The masking entity rate and masked entity rate are set to 60%. The optimizer for the pre-training is Adam W
U As. on
(10)
For each h E As, we draw a positive sample from A, We also draw negative samples A' from

---

Open Entity TACRED SQuAD CoNLL-2003
sage. We utilize SQuAD1.1 (Rajpurkar et al..
Models
Fi
Fi
2016) and report the exact match (EM) and token-
-Con.
76.00
70.53
91.44
93.09 93.64
level F on the development dataset.
-Atten.
76.38
70.90
92.19
Experiment Results
Table 1: Results of different feature fusion modules on LUKE-base+KÉPLET,
In this section, we aim to answer the following experimental questions: EQ1. Which feature fu- sion method can better integrate the topic entity with KEPLMs? EQ2. Can topic entity fusion im- prove the performance of KEPLMS on downstream tasks? and EQ3. How should the topic entity fu- sion module of KÉPLET be added to KEPLMs to achieve optimal performance?
and is warmed up for 2.5K steps with learning rate as le-5. During the downstream tasks' train- ing and evaluation, the topic entity fusion module will be discarded. This is because there is no topic entity for downstream inputs and the knowledge loss for topic entity, and mentioned entity has al- ready been complemented in the KEPLMS during pre-training. For all these baseline methods, we fine-tune their checkpoints on the same hardware and package settings like KÉPLET. This may cause the performance gap between our reproduction and their previously reported results. The hyperparam- eter settings are described in Appendix B.
5.1 Comparing Entity Feature Fusion Approaches
To answer EQ1, we conduct the experiment on en- tity typing and relation extraction tasks and make comparison between KÉPLET-Con and KÉPLET- Atten. As the results shown in Tab. observe that KÉPLET-Atten achieves better perfor- mance than KÉPLET-Con all the tasks. This indicates there may cxist the unnccessary infor- mation or noise from the topic entity towards the words and mentioned entities, so it is important to selectively to do the feature fusion. Based on this observation, in the following sections, will only report better feature fusion model: KÉPLET-Atten.
4.3 Entity-Centric Tasks
The fine-tuning of entity centric tasks cannot sim- ply take the hidden representation of "[CLS]" to represent the whole sentence. These tasks require special procedures to better represent the entities inside the sentence. Readers can refer to (Yamada et al., 2020; Zhang et al., 2019) for detailed fine- tuning procedures. The summaries and evaluation metrics for these entity centric tasks are as follows: Entity Typing is predict the types of tity given the entity mention and the contextual sentence around the entity mention. Following the previous experiment setting (Zhang et al., 2019; Yamada et al., 2020), we use the Open En- tity dataset (Choi et al.. 2018) and only consider nine popular entity types. We report the precision, recall and micro-F; scores, and use the micro-Fi score for comparison.
5.2 Effectiveness Topic Entity Fusion
To answer EQ2, compare the performance of KÉPLET among the PLMs and KEPLMs. We report the experiment results in Tab. Note that in the lit- erature, different KEPLMS may be initialized with different base PLMs, e.g., ERNIE with BERT-base but KEPLER with RoBERTa-base. Therefore, we conduct three groups of comparisons, each corre- sponding one of the three possible base PLMs (i.e... BERT-base, RoBERTa basss,, and RoBERTa a- large) well as the baseline KEPLMS basing on it. KÉPLET is implemented on the best performing baseline KEPLMS, i.e., ERNIE or LUKE, whose performance is reported in the same group. In general, we have the following observations:
Relation Classification is classify the correla- tion between two entities. The input data includes two entity mentions and contextual information around these two entities. We utilize TACRED dataset (Zhang et al., 2017) and also report the micro-Fi score for the comparison.
Effectiveness of KEPLMs in entity-centric tasks. We observe that in most cases KEPLMs achieve better performance compared with their initialized language models. This is because KIE- PLMs can better capture the entity-level factual knowledge besides the syntax based word co- occurrence information.
Named Entity Recognition is to identify the en- tity from the given sentence. We utilize CoNLL- 2003 (Sang and Meulder, 2003) dataset and report the sean-level Fi scores.
Extractive Question Answering is to answer a question by extracting text span from a given pas-

---

Open Eatity
TACRED
SQUAD 1.1
C-
Models
Prec.
Rec.
Fi
Fi
FM
F
Prec.
Rec.
Fg
BERT-basc
76.37
70.96
73,56
66.00
80.90
88.20
91.28
87.45
89.32
ERNIE
78.42
72.90
75.56
67.97
EKNPEE
79.85
72.01
75.72
70.18
RoBERIa-base
80.62
71.43
75.74
67.95
85.18
91.49
93.24
92.49
92.86
KEPLER
76.78
72.43
74.54
70.70
LUKE-base
79.70
72.37
75.86
70.30
85.18
91.88
92.81
93.27
93.04
LUKE- XXPEEEE
78.39
74.47
76.38
70.90
85.66
92.19
93.48
93.80
93.64
RoBERTa-large
79.36
73.90
76.53
71.73
87.92
94.16
91.23
92.89
92.37
K-Adapter 79.30
75.84
77.53
71.89
LUKE-large 79.53
75.11
77.26
71.81
88. 17
94.29
94.27
94.39
94.33
LUKE-lage+KÉPLET 79.98
75.32
77.58
72.33
88.41
94.47
94.03
94.10
94.07
Results on entity-centric tasks, i.e., entity typing (Open Entity), relationship extraction (TACRED), extractive question answering ($QuAD1.1), and named entity recognition (CoNLL-2003). Bold text indicates best performance and underlined text indicates the second-best performance. We mark "_" where the corresponding baseline was not previously applied on this task.
Table 2:
Effectiveness of advanced KEPLMs. We observe that LUKE achieves better performance than ERNIE all the centrrc tasks. This is because the LUKE learn the better contextual embedding of mentioned entities, while the ERNIE utilizes the static entity embedding.
Open Entity TACRED SQuAD 1.1
CoNLL-2003
Layers
Pj
1
76.38
70.90
92.19
93.64
3
75.66
70.01
91.58
93.31
Table 3: Results of different number of topic entity fusion module layers.
Effectiveness of KÉPLET over KEPLMs. By con- sidering the topic entity knowledge, KÉPLET gets considerable improvement on most tasks, achiev- ing the best second-bess performance within the same comparison group. Specifically, TA- CRED, KÉPLET gets 0.6% performance improve- ment compared with LUKE-base. This not only in- dicates the importance integrating the topic en- tity into KEPLMs pretraining, but also represent the generalization of KÉPLET. Overall, KÉPLET achieves the best performance on all the datasets and metrics. KÉPLET brings the consistent perfor- mance improvement over the backbone KEPLMs.
to the LUKE-base model, which has 12 transformer layers. We first conduct a parameter analysis of adding the fusion module between layers and /- 1" layer As shown Fig. 3, we can observe that fu- sion at lower layerachieve better performance than higher layer. Specifically, performance compared with / = 7and12 22. Since the lower layers contain more general information and the higher layers contain more task-specific infor- mation (Rogers al., 2021), the topic entity bring the general and comprehensive knowledge for the mentioned entities and word semantics.
achieves the best
Task Open Entity
Task CoNLL-2003
In addition, we also evaluate the effectiveness of inserting different numbers of topic entity fusion modules in lower layers. We make the comparison among the single, double and triple topic entity fusion modules on LUKE-base backbone. As the results shown in Tab. 3, single position fusion layer achieves the best performance. This observation represents the efficiency of feature fusion module that can bring enough topic entity knowledge by only doing one time feature fusion.
76
93.50
W
75
93.25
Task 1QuAD1.1
TACRED
92.0
70.5
70.0
91.5
69.5
12
12 Integration Layer
Integration Layer
Figure 3: Parameter analysis of topic entity fusion layer I on KUKE-
5.4 Ablation Study on Topic Entity Fusion
5.3 Parameter Analysis
To delve deeper into the impact our entity fus- ing position identification (FPI) and entity feature fusion (FF) components, we performed an abla-
To address EQ3, we vary where and how many layers of topic entity fusion modules to be added

---

tion study. This allowed us empirically measure the contribution of each module. Given compu- tational constraints, we limited our evaluation to the LUKE-base backbone model. As illustrated in Tab. 4, omitting any particular component ad- versely affected the model's 's performance. These findings bolster our claim that arbitrary position as- signments for topic entity fusion can harm overall effectiveness.
7 Related Work
We group the related KEPLMS works based on what context information is used when encoding sentences with entity knowledge (i.e., mentions), which are depending on sentential context only or additional non-sentential context.
Sentential Context Only KEPLMS. Most KE- PLMs pre-trains by encoding sentences with men- tioned entities and aligning the encoded spans with their ground truth entities. ERNIE (Zhang et al., 2019) integrates static entity embeddings into their PLM with a novel fusion layer to fuse spans' rep- resentations with entities'. LUKE (Yamada et al., 2020) treats both words and entities as tokens but different embeddings, self-attention layers, and masked token/entity prediction heads. It av- erages position embeddings of words in a span as the position embedding of the entity. Differ- ent from ERNIE and LUKE, which require men- tion spans to be available. KnowBERT (Liu et al., 2020) and EAE (Févry et al., 2020) incorporate en- tity linking modules to identify the spans, which trained in end-to-end manner. To deal with entity co-references in sentences, CorefBERT (Ye et al., 2020) and TOME (de Jong al., 2021) pro- pose extending the alignments to not only ex- plicit entity mentions but also co-refeecc spans. WKLM (Xiong et al., 2020) pre-trains by distinguisinng the ground truth entity's embed- dings from those of randomly corrupted ones. K- K-Adaptor (Wang al., 2021a) utilizes plug-in adapters (Houlsby et al., 2019) inject factual and linguistic knowledge without updating the LMs. Non-Sentential Context KEPLMs. While KE- PLMs depending only on sentential context have shown efficacy, other works argue that the align- ment between mention spans and entities can be more effectively done if the mention spans are riched with additional context outside the current sentence. GRAPHCACHE (Wang et al., 2022) con- structs non-sentential context by building a hetero- geneous graph with sentence and property nodes, and their interaction edges. Besides textual con- text, there are also works (Wang et al., 202 Liu et al., 2020; He et al., 2020; Su et al., 2021; Sun et al., 2020; Zhang et al., 2021) utilizing external knowledge graphs (KGs) as more informative non- sentential context. For example, KEPLER (Wang et al., 2021b) injects mentioned entities' descrip- tions and KG facts into its PLM. K-BERT (Liu et al., 2020) fuses KG facts into sentences by con-
Open Entty F,
TACRED
SQUAD CoNLL-2003
# Layers
F,
XÉPLET
76.38
72.33
94.47
94.07
FPI FPI
74.96
70.97
91.71
93.40
w/o FF
75.93
70.22
91.77
93.41
Table Ablation study of KÉPLET on LUKE-base, w/o stands for without, FPI is fusion position identification and FF is feature fusion.
5.5 Case Study of Fusion Position Identification
We aim highlight the sentence segments that KÉPLET perceives as probable integration points. As delincated Tab. 5, we visualize the rank- ings of integrated positions derived from the (i) rank of the LUKE-base backhone. Notably, the model frequently targets pronouns such as "he" and named entities like Farrukhsiyar and Western texe- This pattern aligns with the theo- retical expectation: introducing a topic entity can refine sentence syntax and augment the representa- tion of the cited entities. Such findings corroborate that Model primarily integrates broad topic entity knowledge rather than task-centric details.
6 Computational Overhead Discussion
Our model, trained on Knowledge-Enhancee PLMs (KEPLMs) for single epoch, requires 128 hours an A10 far less than the 11,520 hours needed for KEPLM. Assuming each minibatch holds M documents of C tokens each, the added computational burden over KEPLMs is computed as M M C C M x (M-1), 1). Here, M C represents the topic entity fusion position identifi- cation, M x C x C corresponds to attention-based entity fusion, and Mx(M-1) pertains to the Topic- Antity-Aware Contrastive Loss. This overhead is minimal as both M and C are small constants. Fur- thermore, during inference, KÉPLET bypasses topic entity position and fusion identification, keeping its computational demands in line with standard knowledge-enhance language models.

---

Topic Entity
Words and Mentioned Entities
Hugh Williams Jared November ACP professionally Bon ville English crew is best 408 Rohert Craw ley in the series CRA 2010 2015 surging he a Globe Award two Eimy rockets SHARF was born in Tues puck folder Oscar, to ight who SAY father who $. He was educated ributes elothing College at Sher borne School, independent in D orset Following secondary Bon ville Corpus Christ i. PASS predictions Web ber Douglas of Dire Cambridge exoré beer in theology that he lended do bathi than academic work Bon ville is of the Youth Theatre. Bon ne ville professional IRC appearance was the Open Air avalanche In 1987 Theatre foreground he the Royal 1991 where 1966 's Ham let 1992 1993). ). He in railroad of T 's Wh
K ast ril and later in The Alchemist .. Bon ne ville made television debut, as Bon ville. His film was 1994 Mary Shelley opot and Kenneth roles nat ured umbling characters like 1999 Mr Park Gap BBC series Armored (000 and played more villain ous characters, , leading the dom inc cring Hen leigh Der onda 2002 ) and resilience seized risky) In Again he played hours poet arrii. Iris young Bay ley Orth his by and B AFTA Mountain Luther in The Man ridicule. Bon ne ville works in radio MIA MJoachim Murat! #Primetime Emmy Award for Outstanding Lead Actor in Drama Series#
Hugh Bonneville
#Padkingtont #National Youth Theatre! #Stafford# 42014 FIFA World Cupit #Single-elimination tournament #Gia Longl #Farrukhsiyartt #Bridget Jones's ' Diary# #Cousin# #Juicy Couture# #Nationalism #British Rail Class 08#
Roman active the Grind televised of Sept imus He was Dating tally was a of ens | It is ip stan us who may have Statements design atus but died he to the consulate | Thr ase gent icum descent from the down from the ERA. held the office Carr dime nefarious
Val ius The ase ( ied
ul 196 inate colleague
in Rome around AD 198, listed been Sept im ius Get a, the brother and rival emperor Hc became one of of
trivial upper Pub| earliest ac alla 's autical Get initions Christian has speculated that slight unarmed brightly Pr isc us married possibly sought Caribhean, a speech close Brom the future is isc save son Lucius Mess alla Ap oll aris in ronic ge and Roman Empire AD 193 284 2011 Mess alla etus Lucius #Lobez! #Ukrainiansi #Bernie Sanders#l #Roman aqueducti #Budapest!# #Andrés de Santa Cruzii #Women's Tennis Associationf #Western text-typeff #Ukrainianstt
Lucius
Valerius
Messalla Thrasea
Priscus
Table 5: Examples of fusing positions. Top 50 important words are highlighted. The color saturation indicates the importance. Bold and colored words are either pronouns relevant to the topic entity and the phrases wrapped by # are mentioned entities.
structing a sentence tree. Compared with textual context, KG-based context often requires specially processed and linked knowledge from Wikipedia or WikiData, where noises could be introduced in those processes. In our work, we explore topic entities as unique non-sentential context that is directly available in the Wikipedia structure with- out additional processing.
to make sentence representations capable of pre- dicting sentences' relations. Unlike those works, we construct positive and negative pairs with the unique topic entity information to guide the seman- tic learning of entities and relation words.
8 Conclusion
We propose KÉPLET to address the oversight of topic entities in KEPLM learning. KÉPLET per- forms topic entity fusion by identifying the fusing positions and fusing topic entity features through concatenation- or attention-based fusion. Further- more, we design new pre-training task, i.e., topic- entity-aware contrastive learning, for better topic entity fusion. Experiment results on several entity- centric tasks prove the effectiveness of KÉPLET Potential directions of the future work include: 1) Adopting KÊPLET to multi-language setting. Since topic entities are language invariant (Nishikawa et al., 2022), we can expand the lopic-entity-awann contrastive learning for the same topic entity un- der different languages; 2) Applying KÉPLET to Nomain-specint knowledge-centrr tasks like fact checking and fake news detection.
Contrastive Learning in KEPLMs. Orthogo- nal to various types of context, multiple objectives or tasks have been introduced by previous stud- ies pre-train those KEPLMs. Most of those attempts fall in the contrastive learning setting to minimize distances between positive pairs while enlarging those between negative ones (Hadsell et al., 2006; Gao et al., 2021; Chen et al., 2020; Meng et al., 2021). Therefore, the construction of positive pairs and negative pairs is crucial in this setting. In UCTopic c (Li et al., 2022), positive pairs are sentences that share the same entity mentions, while negative ones are from different mentions. EASE (Nishikawa et al., 2022) extends UCTopic by incorporating entity representations into positive pairs. LinkBERT (Yasunaga et al., 2022) intro- duces a sentence relation classification objective

---

9 Limitations
Tianyu Gao, Xingcheng Yao, and Dangi Chen. 2021. Simese: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Domini- can Republic, 7-11 November, 2021, pages 6894- 6910. Association for Computational I Linguisties.
Since our method has incorporated the topic en- tity into language model pretraining, requires an entity-rich pretraining dataset. The dataset's layout should have topic entity and mentioned en- tities. The demand for special pretraining dataset is a clear limitation of our method. In addition, our pretraining stage require many computational resources (eight A10 cards for up 16 hours, so totally 128 GPU hours). This is because we want to inject the previously neglected topic entity knowledge from Wikipedia into KEPLMs. This will require us to do the further pretraining on the whole Wikipedia dataset. However, given the per- formance improvements brought by our method on many entity-centric tasks, the additional computa- tional cost is totally worthy.
Raia Hadsell, Sumit Chopra, and Yann LeCun. 2006. Dimensionality reduction by learning invariant mapping. In 2006 IEEE Computer Society Confer- ence on Computer Vision and Pattern Recognition (CVPR 2006). 17-22 June 2006, New York, NY, USA, pages 1742. IEEE Computer Society.
Bin He, Di Zhou, Jinghui Xiao, Xin Jiang, Qun Liu, Nicholas Jing Yuan, and Tong Xu. 2020. BERT-MK: Integrating graph contextualized knowledge into pre- trained language models. In Findings the Associ- ation for Computational Linguistiex: EMNLP 2020, pages 2281 2281-201 1-2900, Online. Association for Compu- tational Linguistics.
Sepp Hochreiter and Jürgen Schmidhuber. 1996. Lstm can solve hard long time lag problems. In Proceed- ings of the 9th International Conference on Neu- ral Information Processing Syxtems, NIPS 96, page 473-479, Cambridge, MA, USA. MIT Press.
References
Ting Chen, Simon Kornblith, Mohammad Norouzi, and
Geoffrey E. Hinton. 2020. A simple framework for
contrastive learning of visual representations. In
Proceedings of the 37th International Conference on
Machine Learning, ICMI. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Ma- chine Learning Research, pages 1597-1607, PMLR.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo. Mona Attariyan, and Sylvain Gelly, 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceed- ings of Machine Learning Research, pages 2790- 2799. PMLR.
Eunsol Choi, Omer Levy, Yejin Choi, and Luke 7etll- moyer. 2018. Ultra-fine entity typing. In Proceed- ings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pa- pers). pages 87-96, Melbourne, Australia. Associa- tion for Computational Linguistics.
Jiacheng Li, Jingho Shang, and Julian McAuley. 2022. Uctopic: Unsupervised contrastive learning for phrase representations and topic mining. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume I: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022. pages 6159-6169, Association for Computa- tional Linguistics.
Michiel de Jong, Yury Zemlyanskiy, Nicholas FitzGer- ald, Fei Sha. and William Cohen. 2021. Mention memory: incorporating textual knowledge into trans- formers through entity mention attention. CoRR, abs/2110.0611
Jacob Devlin, Ming-Woi Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019,, Minneapolis. MN, USA, June 2-7, 2019. Volume (Long and Short Papers), pages 4171-4186. Association for Compu- tational Linguistics.
Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang. Qi Ju, Haotang Deng, and Ping Wang. 2020. K- BERT: enabling language representation with knowl- edge graph. In The Foirty-Forrth AAAI Conference Artificial Intelligence, AAAI 2020. The Thirty- Second Innovative Applicutions of Artificial Intelli- gence Conference, IAAI 2020, The Tenth AAAI Sym- posium on Educational Advances in Artificial Intel- ligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 2901-2908 AAAI Press
Thibault Févry, Livio Baldini Soares. Nicholas FitzGer- ald, Eunsol Choi, and Tom Kwiatkowski. 2020. Enti- ties experts: Sparse memory access with entity pervision. In Proceedings of the 2020 Conference on Empirical Methods in Natural Longuage Processing, EMNLP 2020, Online, November 16-20, 2020, pages 4957-4951. Association for Computational Linguis- tics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, 190.11997.119999

---

Robert Logan, Nelson F Liu, Matthew E Peters, Matt Gardner, and Sameer Singh. 2019. Barack's 's wife hillary: Using knowledge graphs for fact-aware lan- guage modeling, In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, pages 5962-5971.
Punta Cana, Dominican Republic, 7-11 November, 2021, pages 1053-1067. Association for Computa- tional Linguistics.
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. In- troduction to the conll-2003 shared task: Language- independent named entity recognition. In Proceed- ings of the Seventh Conference on Natural Language Learning, CoNLL 2003, Held in cooperation with HLT-NAACL 2003, Edmonton, Canada, May 31- June 2003, pages 142-147. ACL.
Yinquan Lu, Haonan Lu, Guirong Fu, and Qun Liu. 2021. KELM: knowledge enhanced pre-trained lan- guage representations with message passing on hier- archical relational graphs. CoRR, 0ttp//109902222
Yu Meng, Chenyan Xiong, Payal Bajaj. Saurabh Ti- wary. Paul Bennett, Jiawei Han, and Xia Song. 2021. COCO-LM: correcting and contrasting text sequenees for language model pretraining. Ad- vances in Neural Information Processing Systems 34: Annual Conference on Neural Information Process- ing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 23102-23114.
Yusheng Su, Xu Han, Zhengyan Zhang, Yankai Lin, Peng Li, Zhiyuan Liu, Jie Zhou, and Maosong Sun. 2021. Cokebert:
Contextual knowledge selection and embedding towards enhanced pre-trained lan- guage models. Al Open.
Tianxiang Sun. Yunfan Shao, Xipeng Qiu. Qipeng Guo, Yaru Hu, Xuanjing Huang, and Zheng Zhang. 2020. Colake:
Sosuke Nishikawa, Ryokan Ri, Ikuya Yamada, Yoshi- masa Tsuruoka, and Isao Echizen. 2022. EASE: entity-aware contrastive learning of sentence em- bedding. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACL 2022, Seattle WA, United States, July 10-15, 2022, pages Computational Linguistics.
Contextualized language and knowledge embedding. In Proceedings of the 28th International Conferencee on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020. pages 3660-3670. International Committee on Computational Linguistics.
Association for
Ruize Wang, Duyu Tang. Nan Duan, Zhongyu Wei, Xuanjing Huang. Jianshu Ji, Guihong Cao. Daxin Jiang, and Ming Zhou. 2021a. K-adapter: Infusing knowledge into pre-trained models with adapters. In Findings of the Association for Computational Lin- guistics: ACL/JCNLP 2021, Online Event, August 1-6, 2021, volume ACL/UCNLP 2021 of Findings of ACI., pages 1405-1418. Association for Compu- tational Linguistics.
Yujia Qin, Yankai Lin, Ryuichi Takanobu, Zhiyuan Liu, Peng Li, Heng Ji, Minlie Huang, Maosong Sun, and Jie Zhou, 2021. ERICA: improving entity and re- lation understanding for pre-trained language mod- els via contrastive learning. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/I- JCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 3350-3363. Asso- ciation for Computational Linguistics.
Xiaozhi Wang, Tianyu Gao. Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. 2021b. 20ng.2021b. KEPLER: A unified model for knowledge embed- ding and pre-trained language representation. Trans. Assoc. Comput. Linguistics, 9:176-194.
Alec Radford, Karthik Narasimhan, Tim Salimans, Nla Sutskever, et al. 2018. Improving language under- standing by generative pre training.
Yiwei Wang, Muhao Chen, Wenxuan Zhou, Yujun Cai, Yuxuan Liang, and Bryan Hooi. 2022. GRAPH- CACHE: message passing as caching for sentence- level relation extraction. CoRR, abs/2205.03786.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang- 2016. Squad: 100, 000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 2383-2392. The Association for Computational Linguisties.
Wenhan Xiong, Jingfei Du, William Yang Wang, and Veselin Stoyanow. 2020. Pretrained encyclopedia: Weakly supervised knowledge- pretrainee language model. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2021. A Primer in BERTology: What We Know About How BERT Works. Transactions of the Asso- ciation for Computational Linguistics, 8:842-866.
Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. LUKE: deep contextualized entity representations with entity- aware self attention. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing, EMNLP 2020, Online, Novem- ber 16-20, 2020, pages 6442-6455 Association for Computational Linguistics.
Tara Safavi and Danai Koutra. 2021. Relational world knowledge representation in contextual language models: A review. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, EMNLP 2021, Virtual Event /

---

Michihiro Yasunaga, Jure Leskovec, and Percy Liang. 2022. Linkbert: Pretraining language models with document links. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Pupers), pages 8003-8016.
Deming Ye, Yankai Lin, Jiaju Du, Zhenghao Liu, Peng Li, Maosong Sun, and Zhiyuan Liu. 2020. Corefer- ential reasoning learning for language representation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 7170- 7186. Association for Computational Linguistics.
Da Yin, Li Dong, Hao Cheng, Xiaodong Liu, Kai- Wei Chang, Furu Wei, and Jianfeng Gao. 2022. A survey of knowledge-intensive NLP with pre-trained language models. CoRR, abs/2202.08772.
Taolin Zhang, Zerui Cai, Chengyu Wang, Minghui Qiu, Bite Yang, and Xiaofeng He. 2021. Smedbert: knowledge knowledge-enhancee eehencehhed pre-trained language model with structured semantics for medical text mining. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and 1th International Joini Conference on Natural Lan- guage Processing, ACL/JJCLPP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 5882-5893, Association for Computational Linguis- tics.
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor An- geli, and Christopher D. Manning. 2017. Position- aware attention and supervised data improve slot fill- ing. In Proceedings of the 2017 Conference on Em- piricul Methods Natural Language Processing, pages 35-45, Copenhagen, Denmark. Association for Computational Linguistics.
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: En- hanced language representation with informative en- tities. In Proceedings ACL 2019.

---

A Details of Pretraining
for topic entity position identification and fusion during inference. This ensures that the computa- tional expense remains comparable to the standard knowledge-enhane language model.
Following the work of LUKE(Yamada et al., 2020), we utilize the same pretraining dataset: December 2018 version of Wikipedia and hyperparameters, except learning rate (1e-6). We separate differ- ent input sequences from same Wikipedia page by 512 words. To provide enough positive pairs for hopie-Entity-wware Contrastive Loss, we modify the code of data minibatch generation for pretrain- ing. Specifically, we firstly combine consecutive elements of the pretraining dataset into batches, then do the shuffling and lastly split the batch into multiple elements. The TensorFlow-style code block is like follows:
# LUKE's s emplementation data data. shuffle() # Our implementation
data b
unbatch O
B Details for Fine-Tuning
Following the hyper-parameter setting from LUKE (Yamada et al., 2020), conduct the hyper-parameter search on all the datasets except SQuAD1.1. We use grid search to find the best model based on validation performance. The metrics for model selection are reported $ 4.3.
All in all, we use the following search space:
learning rate le-5, 2e-5, 5e-5, 5e-6
batch size: 4, 8, 16, 32, 64
number of training epochs: 2, 3.
Computational Overhead
Our model is trained existing Knowledge- EnhancedPLMM (KEPLMs) for one epoch, which typically takes 128 hours on A10 GPU. This sig- nificantly less than the 11,520 hours required to train a KEPLM itself. To provide more detailed breakdown, let's assume each minibatch contains M documents, with each document having C' to- kens. The additional computational cost of ours over KEPLMs is M C Mx C+ M (M- I), where M C is the cost for topic entity fusion po- sition identification and M attention-based entity fusion and M X (M 1) is the -opic-EEtity-Aware Contrastive Loss. It is rel- atively modest compared to the previous method because M and C are small constant values. In ad- dition, at inference time, KÉPLET avoids the need
the cost for


Reviews:
[{"review_id": "R7wIIIVSwP", "reviewer": "Reviewer_5vf9", "summary": "", "questions": "A: Regarding topic entities, you state that \"naively assigning them artificial positions and doing insertions/replacements will potentially break the semantics of the sentences, thus deteriorating the LM’s training\". Did you test this, e.g. by ablative testing on the fusing efforts? Given the massive training requirements for your model, quantifying the necessity seems pertinent.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and helpful comments. We have addressed the concerns below. If you have any further questions, please let us know.\n\n> **Q1**. Performance / cost tradeoff While the model is novel and improves on the SOTA, the gains in performance are quite small yet costly to achieve.\nA more thorough evaluation and discussion of the added pre-training overhead would be preferable, rather than just mentinoning it in the limitations. \n\n**A1**. Thanks for the questions. We resolve your concerns as follows:\n1. **Computational Overhead**:  \n   - Test efficiency: Our model avoids the need for topic entity position identification and fusion during inference. This ensures that the computational expense remains comparable to the standard knowledge-enhanced language model.\n   - Training efficiency: Our model is trained on existing Knowledge-Enhanced PLMs (KEPLMs) for just one epoch, which typically takes 128 hours on A10 GPU. This is significantly less than the 11,520 hours required to train a KEPLM itself. To provide a more detailed breakdown, let's assume each minibatch contains M documents, with each document having C tokens. The additional computational cost of ours over KEPLMs is M * C + M * C * C + M * (M - 1), where M*C is the cost for topic entity fusion position identification and M*C*C is the cost for attention-based entity fusion and M*(M-1) is the Topic-Entity-Aware Contrastive Loss. It is relatively modest compared to the previous method because M and C are small constant values. \n2. **Performance Evaluation**: Thanks for raising concerns about our improvements being insignificant. However, it has been challenging to consistently yet significantly improve downstream tasks via improving the PLMs:\n   - We follow previous experimental protocols to fine-tune the same PLM to obtain models on different tasks. But those tasks may have different optimal PLMs to start with. \n   - The insufficient and noisy labels on downstream tasks may potentially limit the final performance regardless of any superior PLMs used.\n   - While we achieved consistent improvements, in fact, our improvements on top of ERNIE/LUKE-base/LUKE-large (average F1 gain across the tasks: 1.185%, 0.5075%, 0.19%) are already comparable with theirs (average F1 gain: 1.985%, 0.76%, 0.725%) on their baselines (BERT/RoBERTa-base/RoBERTa-large). The only smaller average improvement of 0.19% on LUKE-large is due to it having a 2% gain on CONLL-2023 while we had none.\n\nTo summarize, while our absolute improvements did not look significant due to the nature of applying PLMs, we achieved consistent improvements on top of our strong baselines (ERNIE/LUKE) which is comparable with what they had on their baselines.\n\n> **Q2**. Regarding topic entities, you state that \"naively assigning them artificial positions and doing insertions/replacements will potentially break the semantics of the sentences, thus deteriorating the LM’s training\". Did you test this, e.g. by ablative testing on the fusing efforts? Given the massive training requirements for your model, quantifying the necessity seems pertinent.\n\n**A2**. Thanks for your suggestion. In fact, we have conducted an ablation study using LUKE-base to empirically evaluate the contribution of each component. The results confirmed that on removing each specific component, performance was negatively impacted. This provides empirical support to our assertion that arbitrary position assignments for topic entity fusion can be detrimental to performance.\n\nWe acknowledge the importance of this result, and we will ensure this experiment result is incorporated into the revised manuscript for better clarity and thoroughness.\n\n|                                            | Open Entity | CoNLL-2003 | Dev SQuAD 1.1 | Tacred    |\n|--------------------------------------------|-------------|------------|---------------|-----------|\n| Methods                                    | F1          | F1         | F1            | micro     |\n| LUKE-base+KÉPLET                           | **76.38**   | **94.07**  | **94.47**     | **72.33** |\n| LUKE-base+KÉPLET w/o Entity Identification | 74.96       | 93.40      | 91.71         | 70.97     |\n| LUKE-base+KÉPLET w/o Entity Fusion         | 75.93       | 93.41      | 91.77         | 70.22     |\n\n> **Q3**. Relation Classification is to classify the correlation between two entities.\" seems misleading - it isn't really a correlation, is it?\n\n**A3**. Thanks for pointing out the misleading part. Yes. It is better to replace the correlation with relationship.\n\n> **Q4**. Why is SQuAD 1.1 used when it is well known to have limitations and SQuAD 2.0 is available?\n\n**A4**. Thank you for highlighting this. We opted for SQuAD 1.1 primarily to maintain consistency with the experimental settings of other notable knowledge-enhanced pretrained language models such as LUKE[1], CorfBERT[2], and KELM[3]. While SQuAD 2.0 addresses certain limitations by introducing adversarially unanswered questions, SQuAD 1.1 remains a valuable benchmark. It compels KEPLMs to fully leverage entity interactions and capture word semantics without bias, essential for pinpointing the correct answer span.\n\n[1] Yamada, Ikuya, et al. \"LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention.\" EMNLP 2020.\n\n[2] Ye, Deming, et al. \"Coreferential reasoning learning for language representation.\" arXiv preprint arXiv:2004.06870 (2020).\n\n[3] Lu, Yinquan, et al. \"KELM: knowledge enhanced pre-trained language representations with message passing on hierarchical relational graphs.\" arXiv preprint arXiv:2109.04223 (2021).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The model design is novel, adding to PLMs in an intuitive manner that stand to benefit the community and are easy to include in existing methods.\n\n* The proposed model is evaluated thoroughly, demonstrating improvements to SOTA performance. The experiments clearly outlines the datasets and models that are used, ensuring reproducibility of the findings.\n\n* The paper is very well written, with the model being well explained and supported by clear examples and strong arguments.", "reasons_to_reject": "* Performance / cost tradeoff\nWhile the model is novel and improves on the SOTA, the gains in performance are quite small yet costly to achieve. A more thorough evaluation and discussion of the added pre-training overhead would be preferable, rather than just mentinoning it in the limitations. With the trend towards larger language models and given the small observed benefit of adding topic enities, having sufficient information to make an informed decision with regard to this trade-off would be important. It would also be be good to include this tradeoff in an ethics discussion.", "questions_to_authors": "A: Regarding topic entities, you state that \"naively assigning them artificial positions and doing insertions/replacements will potentially break the semantics of the sentences, thus deteriorating the LM’s training\". Did you test this, e.g. by ablative testing on the fusing efforts? Given the massive training requirements for your model, quantifying the necessity seems pertinent.", "paper_topic_and_main_contributions": "The authors present KEPLET, a novel method for pre-training knowledge-enhanced pretrained language models (KEPLMs) that leverage inter-article hyperlinks from knowledge bases in pre-training language models for entity related tasks. To this end, the authors suggest the use of topic entities (i.e., knowledge base page title entities) as an additional training signal, by fusing semantic information about word tokens, mentioned entities and topic entities into the hidden representation learned during pre-training. This pre-training is effectively implemented as an adapter, (1) utilizing a GAN-based model that is jointly trained with the other transformer-based layers, and (2) conditioning the model to predict both the masked token and the mentioned entities in a sentence.\nThe proposed method is in experiments with BERT, RoBERTa, and ERNIE-based KEPLMs, demonstrating improvements on SOTA performance on several entity-centric benchmarks.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "-"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The paper is well written end enjoyable to read. Great job!\n\nTypos:\nline 50: references are missing years\nLine 207: \"Wikipedia corpora\" -> \"Wikipedia corpus\"\nline 436: \"Relation Classification is to classify the correlation between two entities.\" seems misleading - it isn't really a correlation, is it?\n\nWhy is SQuAD 1.1 used when it is well known to have limitations and SQuAD 2.0 is available?"}}, {"review_id": "H1gJ8aWhty", "reviewer": "Reviewer_pRqK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and helpful comments. We have addressed the concerns below. If you have any further questions, please let us know.\n\n\n> **Q1**. The results reported in Tables 1,2 and 3 show a small improvement with respect to the state of the art. Without a significance test, it is hard to say that the proposed approach outperforms significantly the approaches available in the literature.\n\n**A1**. Thank you for raising concerns about our improvements. However, it has been challenging to consistently and significantly improve downstream tasks via improving the PLMs.\n\nWe follow the previous experimental protocols to fine-tune the same PLM to obtain models on different tasks. But those tasks may have different optimal PLMs to start with. \nThe insufficient and noisy labels on downstream tasks may potentially limit the final performance regardless of any superior PLMs used. \n\nWhile we achieved consistent improvements, in fact, our improvements on top of ERNIE/LUKE-base/LUKE-large (average F1 gain across the tasks: 1.185%, 0.5075%, 0.19%) are already comparable with theirs (average F1 gain: 1.985%, 0.76%, 0.725%) on their baselines (BERT/RoBERTa-base/RoBERTa-large). In other words, our improvements are worth to the research community. The only smaller average improvement of 0.19% on LUKE-large is due to it having a 2% gain on CONLL-2023  while we had none.\n\nTo summarize, while our absolute improvements did not look like a lot due to the nature of applying PLMs, we achieved consistent improvements on top of our strong baselines (ERNIE/LUKE) which are comparable with what they had on their baselines reported in the previous papers [1, 2].\n\n[1] Zhang, Zhengyan, et al. \"ERNIE: Enhanced Language Representation with Informative Entities.\" ACL 2019.\n\n[2] Yamada, Ikuya, et al. \"LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention.\" EMNLP 2020.\n\n\n> **Q2**. The cost-benefit of the proposed loss is not evident. The training phase is more complex, due to the contrastive loss, without a significant improvement of the performance in all the downstream tasks.\n\nThank you for raising concerns about the cost-benefit trade-off of our proposed loss. We would like to resolve your concerns as follows:\n- **Comparison with KEPLMs**: Firstly, it's important to highlight that while our model incorporates a contrastive loss, it's trained on existing Knowledge-Enhanced PLMs (KEPLMs) for just a single epoch. This translates to approximately 128 hours on an A10 GPU. In contrast, training a KEPLM from scratch requires a substantial 11,520 hours. This means our method leverages the power of KEPLMs but does so more efficiently.\n\n- **Computational Breakdown**: To offer a clearer perspective, consider a minibatch containing M documents, where each document encompasses C tokens. The additional computational overhead introduced by our method over traditional KEPLMs is M * C + M * C * C + M * (M - 1), where  M*C\nis the cost for topic entity fusion position identification \nand M*C*C is the cost for attention-based entity fusion and M*(M-1) is the Topic-Entity-Aware Contrastive Loss. It is relatively modest compared to the previous method because M and C are small constant values. \n\nIn summary, we believe our approach strikes a balanced trade-off between computational efficiency and performance enhancements, especially when contextualized against the computational demands of traditional KEPLMs.\n\n\n> **Q3**. Low Reproducability.\n\n**A3**. Thank you for highlighting your concerns regarding the reproducibility of our results. We tried our best to detail the hyper-parameter settings and their respective analysis and the configurations but there were still space limitations. If accepted we plan to use the additional page given in camera ready to list more details\".\n\nWe have also uploaded our source code along with the training scripts to the supplementary materials. These resources are intended to offer researchers an in-depth view of our implementation, further aiding in reproduction. After the acceptance of our paper, we plan to release the model checkpoints.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- the paper, given the intuition that topic-entity relations could play a fundamental role in downstream tasks (such as entity linking), introduces a contrastive loss to include such topic entity information into the sentence and entity representations. They use a sort of distant supervision (co-occurrences of sentences around the same entity) to create positive examples to keep sentence and entity representation closer, and negative samples to keep instances far from the correct association. \n", "reasons_to_reject": "The main reasons to reject the paper are:\n\n1) The results reported in Tables 1,2 and 3 show a very small improvement with respect to the state of the art. Without a significance test, it is hard to say that the proposed approach outperforms significantly the approaches available in the literature.\n\n2) The cost-benefit of the proposed loss is not evident. The training phase is more complex, due to the contrastive loss, without a significant improvement of the performance in all the downstream tasks. \n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces a framework named  Knowledge-Énhanced Pre-trained LanguagE model  with Topic entity awareness (KEPLET) to overcome the poor relations semantics in the state-of-the-art KEPLMs model. The main contributions of the paper are:\n1) systematic analysis of the topic entities  that affect the original KEPLM model\n2) an  entity fusion module and a suitable loss function to deal with topic-entity relationships\n", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aSzzgUSy1t", "reviewer": "Reviewer_pHNa", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and helpful comments. We have addressed the concerns below. If you have any further questions, please let us know.\n\n> **Q1**. It's still unclear how topic entities can improve the relationship representations. This claim is less intuitive.\n\n**A1**. Thank you for bringing up the significance of topic entities in relationship representation. We will address your concerns as follows:\n\n- **The Role of Topic Entities**: Topic entities play a pivotal role in enhancing the representations of mentioned entities and counteracting biases in sentence representation learning. In many entity-rich textual resources like Wikipedia, topic entities might not be explicitly stated in the text, even though they underpin the narrative. Mentioned entities, on the other hand, are directly present.\n- **Practical Illustration**: To elucidate, consider the example from lines 78-85: “She released Crazy in Love and Baby Boy.” Here, while \"Crazy in Love\" and \"Baby Boy\" are the mentioned entities, the topic entity, \"Beyoncé\", is implied but not stated. Without recognizing the topic entity, methods like LUKE[1] and ERNIE[4] might form biased associations between “released” and \"Crazy in Love\", or even form irrelevant entity pairs, missing out on the actual <topic entity, mentioned entity> relationship.\n- **Our Approach with KEPLET**: Contrary to this, our method, KEPLET, mitigates such pitfalls. As depicted in Figure 2, KEPLET adeptly fuses the topic entity with relevant pronoun tokens, ensuring unbiased learning of relationship representations. This results in relation-specific words like \"released\" being tightly associated with numerous <topic entity, mentioned entity> pairs, thereby leading to more precise representations.\nEmpirical Evidence: To further validate our claims, we've included a case study in the appendix. It provides a qualitative analysis of our topic entity fusion position identification module. Since these fusion positions encapsulate pronouns pertinent to both topic and mentioned entities, they better capture the sentence's syntax, further solidifying the relationship between topic and mentioned entities.\n\nIn essence, topic entities, as integrated by our method, serve as foundational anchors, enhancing the quality of relationship representations and making them more robust against biases.\n\n[1] Yamada, Ikuya, et al. \"LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention.\" EMNLP 2020.\n\n> **Q2**. The improvements on different datasets are trivial and the novelty of this paper is limited. Lots of previous works focus on this topic. Just adding topic entities seems incremental.\n\n**A2.1**. Thank you for raising concerns about our improvements being incremental. We aim to resolve your concerns as follows:\n\n- It has been challenging to consistently yet significantly improve downstream tasks via improving the PLMs: We follow previous experimental protocols to fine-tune the same PLM to obtain models on different tasks. But those tasks may have different optimal PLMs to start with. The insufficient and noisy labels on downstream tasks may potentially limit the final performance regardless of any superior PLMs used.\n- While we achieved consistent improvements, in fact, our improvements on top of ERNIE/LUKE-base/LUKE-large (average F1 gain across the tasks: 1.185%, 0.5075%, 0.19%) are already comparable with theirs (average F1 gain: 1.985%, 0.76%, 0.725%) on their baselines (BERT/RoBERTa-base/RoBERTa-large). In other words, our improvements are worth to the research community. The only smaller average improvement of 0.19% on LUKE-large is due to LUKE-large having a 2% gain on CONLL-2023 while we had none.\n\nTo summarize, while our absolute improvements did not look significant due to the nature of applying PLMs, we achieved consistent improvements on top of our strong baselines (ERNIE/LUKE) which is comparable with what they had on their baselines.\n\n**A2.2**Thank you for the question about the novelty of our work. Our contribution is focused around the notion of topic entities and it will great impacts on future KEPLM research. KEPLM is a large research direction as evidenced by the 2 surveys[1][2]. \n\nIn this project, we identified topic entity as a source of supervision signals, a systematic negligence of hundreds of papers in this research direction; we  thoroughly studied how to bring it back; we show that our approach can be applied to multiple KEPLMs so we are orthogonal to potentially hundreds of existing studies. \n\n[1] Zhen, Chaoqi, et al. \"A survey on knowledge-enhanced pre-trained language models.\" arXiv preprint arXiv:2212.13428 (2022).\n\n[2] Wei, Xiaokai, et al. \"Knowledge enhanced pretrained language models: A compreshensive survey.\" arXiv preprint arXiv:2110.08455 (2021).\n\n> **Q3**. Some methods (e.g., KnowBERT, CorefBERT) in related work are not selected as baselines for comparison.\n\n**A3**. We appreciate your concern regarding the selection of baseline methods for comparison.\nWhile KnowBERT[1] and CorefBERT[2] are indeed noteworthy models, we initially selected LUKE as our primary baseline because it demonstrated superior performance over both KnowBERT and CorefBERT in prior evaluations. This made LUKE a more challenging and representative benchmark for our method.\n\nHowever, recognizing the importance of comprehensive comparisons, we have also extracted performance metrics of KnowBERT and CorefBERT from their original publications for reference. As presented in the following table, our proposed method consistently surpasses the performance of not only LUKE but also that of CorefBERT and KnowBERT.\n\nWe believe this comprehensive comparison further establishes the effectiveness of our approach and provides readers with a clearer perspective on the landscape of available models.\n\n|                   | Open Entity | CoNLL-2003 | Dev SQuAD 1.1 | Tacred    |\n|-------------------|-------------|------------|---------------|-----------|\n|                   | F1          | F1         | F1            | micro     |\n| CorefBERT         | -           | -          | 91.8          | -         |\n| KnowBERT          | 76.10       | -          | -             | 71.5      |\n| LUKE-large        | 77.26       | **94.33**  | 94.29         | 71.81     |\n| LUKE-large+KÉPLET | **77.58**   | 94.07      | **94.47**     | **72.33** |\n\nWe also believe our method has the potential to be adapted and give benefit to other KEPLMs, including KnowBERT and CorefBERT, which opens avenues for future work.\n\n> **Q4.** Missing related work.\n \n**A4**. We will add the suggested related work in our revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Authors identify the topic entities in Wikipedia pages for knowledge-enhanced language model pretraining. Topic entities are absent in previous work.\n2. To incorporate topic entities to represent entities and relations, authors propose a fusion method and topic-entity-aware contrastive loss.", "reasons_to_reject": "1. It's still unclear how topic entities can improve the relationship representations. This claim is less intuitive.\n2. The improvements on different datasets are trivial and the novelty of this paper is limited. Lots of previous works focus on this topic. Just adding topic entities seems incremental. \n3. Missed related work.\n4. Some methods (e.g., KnowBERT, CorefBERT) in related work are not selected as baselines for comparison.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors of this paper recognize that topic entity in the Wikipedia pages can improve the entity representations and their relation representations for pre-trained language models. Hence, in this paper, the authors propose a topic entity-aware method for language model pretraining to fuse such information into tokens and mentioned entities.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Jiacheng Li, Yannis Katsis, Tyler Baldwin, Ho-Cheol Kim, Andrew Bartko, Julian McAuley, and Chun-Nan Hsu. 2022. SPOT: Knowledge-Enhanced Language Representations for Information Extraction. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management (CIKM '22). Association for Computing Machinery, New York, NY, USA"}}]

Author rebuttal:

Meta-review: **Strengths**:

1. The paper adds to line of work that improves PLMs for entity-centric data by also including topic awareness. The model design is novel. 

2. Model is evaluated thoroughly on two representative KEPLMs, and with four entity-centric tasks.

3. Paper is written well. Code is released. Authors promise to release model checkpoints.


**Weaknesses**:


1. Gains in performance are quite small yet costly to achieve. Without a significance test, it is hard to say that the proposed approach outperforms significantly the approaches available in the literature.

2. Motivation for hope of significant gains by inclusion of topic entities is weak. Just adding topic entities seems incremental.

**Suggestions**:


1. Please add discussion on added pre-training overhead comparison to the main draft.

2. Please also include the LUKE-base+KÉPLET w/o Entity Identification and LUKE-base+KÉPLET w/o Entity Fusion ablations to the main draft.

3. Please add extra references as pointed out by reviewers.
Decision: Accept-Findings