Coherent Entity Disambiguation via Modeling Topic and Categorical Dependency
Authors: Zilin Xiao
Linjun Shou
Xingyao Zhang
Jie Wu
MING GONG
Daxin Jiang
Venue: EMNLP 2023 Findings (2023)
Keywords: Entity disambiguation
Knowledge base
Entity linking
TL;DR: Improving entity disambiguation with step-by-step neural-based topic guidance and category interaction.

Abstract: Previous entity disambiguation (ED) methods adopt a discriminative paradigm, where prediction is made based on matching scores between mention context and candidate entities using length-limited encoders. 
However, these methods often struggle to capture explicit discourse-level dependencies, resulting in incoherent predictions at the abstract level (e.g. topic or category).
We propose CoherentED, an ED system equipped with novel designs aimed at enhancing the coherence of entity predictions.
Our method first introduces an unsupervised variational autoencoder (VAE) to extract latent topic vectors of context sentences.
This approach not only allows the encoder to handle longer documents more effectively, conserves valuable input space, but also keeps a topic-level coherence.
Additionally, we incorporate an external category memory, enabling the system to retrieve relevant categories for undecided mentions.
By employing step-by-step entity decisions, 
this design facilitates the modeling of entity-entity interactions, thereby maintaining maximum coherence at the category level. 
We achieve new state-of-the-art results on popular ED benchmarks, with an average improvement of 1.3 F1 points. Our model demonstrates particularly outstanding performance on challenging long-text scenarios.

Coherent Entity Disambiguation via Modeling Topic and Categorical Dependency
Zilin Xiao
Linjun Shou
Xingyao Zhang*
Jie Wu
Ming Gong Daxin Jiang*
Rice University Microsoft STCA*
zilin@rice.ede
(lisho, xingyaozhang, jiewul, migon, jiiaan)emicroooft.com
Abstract
Then simple vector dot product is used capture the alignment between mentions and candidate en- tities. While this method enables quick maximum inner product search (MIPS) over all candidates and efficiently determines the linked answer, it suf- fers from late and simplistic interaction between mentions and entities (Barba et al., 2022; Cao et al.. 2021). Recently, researchers have proposed alter- native paradigms for solving the ED problem, such as formulating it as span extraction task (Barba et al., 2022). this approach, Longformer (Belt- agy et al., 2020) is fine-tuned to predict the entity answer span within a long sequence consisting of the document and candidate entity identifiers. An- other paradigm (Cao et al., 2021; De Cao et al., 2022) reduces the ED task to auto-regressive style in which generation models are trained produce entity identifiers token-by-token.
Previous entity disambiguation (ED) methods adopt discriminative paradigm, where predic- tion is made based matching scores between mention context and candidate entities using length-limites encoders. However, these meth- ods often struggle capture explicit discourse- level dependencies, resulting in incoherent pre- dictions at the abstract level (e.g. topic or ED
egory). We propose COHERENTED, system equipped with novel designs aimed at enhancing the coherence of entity predictions. Our method first introduces unsupervised variational autoencoder (VAE) to extract la- tent topic vectors of context sentences. This approach not only allows the encoder to han- dle longer documents more effectively, con- serves valuable input space, but also keeps topic-level coherence. Additionally, we incor- porate external category memory, enabling the system to retrieve relevant categories for undecided mentions. By employing step-by- step entity decisions, this design facilitates the modeling of entity-entity interactions, thereby maintaining maximum coherence at the cate- gory level. We achieve new state-of-the-art results on popular ED benchmarks, with an average improvement of 1.3 Fl points. Our model demonstrates particularly outstanding performance on challenging long-text scenar- ios.
Although these approaches offer some mitiga- tion for the late-and-simple interaction problem, they still exhibit certain vulnerabilities. For in- stance, Transformer-base encoders impose inher- ent limitations on input length, preventing the cap- ture of long-range dependency for specific men- tions. Also, these methods do not explicitly con- sider coherence constraints, while coherence is con- sidered important context in early ED works (Hoffart et al., 2011; Chisholm and Hachey, 2015). We first propose to condition the model on com- pressed topic tokens, enabling the system to sustain topic coherence at the document level.
1 Introduction
Entity disambiguation (ED) is a typical knowledge- intensive task of resolving mentions in a document to their corresponding entities in a knowledge base (KB), e.g. Wikipedia. This task is of great im- portance due to its active presence in downstream tasks such information extraction (Hoffart et al., 201 1), question answering (Yih al., 2015) and web search query (Blanco al., 2015).
In addition, the relationship among entities holds significant importance in the ED task. For example, mentions in the document exhibit a high correla- tion at the category level, where we name it cat- egory coherence. However, previous bi-encoder and cross-encoder solutions have overlooked these entity dependencies and focused solely on learn- ing contextualized representations. Among other works, extractive paradigm (Barba et al., 2022) neglects entity-entity relation as well; generative
To perform efficient entity disambigus gunubiiggatinn (ED),
one common approach is to encode mentions and candidate entities into different embedding spaces.

---

EL (Cao al., 2021; De Cao al., 2022) do pos- sess some dependencies when linking an unknown mention. However, these dependencíes arise from the auto-regressive decoding process and require heavy inference compute.
2 Related Works
Entity disambiguation (ED) is task of determin- ing entities for unknown mention spans within the document. Early ED works (Hoffmann et al., 201 Daiber et al., 2013) commonly rely on matching scores between mention contexts and entities, dis- regarding the showledge-intensivy nature of ED. Many studies aim to infuse external knowledge into ED, Bunescu and Paşca (2006) begin to utilize hyperlinks in Wikipedia to supervise disambigua- tion. Yamada al. (2020, 2022) propose massive pre-training on paired text and entity tokens to im- plicitly inject knowledge for disambiguation usage. Li et al. (2022) first leverage knowledge graphs to enhance ED performance.
To address the above coherence problem, we propose two orthogonal solutions that target topic coherence and entity coherence, respectively. Fol- lowing previous works that decode masked tokens to link unknown mentions (Yamada et al., 2020, 2022), we present the overview of our coherent entity disambiguation work in Figure 1, where doc- ument words and unresolved entities are treated as input tokens of Transformer (Vaswani et al., 2017). First, we bring an unsupervised variational auto-encoder (VAE) (Kingma and Welling, 2014) to extract topic embeddings of surrounding sen- tences, which are later utilized to guide entity pre- diction. By docking these two representative lan- guage learners, BERT (Devlin et al., 2019) and GPT-2 (Radford et al., 2019), the variational en- coder can produce topic tokens of sentenees with- out training on labeled datasets. This approach promotes a higher level of coherence model pre- dictions from an abstract level (Li et al., 2020) (e.g. tense, topic, sentiment).
Another intriguing feature for entities to differ- entiate from each other is their type, as entities with similar surface forms (text identifiers) often pos- sess different types. Onoe and Durrett (2020) dis- ambiguate similar entities solely through a refined entity type system derived from Wikipedia, without using any external information. Ayoola et al. (2022) augment the robustness of such a type system even further. Furthermore, many researchers have ex- plored new paradigms for ED. Cao et al. (2021) propose using prefix-constraieed dictionary on casual language models to correctly generate en- tity strings. Barba et al. (2022) recast disambigua- tion as a task of machine reading comprehension (MRC), where the model selects a predicted entity based on the context fused with candidate identi- fiers and the document.
Moreover, in most KBs, categories serve as valu- able sources of knowledge, grouping entities based on similar subjects. To enhance entity-entity co- herence from a categorical perspective, we design a novel category memory bank for intermediate entity representations to query dynamically. As opposed to retrieving from a frozen memory layer, we introduce direct supervision from ground-truth category labels during pre-training. This enables the memory to be learned effectively even from random initialization.
The architecture of our system seamlessly blends the advantages of type systems and knowledge pre- training: incorporating type systems as dynami- cally updating neural blocks within the model, our design enables simultancous learning through a multi-task learning schema facilitating topic vari- ation learning, masked disambiguation learning. and knowledge pre-training concurrently.
Named COHERENTED, experimental results show that our proposed methods surpass previous state-of-the-art peers on six popular ED datasets by 1.3 FI points on average. Notably, on the chal- lenging CWEB dataset, which has an average docu- ment length of 1,700 words, our approach elevates the score from the previous neural-based SOTA of 78.9 to 81.1. Through model ablations, we verify the effectiveness of the two orthogonal solutions through both quantitative performance evaluation and visualization analysis. These ablations further affirm the superiority of our methods in generating coherent disambiguation predictions.
Prompt Compression a commonly used tech- nique in language models to economize input space, closely related to topics such as prompt compres- sion (Wingate et al., 2022) and context distilla- tion (Snell et al., 2022). Their mutual goal is to dynamically generate soft prompt tokens that place original tokens without hurting downstream application performance. Our topic token design mirrors context compression to some extent but differs in the compression ratio and purpose. Our design is more compact; each context sentence gets

---

Word/ Entity Embedding
Tied Weight
prob
FIFA world cup
Token Type Embedding
Candidate Mask
entty vocab
Position Embedding
Entity Decoder
Transformer
TCCS))
T mess
zai
T
played
Tin
T the
T Tworts
T
T [SEP]
Linnel Mess
E. MASKI
word
word
word
wordd
Cword
woord
word
word
Cword
Cantity
Centity
Po
33
P4
Ps
Ps
P
Pa
E, /
Eg /
[CLS]
meas
##1
played in the
world
cup
[SEP]
Ini (ineel Meas
[MASK]
Document Input
Entity Input
Figure 1: Overview of our baseline model. Legends are presented the upper left corner. Position embedding for entity is averaged on corresponding document positions, for example, E₁ Eg 2 indicates the entity "Lionel_ _Messi" is predicted based on document tokens mess and ##i at position and 2.
converted into single topic token using vari- ational encoder, and in addition to saving input space, it also retains high-level semantics to guide a more coherent ED.
Type embedding is for discrimination usage. There three types of tokens available, each of which corresponds to a dedicated group of parame- ters, Cword Centity Cropic-
Position embedding marks the position of in- put words and entities, avoiding the permutation- invariant property of the oeff-attention mechanism. The entity position embedding also indicates which word tokens the entity corresponds to. This is achieved by applying absolute position embedding to both words and entities. a mention consists of multiple tokens, the entity position embedding is averaged over corresponding positions.
3 Methodology
3.1 Entity Disambiguation Definition
Let X be document with N mentions [m1, m2, mN). where each mentions m, is associated with a set entity candidates C; = [ea, C22 (ca,,02 -]cc]]- Given a KB with set of triplets G = {(h,r, r, rr,,),, EE e R), where h, r and t denote the head entity, relation and tail en- tity respectively, the goal of entity disambiguation is to link mentions m, to one of the corresponding mention candidates C C E.
3.3 Topic Variational Autoencoder
To preserve maximum topic coherence and opti- mize input space utilization, we introduce an ex- ternal component that facilitates topic-level guid- ance in ED prediction. Among various latent vari- able models, variational autoencoders (VAEs) have demonstrated success in modeling high-level syn- tactic latent factors such style and topic. Our setup is based on the motivation that, after being trained massive corpus using the variational objective, the encoder gains the capacity encode sentences into topie latent vectors. The entire topic VAE is composed of two parts, BERT and GPT- 2, both of which are powerful Transformer- language encoder and decoder.
3.2 Overview
We present the overview of COHERENTED in Fig- ure 1. Following Yamada et al. (2020), both words in the document and entities are considered input tokens for the BERT model. The final input repre- sentation sums over the following embeddings:
Representation embedding denotes the topic latent, word embedding or entity embedding for topic inputs, document inputs entity inputs ac- cordingly. We set up two separate embedding lay- ers for word and entity input respectively. X xH denotes the word embedding matrices and Y E TR Vexi denotes the entity embedding matri- ces, where Vi and Ve represents the size of word vocabulary and entity vocabulary.
V..
3.3.1 Encoder
Given BERT encoder LM, and input sentence token sequence æ, we collect the aggregated em-

---

SIAA wodd
fp Easess Nooga (x 2)]
(L m))))
AE Loss
Entity Decoder
Transformer
GPT-2
litent
[cus]
T
Tami
wordd
Tuu
[SEP]
EjoneL Meso!
[MASKI
VAE Loss
Capir
word
word
word
word
Cord
Cwnrd
Cyntit
Cartiy
We
P
P
E, E
E, /
*BERT
topic tokens
CL.S)
V#
world
cup
[SEP]
Mooee! Meeeet
x
Document: Input
Entity Input
Figure2 2: Topic variational autoencoder injection illustration. K topic sentences are converted into topic tokens
appending on the start of model's input.
bedding X from the last layer hidden states corre- sponding to the [CLS] token: X = (CL) With this, we can construct multivariate Gaus- sian distribution from which the decoder will draw samples. The following formula describes the vari- ational distribution for the approximation of the posterior:
tokens. Specifically, our decoder generates auto- regressively via:
T
pe(x 22)
= pe (xt
II t=1
I<t z).
(3)
As stated above Part 3.3.1, the intractable pos- terior for 2n is approximated by de (=n Øznn * <n) in Equation 1. Now we see the difference in the VAE decoder: the generation relies on high-level semantics and has the ability to produce compact representation.
qs (Zn Dgni zen)
(I)
N (zn fug (In) fo (In)),
where fug and for denote separate linear layers for mean and variance representations, / denotes Gaussian distribution and denotes intermediate information bottleneck.
3.3.3 ELBO Training
Both the encoder and decoder need be trained optimize their parameters. Supported by the above approximation, the entire training objective can be interpreted as evidence lower bound objective (ELBO):
3.3.2 Decoder
Given a GPT-2 decoder LMg, we first review how to generate text sequence of length T using such neural models. To generate word tokens of length L, [T1 .2, .......... ITT, language decoder uti- lizes all its parameters 0 to predict the next token conditioned all previous tokens generated Fitts formulated as follows:
log PogPooa)) 2 LELBO Byy(=(a) [logre( (o z)] KL (ap(= #)]lp(z))
Detailed derivations are emitted for clear idea depiction. In practice, we apply the re- parametrization trick (Kingma and Welling, 2014) to allow back-propagation through all deterministic nodes and for efficient learning.
p(x)= I po (at xec).
When training language generator alone, the decoder is usually learned using the maximum like- lihood estimate (MLE) objective. However, in our VAE setting, the decoder conditions on the vec- tor z dynamically drew from a Gaussian distribu- tion, instead of purely from previously generated
Intuitively, consider the first term auto- encoder objective, since it requires the model do reconstruction based on the intermediate latent. The second term defines the KI. divergence be- tween the real distribution (s x) and p(z). To better implement these objectives, we refer to

---

the regularized version of ELBO (Li et al., 2020),
where ELBO is considered the linear combina- tion of reconstruction error and KL regularizer:
Haalegory WB
Oij C
(6)
Evariational = LE + BLn, with
where Cij sigmoid (E (Emusked) denotes matching score between i-th masked entity and j- th category. WB is linear projection layer for dimension matching. During training, apply direct supervision from gold category guidance via binary cross-entropy loss:
LE = ( [logpa (a =)] (4)
CR KL (go(= x)))
Finally, we treat the loss term variiaiionl as one of our minimization objectives in the following multi-takk learning.
|0
1 |C|
3.4 Category Memory
Leategory
aij
Loracles
(7)
We now formally define the category memory layer inserted into the intermediate Transformer layers. Let E = {e1, .,..... e. be the set of all possible candidate entities, and correspondingly, each entity e, has a predefined set of categories Ce.- The union set of category sets of all candidate entities will be C. Based on this motivation, we construct a category vocabulary and a category embedding ta- ble C E RlC|xdangyy The vocabulary establishes mapping from textual category labels to indices. As sometimes category labels can be too fine-gra fineggrined, we refer readers to Appendix A for the detailed design of category system.
where Loracle denotes the indicator function of the oracle category labels for the i-th masked entity,
3.5 Multi-task Pre-training
This part discusses the pre-training stage for Co- HERENTED. First, we define the disambiguation loss, which is analogous to the well-known masked language modeling objective. In each training step, 30% of the entity tokens are replaced with special [MASK] token. We employ a linear decoder at the end our model to reconstruct the masked tokens, as shown in Equation 8.
The embedding table C stores category repre- sentations that can be updated during massive pre- training. To be specific, we first formulate our model's s forward (Figure 3) as follows:
È softmax WD E2 Emasked + bD). (8)
Equation 9 represents the cross entropy loss the entity vocabulary, where Ieg denotes the indi- cator function of the k-th masked entity's 's ground- truth.
T' W E' - TransformerN (T", W w",
H CategoryMemorry (E*).
LayerNorm (H+
2 2 E Transformery (T', w'),,,
Nmastedd
Cdismbiiguation
È Iec.
(9)
where symbols T. W. stand for hidden states of topics, words and entities respectively.
Nmaaked
By incorporating all these losses, we derive our final multi-task learning objective being:
Each unresolved entity in the document is as- signed with [MASK] token in the input sequence. During the forward pass, all intermediate entity rep- resentations corresponding to [MASK] token e'nasked are projected from deaitty Rd deaaggor using linear layer without bias terms:
C Cdisambiguation + C.eateerrryyttt (10)
where coefficients a and control relative impor- tance of two auxiliary tasks.
Ei Emaaseed = WA A
maskeed
(5)
3.6 COHERENTED Inference
Subsequently, the adapted intermediate entity representations will query all entries in the category embedding table. The aggregated weighted hidden states are computed as follows:
Given a document X with N mentions, M {m1, M2,..., my). we now describe the coherent ED inference process.

---

Train
Inference
Quadrenebe sporting events
Hil Cantury Cub
0.53
a' known mention is assigned oracle category weight
BCELON
Association football events
FITA competitioon
Tixk:
0.53
0.89
Intermediate Contextualized Entity Representation
Entity Decoder
%
Transformer Layers
We
June sporting events
K,V
Q
Hgus
Hinss
Has
World
Hup
Hysey
Honnet
Hyuase
W₂
Transformer Layers x M
Tooss
Tux
Tworls
Toup
Tiar
Fuord Mos
FINUSSK)
CL91
mesa
#41
world
cup
[SEP]
Monel Noso!
[MA.5K]
Category Memory
Figure 3: Category memory illustration. The difference between training and inference depicted using dotted colored box.
Considering a Transformer with an input length limit being tokens, we reserve l tokens for topic latent, n. for shallow entity representation input, leaving the word input window being L - k ne tokens. Note that ne indicates the number of men- tions in certain sentence and varies among differ- ent training batches, so we set ne to the maximum number of mentions within the batch. We refer in- terested readers to Appendix C for how sample topic sentences and prepare input tokens.
4 Experiments
4.1 Datasets and Settings
For a fair comparison with previous works, we adopt the exact same settings used by Cao et al. (2021). Specifically, we borrow pre-generated can- didate entity sets from (Le and Titov, 2018). Only entities with the top 30 p(e m) score consid- ered candidates, and failure include the oracle answer in the candidate set leads to false negative prediction. For evaluation metrics, we report InKB micro Fl scores on test splits of AIDA-CoNLL dataset (Hoffart et al.. 201 1) (AIDA), cleaned ver- sion of MSNBC, AQUAINT. ACE2004, WNED- CWEB (CWEB) and SNNEDWWIK (WIKI) (Guo and Barbosa, 2018). For training data, we use the 2022-09-0F Wikipedia dump without any weak pseudo labels utilized in Orr al. (2021); Broscheit (2019). We do not run hyperparameter search due to limited training resources. Note that few previ- ous works use a mixture of AIDA and Wikipedia as training split, which we will indicate in the table caption as unfair setting. Refer to Appendix B for dataset details, and Appendix E for implemen- tation, training and hyperparameter choice.
With all input tokens ready, we predict entities for N steps. Unlike language generator which decodes the next token, at each step the model de- codes all [MASK] tokens into entity predictions by selecting maximum indices the logits. The entity prediction at step is decided using the highest- confidence strategy, i.e., the entity with the highest log probability is resolved, while others have to wait until the next step.
is worth noting that utilizing candidate en- tity information can significantly reduce noisy pre- dictions, indicated by the green bars in Fig- ure 1. During inference in the category memory layer, only top-k category entries are selected for weighted aggregation.
Two variants are proposed for a fair compar- ison with baseline models in terms of the num- ber of Transformer layers. COHERENTED con- tains 3, 3, and 6 layers for the VAE encoder, decoder and base model respectively, while in COHERENTED these numbers become 6, 6 and 12. Both variants are with a comparable number of parameters with Yamada et al. (2022), namely 10M for the base variant and 440M for the large one.
Once an entity prediction is determined, the cat- egory memory ceases to be queried at that position and instead receives real category indicator to aggregate entries from the category memory. We refer this as oracle category guidanee because it allows for potential category-level guidance in disambiguating remaining mentions.

---

Method
AIDA MSNBC
AQUAINT ACE2004 CWEB WIKI
AVG-6 AVG-5
Yosef et al. (2011)
78.0
79.0
56.0
80.0
58.6
63.0
69.1
67.3
van Hulst et al. (2020)
89.4
90.7
84.1
85.3
71.9
73.1
82.4
81.0
Cao et al. (2021)
93.31
94.3
89.9
90.1
77.3
87.4
88.71
87.8
Orr et al. (2021)"
80.9
80.5
74.2
83.6
70.2
76.2
77.6
76.9
Yang al. (2018)
93.0t
92.6
89.9
88.5
81.8
79.2
87.5f
86.4
Barba al. (2022)
92.6
94.7
91.6
91.8
77.7
88.8
89.51
88.9
Ayoola et al. (2022)
87.5
94.4
91.8
91.6
77.8
88.7
88.6
88.8
Yamada et al. (2022)
96.3
93.5
91.9
78.9
89.1
89.9
Our CoherentE.D base
88.2
94.9
93.7
92.3
77.2
87.8
89.0
89.2
Our CoherentED large
89.4
96.3
94.6
93.4
81.1
90.6
90.9
91.2
Model Ablations on CoherentED Jarge
Topic Tokens
88.4
94.6
93.4
92.3
77.9
88.9
89.3
89.4
Category Memory
89.1
95.6
93.8
92.3
79.8
90.1
90.1
90.3
w/o Category Oracle Guidance
89.9
95.2
92.1
93.4
80.2
90.8
90.3
90.3
Table 1: ED InKB miero F1 scores test datasets. The best value is in bold and the second best is in underline. means results come from reproduced results official open-source code. indicates non-comparable metrics due to unfair experimental setting. indicates not reported in the original paper. For direct comparison with Yamada et al. (2022), AVG-5 reports average micro Fl scores all test datasets except AIDA.
4.2
Main Results
Compared with the model without topic token injections, COHERENTED improves greatly from 89.3 to 90.9 in average micro F1 score with a particular gain on the lengthy CWEB dataset from 77.9 to 81.1. Such performance gain on CWEB only falls short of Yang al. (2018) which requires extensive feature engincering targeted at document- level representations. All other test sets also benefit from the powerful abstract modeling ability of topic VAF.
We report peer comparisons in Table 1. Note that we consider the Wikipedia-only training setting, meaning no further fine-tuning or mixture training on AIDA is allowed and all evaluations are out-of- domain (OOD) tests.
In general, we achieve new state-of-the-art re- sults on all test datasets except CWEB and AIDA datasets, surpassing the previous best by 1.3 F1 points and eliminating 9% errors. On the CWEB dataset, our work still shows superiority over other neural-based methods, the lengthy samples are too unfriendly to be understood globally by tive neural encoders. On the ACE2004 dataset, since the number of mentions is relatively small, many reported numbers are identical. On other datasets, the relative improvements are consis- tent for CoHERENTED even though only addi- tional cheap category labels are provided during pre-training. Such improvements also confirm the outstanding OOD ability of our methods since no fine-tuning is conducted on the downstream test datasets.
Compared with the model without category memory layer, the CoherentEDaape improves from 90.1 to 90.9, not as significant as the improvement of topic injections but still notable enough. Note that category memory does not bring consistent per- formance gain all test datasets, possibly because not all test samples are sensitive category-level coherence. Furthermore, we attempt to disable the category oracle guidance strategy during evalua- tion, meaning for predicted entities in step-by-step ED, we still query the external category memory and aggregate the retrieved entries. And ablation shows that the oracle guidance does have positive impact on the overall performance metries.
4.3 Ablation Study
4.4 Case Analysis and Visualization
the lower part of Table 1, we report ablation experiments of proposed methods. All ablations are conducted on COHERENTED large-
Besides ablations on evaluation metrics, we con- duct deeper analysis proposed methods by visualizing data samples. Specifically, we per-

---

To evaluate the topic representation ability of our jointly trained topic VAE, we design an elegant probe strategy to investigate the topic modeling ability of COHERENTED. MSNBC covers 20 doc- uments on 10 topics?. By feeding each sentence in the MSNBC test set along with predicted en- tities tokens, we extract the [CLS] representation of CoherentED and run t-SNE on these joint representations.
Popular Entries
Category Heradhy
In Figure 5, topic latent vectors of sentences in these documents are plotted into 511 data points, whose colors denote their oracle topic labels. We see that the majority of sentences under the same topic cluster into polarized groups, despite few outliers possibly because they are for general pur- poses such greeting and describing facts. Con- sistent with our expectations, similar topics are intertwined as they share high-level semantics some extent.
Figure 4: T-SNE visualization of category emheddings after pre-training. Zoomed part denotes the specific cat- egory labels and corresponding sample points. Category "Stock market indices" is emitted for clear depiction.
form t-SNE on category memory entries after joint training and topic vectors of sentences in MSNBC dataset.
To validate the effectiveness of the learned cate- gory memory layer, we expect the embeddings of category entries stored in the memory to exhibit certain degree of similarity with the category hierar- chy in Wikipedia, i.e., similar category entries close to each other. In Figure 4, we present (-SNE visualization of the top 500 popular category en- tries and additional colored category group, where black cross data points represent popular category entries and blue circular points represent examples of structured hierarchy from the Wikipedia cate- gory system'.
5 Conclusion
We propose a novel entity disambiguation method COHERENTED, which injects latent topic vectors and utilizes cheap category knowledge sources to produce coherent disambiguation predictions. Specifically, introduce unsupervised topic variational auto-encoder and external category memory bank to mitigate inconsistent entity pre- dictions. Experimental results demonstrate the ef- fectiveness of our proposed methods in terms of accuracy and coherence. Analysis of randomly picked cases and vector visualizations further con- firm such effectiveness.
Limitations
Interteined Whrld News 6 U.S. News
Still, our COHERENTED remains with two limita- tions: scalability and performance. Future works are expected to alleviate these limitations. First, COHERENTED can hardly handle emerging enti- ties as this requires extending both the entity em- bedding layer and category memory layer. The evaluation metrics will degrade if no further train- ing is conducted after such expansion. Second, despite the count of parameters and FLOPs of Co- HERENTED being quite comparable with baseline models, the advantage of coherent prediction only reveals itself in the scenario of step-by-step reason- ing, i.e., mentions are resolved one by one. This
Beatth
Tweel U.S.
U.S. Mica
Entertaimmnn:
Figure 5: T-SNE visualization of sentence vectors ex- tracted by the topic probe strategy. Some polarized topic groups such as "Health" and "Sports" " are denoted using colored dashed circles. Zoomed part reveals the intertwined nature between similar topics (e.g. "World News" and "U.S. News"). Best viewed in color.
Business, U.S. Politics, Entertainment, Health, Sports, Tech & Science, Travel, TV News, U.S. News, and World News.
"ccoommic indicators + Stock market indices + Ameri- can stock market indices - Companics in the Nasdaq- 100

---

means multiple forward passes are needed for each document achieve the most accurate results.
Joachim Daiber, Max Jakob, Chris Hokamp, and Pablo N. Mendes. 2013. Improving efficiency and accuracy in multilingual entity extraction. In I- SEMANTICS 2013 9th International Conference on Semantic Systems, I-SEMANTICS '13, Graz, Aus- tria, September 4-6, 2013, pages 121-124. ACM.
References
Tom Ayoola, Shubhi Tyagi, Joseph Fisher, Christos Christodoulopoulos, and Andrea Pierleoni. 2022. Re-
Nicola De Cao, Ledell Wu, Kashyap Popat, Mikel Artelxe, Naman Goyal, Mikhail Plekhanov, Luke Zettlemoyer, Nicola Cancedda, Sebastian Riedel, and Fabio Petroni. 2022. Multilingual autoregressive tity linking. Transactions of the Association for Com- putational Linguistics, 10:274-290.
FinED: An efficient pero-shot-capable approach to end-to-end entity linking. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Track, pages 209- 220. Hybrid: Seattle, Washington + Online. Associa- tion for Computational Linguistics.
Jacob Devlin, Ming-Wai Wei Chang, Kenton Lec, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Pupers), pages 4171-4186 Minneapolis, Minnesota. Association for Computational Linguistics.
Edoardo Barba, Luigi Procopio, and Roberto Navigli. 2022. ExtEnD: Extractive entity disambiguation. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics ( (Volume 1: Long Papers), pages 2478-2488, Dublin, Ireland. As- sociation for Computational Linguistics.
Iz Beltagy, Matthew E, Peters, and Arman Cohan, 2020. Longformer: The long-document transformer. CoRR, abs/2004.05150.
George Doddington, Alexis Mitchell, Mark Przybocki. Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel. 2004. The automatic content extrac- tion (ACE) program tasks, data, and evaluation. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC'04), Lisbon, Portugal. European Language Resources As- sociation (ELRA).
Roi Blanco, Giuseppe Ottaviano, and Edgar Meij. 2015. Fast and space-efficient entity linking for queries. In Proceedings of the Eighth ACM International Con- ference on Web Search and Data Mining, WSDM 2015, Shanghai, China, February 2-6, 2015, pages 179-188 ACM.
Evgeniy Gabrilovich, Michael Ringgaard, and Amarnag Subramanya. 2013. Face1: Freebase annotation of clueweh corpora, version (release date 2013-06-26, format version correction level 0).
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals. An- drew Dai, Rafal Jozefowicz, and Samy Bengio. 2016.
Generating sentences from a continuous space. In Proceedings of the 20th SIGNLL Conference on Com- putational Natural Language Learning, pages 10-21 (1-21, Berlin, Germany. Association for Computational Lin- guistics.
Zhaochen Guo and Denilson Barbosa. 2018 Robust named entity disambiguation with random walks. Se- mantic Web, (4):459-479.
Samuel Broscheit. 2019. Investigating entity knowl- edge in BERT with simple neural end-to-end entity linking. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 685, Hong Kong, China, Association for Computational Linguistics.
Johannes Hoffart, Mohamed Amir Yosef, Ilaria Bordino, Hagen Fürstenau, Manfred Pinkal, Mare Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. 2011. Robust disambiguation of named entities in text. In Proceedings of the 2011 Conference on Em- pirical Methods in Natural Language Processing, pages 782-792, Edinburgh, Scotland, UK. Associa- tion for Computational Linguistics.
Razvan Bunescu and Marius Paşca. 2006. Using en- cyclopedic knowledge for named entity disambigua- tion. In 11th Conference of the Eurppan Chapter of the Association for Computational Linguistics, pages 9-16, Trento, Italy. Association for Computational Linguistics.
Raphael Hoffmann, Congle Zhang. Xiao Ling, Luke Zeuteeoyer, and Daniel S. Weld. 2011. Knowledge- based weak supervision for information extraction of overlapping relations. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 541-550, Portland, Oregon, USA. Association for Computational Linguistics.
Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2021. Autoregressive entity retrieval. In 9th International Conference on Learning Repre- sentations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OvenReview.net.
Diederik Kingma and Max Welling, 2014. Auto- encoding variational bayes. In 2nd International Conference Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Truck Proceedings.
Andrew Chisholm and Ben Hachey. 2015. Entity dis- ambiguation with web links. Transactions of the As- sociation for Computational Linguistics, 3:165-156.

---

Phong Le and Ivan Titov. 2018. Improving entity link- ing by modeling latent relations between mentions. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Pupers), pages 1595-1604, Melbourne, Aus- tralia, Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017, Attention is all you need. In Advances in Neural Information Pro- cessing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Chunyuan Li, Xiang Gao, Yuan Li, Baolin Peng. Xiujun
Li, Yizhe Zhang, and Jianfeng Gao. 2020. Optimus: Organizing sentences via pre-trained modeling of a latent space. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 4678-4699, Online. Association for Computational I Linguistics.
David Wingate, Mohammad Shoeybi, and Taylor Sorensen, 2022. Prompt compression and contrastive conditioning for controllability and toxicity reduction in language models. In Findings of the Association for Computational Linguistics; EMNLP 2022, pages 5621-5634, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Qijia Li, Feng Li, Shuchao Li, Xiaoyu Li, Kang Liu, Qing Liu, and Pengcheng Dong. 2022. Improving en- tity linking by introducing knowledge graph structure information. Applied Sciences, 12(5):2702.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi. Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of
Yasumasa Onoe and Greg Durrett. 2020. Fine-grained entity typing for domain independent entity linking. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innova- tive Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educa- tional Advances in Artificial Intelligence, EAAI 2020, New York, NY. USA, February 7-12, 2020, pages 8576 8583. AAAI Press.
taite-of-theaate tht-hhh-ttt natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. LUKE: Deep contextualized entity representations with entity- aware self-attention. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP),, pages 6442-6654, On- line. Association for Computational Linguisties.
Laurel J. Orr, Megan Leszezynski, Neel Guha, Sen Wu, Simran Arora, Xiao Ling, and Christopher Ré. 2021. Bootleg: Chasing the tail with self-supervised named entity disambiguation. In 1 11th Conference on In- novative Data Systems Research, CIDR 2021, Vir- tual Event, January 11-15, 2021, Online Proceedings. www.cidrdb.orgg
Ikuya Yamada, Koki Washio, Hiroyuki Shindo, and Yuji Matsumoto. 2022. Global entity disambiguation with BERT. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 3264-3277 3271- Seattle, United States. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. 2019. Language models are unsupervised multitask learners. OpenAl blog. 18):9.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimiza- tions enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference edge Discovery &amp; Data Mining, KDD '20, page 3505-3506, New York, NY, USA. Association for Computing Machinery.
Yi Yang, Ozan Irsoy, and Kazi Shefaet Rahman. 2018 Collective entity disambiguation with structured gra- dient tree boosting. In Proceedings of the 2018 Con- ference of the North American Chapter of the Asso- ciution for Computational Linguistics: Human Lan- guage Technologies, Volume I (Long Papers), pages 777-786, New Orleans. Louisiana. Association for Computational Linguistics.
Knowl-
Charlie Snell, Dan Klein, and Ruiqi Zhong. 2022. Learning by distilling context. CoRR, abs/2209.15188
Wen-tau Yih, Ming- Wei Chang, Xiaodong He, and Jian- feng Gao. 2015. Semantic parsing via staged query graph generation: Question answering with knowl- edge base. In Proceedings of the 53rd Annual Meet- ing of the Association for Computational Linguistics and the 7th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 1321-1331, Beijing, China. Association for Computational Linguistics.
Johannes M. van Hulst, Facgheh Hasibi, Koen Dercksen, Krisztian Balog, and Arjen P. de Vries. 2020. REL: an entity linker standing on the shoulders of giants. In Proceedings of the 43rd International ACM SIGIR conference research and development in Infor- mation Retrieval, SIGIR 2020, Virtual Event. China. July 25-30, 2020, pages 2197-2200. ACM.

---

Mohamed Amir Yosef, Johannes Hoffart, Ilaria Bor- dino, Mare Spaniol, and Gerhard Weikum. 2011. AIDA: an online tool for accurate disambiguation of named entities in text and tables. Proc. VLDB Endow. 15:00144544
A Category System Design
Sometimes category labels can be too fine-grained, e.g. Apple Inc. is with one category label of Computer companies established in 1976, which is less informative compared with two separate la- bels Computer companies and in 1976. To mitigate this issue, we extend category construction meth- ods from Onoe and Durrett (2020), where preposi- tions are considered as stop words to disassemble original category labels. We further ignore dif- ferences among prepositions, i.e., of the United States and in the United States considered be indistinguishable and unified into [PERP] the United States. This helps simplify the category labels and ensures that the model focuses on the relevant semantic information rather than specific prepositions.
B Evaluation Dataset Details
The brief dataset descriptions follows:
AIDA contains 18, 448 training samples, 4791 validation samples and 4485 test samples. It also served as one of the largest manually an- notated EL and ED datasets. Note that other datasets contain test split only.
2. MSNBC is news corpus with 20 documents and 656 mentions. Despite its small scale, MSNBC covers 10 obvious topics and thus acts as a perfect testbed for our topic VAE approach.
3. AQUAINT is another news corpus containing 50 documents from Xinhua News, the New York Times and the Associated Press, cover- ing 727 samples.
4. ACE2004 is manually annotated subset of Doddington et al. (2004) containing 257 sam- ples.
CWEB is an automatically constructed dataset from ClueWeb corpus* in Guo and Barbosa (2018), containing 11,154 samples. Is worth mentioning that most ED works per- form similarly on CWEB, partially because the average length of documents in CWEB is
2 SSpecifically, prepositions refer those are frequently used in category such as 'in', 'from', 'for', 'of'. 'by" 'for.. 'involving" whttps:: htttp::///emurrojjeec.t wttp:c/www//b//////ooooo

---

significantly longer than others. With approxi- mately 1,700 words on average per document, none of the trivial BERT-based models can handle the entire document. Yang et al. (2018) got relatively better performance on it as their models involve heavy hand-crafted features designed to capture document-level semantics. We later show that our topic worming-basees methods achieve similar performance with minimal human involvement.
Bootleg (Orr et al., 2021) focuses on model- ing reasoning patterns for disambiguation in a self-supervised manner. Tail entities who rarely appear in KB and documents are espe- cially investigated in this work.
BiBSG (Yang al., 2018) is the first in- troduce the structured gradient tree boosting (SGTB) algorithm to collective entity disam- biguation with many efforts making use of global information from both the past and future to perform better local search.
6. WIKI is another automatically extracted test dataset from Gabrilovich et al. (2013), cover- ing 6,821 samples.
ExtEND (Barba al., 2022) formulates the ED problem into a span extraction task sup- ported by a Longformer model that predicts entity span in the input sequence.
C Preparing Input Tokens for COHERENTED
ReFinED (Ayoola et al., 2022) is an efficient zero-shot end-to-end entity linker using score- based bi-encoder architecture, which seeks a trade-off between performance and efficiency.
First, the document is tokenized into LD text to- kens and split into sentences. If text tokens fit in the word input window (i.e. I.p S I. k ne), we uti- lize all word tokens and sample & topic sentences uniformly. Otherwise, with the sentence be disambiguated as the cen- ter5, and prioritize sampling & sentences outside the trimming range as the topic sentences. Selected sentences get encoded through pre-trained topic encoder and prepend their topic representations on the input sequence. Note that the topic decoder is no longer needed as it only supports variational learning in the auxiliary branch. Lastly, N [MASK] tokens are appended to the sequence, indicating all N mentions are unresolved. For training sam- ples where < Ne, we concatenate more ne [PAD] tokens.
truncate the document
GlobalED (Yamada al.. 2022) considers ED masked token prediction problem and is also the baseline our work.
E Implementation, Training and Hyperparameters
Hyperparameter
Value
learning rate (stage 1) learning rate (stage weight Boight_decay batch size per device effective batch size learning rate strategy optimizer
5e-4
5e-5
le-2
4
2048
D Brief Introductions of Peers
WarmupDecayl.R
AIDA (Yosef et al., 2011) is traditional framework and online tool for entity detec- tion and disambiguation.
AdamW
dropout
0.1
gradient clipping
1.0
REL (van Hulst et al., 2020) is modern open- source toolkit for entity linking equipped with customized deep models.
Table Hyperparameters used for training COHER- ENTED
We use the Huggingface (Wolf et al., 2020) version of Transformer as the codebase. A to- tal of 127,314 entities are considered in entity vocabulary and entity prediction head, resulting in a category vocabulary of size 391, 234. Deep- Speed (Rasley et al., 2020) is used for maximum hardware utilization and parallel training manage- ment. Table presents most of the hyperparam- eters in training. Due to the limited computation
GENRE (Cao et al., 2021) is the first to for- mulate entity linking and disambiguation into the constrained text generation task via pre- defined trie. Auto-regressive decoding nature makes it hard for real-time usage.
'Different sampling strategies bring negligible improve- ment. As such, we will not discuss performance differences caused by sampling.

---

Topic: Economics
There were more declining shares than adv anoenss on the New York Stock Exchange and the Nasdag Stock Market ... Investos are looking for signs that consumer spending, one of the biggest drivers of the U.S. economy, will recover during the holidays. A report on industrial production weighed on the market The Fed suid output at the nation's factories, mines and utilities rose 0.1 percent in October, less than the 0.4 percemt predicted by economists polled by Thomson Reulers...
The Federal Reserve
Bank regulation United States, Central Banks...
The Fed (newspaper)
The Federalist Papers
The Federation (group)
Musical groups established in 2002, Hip hop groups from California...
Topic: Music
The song induced riot when The Fed performed "Hypby" during halftime of the ANDI Live Tour at Oracle Arena 2004. On the strength of "Hyphy" and their second single "Donkey", the group's self self-titldd debut album was released under Virgin Records to crtical reception...
Figure ED case analysis revealing two critical motivations of our work, topic coherence and categorical coherence. Mentions annotated with gray background. Two documents share the mention "The Fed". The document in the upper part is centered around the economics topic while the lower one elaborates on the music topic. The middle part lists four entity candidates for the mention "The Fed", with some corresponding category labels on yellow background.
resource, we do not run massive hyperparameter searches. Multi-task coefficients are set to a 0.1 and = 10 with a few empirical trials done. Dur- ing inference, the k in Top-K K category retrieval is set to 10, as the average number of categories of all entities present is close to this number.
among entities and focus on learning representa- tions alone.
Consider the upper document in Figure 6, where three mentions are highlighted with a gray back- ground. The correct linked entity for the mention "New York Stock Exchange" shares the exact cate- gory "Stock exchanges in the United States" with the correct linked entity for "Nasdaq Stock Mar- ket." " Moreover, these two mentions can implic- itly guide entity prediction for "The Fed" due to the high correlation between their respective cate- gories.
We train our proposed model in two stages. In stage I, we freeze all the parameters except for the fresh entity embedding layer and category memory layer. Consequently, the variational objective is disabled in stage 1. Then after I epoch, we activate all parameters and enable three objectives in stage 2, which lasts for 6 epochs. Pre-training VAE be diffieult due to the notorious KI vanishing issue (Bowman et al., 2016), causing the decoder completely ignores the topic latent in learning. As a practical solution to mitigate this, a cyclical schedule is applied to the KL. regularizer coeffi- cient 3. The training takes approximately day for COHERENTED hase and 3 days COHERENTED large on 8 A100-SXM4-40GB GPUs.
F
Case Study
Figure 6, we illustrate document samples ex- tracted from the MSNBC test dataset, wherein the mention "The Fed" can be readily disambiguated if provided with the corresponding topic.
Besides topic coherence, the relationship among entities also matters in the ED task. In the upper Figure 6, mentions are highly correlated in their category level, and previous bi-encoder and cross- encoder solutions totally ignore the dependencies


Reviews:
[{"review_id": "KMUZH0B7Ww", "reviewer": "Reviewer_LkD8", "summary": "", "questions": "1. Does the semantics of topic and category partly overlap?\n2. How does the number of topic tokens impact the overall performance?\n2. In the VAE, what is the intuition of using BERT as the encoder and GPT2 as the decoder? Can an encoder-decoder pre-trained model work?\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Firstly, we appreciate the time and effort you invested in reviewing our work. We have thoroughly considered each comment and would like to address the main concerns raised.\n\n### Unclear Motivation\n\nWe would like to emphasize that the Case Study is presented in Lines 926-946 of Appendix F. This study provides an example from our test dataset to clarify the motivation behind our model's design, which is to focus on topic and category coherence. We appreciate that Reviewer AHEW also values this. However, as the case study only includes predictions of our work, more intuitive side-by-side comparisons will be added to the future manuscript to show differences.\n\nIt's worth noting that the method of Generative Entity Linking inherently incorporates dependencies on generated entities during the autoregressive process by attending to their text tokens of entity identifiers. This is fundamentally distinct from our CoherentED design, where **entities and categories are treated as independent and discrete elements** with no entity titles involved. \n\n### The Source of Performance Improvement\n\nWe acknowledge your concerns regarding the source of performance improvement. Indeed, Figures 4 and 5 can only demonstrate the representation learning capabilities of the model for topics and categories at the level of **embedding similarity**. We would like to point out that it is less likely for the number of model parameters to be the source of performance improvement, as we have discussed in Line 431-436.\n\nFor the future manuscript, we will make the following adjustments:\n\n1. A comparative Case Study and Error Analysis between `CoherentED` and previous works will be conducted, with special attention given to scenarios where `CoherentED `captures dependencies at the topic and category levels.\n2. We plan to employ automated tools to design a category-level coherence evaluation metric for popular ED benchmarks, and then compare the baselines and `CoherentED`'s performance on category coherence. Specifically, leveraging the category system established in Appendix A, we can identify sets of categories that belong to known entities in the benchmark. The extent to which these sets overlap provides a measure of a document's sensitivity to category coherence. In this manner, `CoherentED`'s capability to capture category coherence can be quantified by comparing the F1 scores across separated test sets that group by category sensitivity.\n3. We will utilize commonly used evaluation methods in topic modeling, such as Latent Dirichlet Allocation (LDA), to assess topic coherence. By comparing the UMass Coherence Score between sentence encoders trained with and without VAE, we can explore the strong correlations between real topic labels and topic tokens, on which the later model will condition.\n\n### Case Study of Long-text Scenarios\n\nWe understand your concerns regarding the lack of persuasive evidence for CoherentED's performance in long-text scenarios. In addition to our exceptional performance on CWEB, where the average document length exceeds 1700 tokens, we plan to report performance metrics for other benchmark subsets grouped by token length in future manuscripts. Corresponding case studies will also be added to Appendix F for further scrutiny.\n\n### Responses for Questions\n\nQuestion 1: Does the semantics of topic and category partly overlap?\n\nAnswer 1: \n\nIntuitively, topics and categories are likely to have overlapping semantics, as documents discussing similar topics are prone to containing entities with similar category labels. Due to the orthogonal designs in CoherentED, it becomes challenging to experimentally explore this aspect beyond what has been discussed in the section on 'The Source of Performance Improvement'. Investigating the interpretability of CoherentED will be an intriguing avenue for future work.\n\n---\n\nQuestion 2: 1. How does the number of topic tokens impact the overall performance?\n\nAnswer 2: \n\nThe number of topic tokens we are using as the model’s prefix input is $k=8$. As the training took longer than the rebuttal period with our current compute resources, we are not able to give ablations only on $k$. Instead, we report our evaluation results from early experiment records in the table below. \n\nNote that hyperparameters $k$, $\\alpha$ (loss coefficient for variational objective), and $\\gamma$ (loss coefficient for category objective) are chosen using a held-out validation set from Wikipedia dump. Also early records **might not be reproducible** as the model went through changes over time.\n\n| k | $ \\alpha $ | $ \\gamma $ | AVG-6 |\n| --- | --- | --- | --- |\n| 8 | 0.1 | 10 | 90.9 |\n| 4 | 1e-2 | 1 | 89.9 |\n| 16 | 0.1 | 10 | 90.2 |\n\n---\n\nQuestion 3: 1. In the VAE, what is the intuition of using BERT as the encoder and GPT2 as the decoder? Can an encoder-decoder pre-trained model work?\n\nAnswer 3: \n\nThe motivation of docking BERT and GPT-2 is to create an information bottleneck and expect the intermediate latent would carry light and compact representations for the main model to condition on. As Reviewer AHEW points out, this intuition originally comes from Optimus [1] which runs large-scale variational pre-training on corpus. Our work focuses more on producing sentence-level (k uniformly-sampled sentences) topic representations that could support specific usage.\n\nAn encoder-decoder pre-trained model might work in a variational way with some dimension adaption and time-step aggregation layers between the encoder and decoder. \n\n---\nThank you again for your valuable feedback and we hope that our responses clarify any ambiguities.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The two technical design is reasonable and presented clearly. \n2. The experiments show the method's effectiveness, and the ablation study verifies that each key component contributes to the performance improvement.\n", "reasons_to_reject": "1. The motivation is somewhat unclear to me. Generally, the authors claim that previous efforts can not handle entity interaction regarding topic coherence and category coherence. The authors can provide more intuitive examples by comparing existing methods, e.g., generative entity linking techniques.\n\n2. The performance improvements are evident, but whether they are from better topic and category coherence is not verified. Figures 4 and 5's visualization just prove that the same or similar topics/categories will generate grouped embeddings, and more in-depth analyses are needed from the perspective of coherence.\n\n3. The authors claim their model demonstrates outstanding performance in challenging long-text scenarios. More experiments and discussion on the context of different lengths can make this claim more convincing.\n", "questions_to_authors": "1. Does the semantics of topic and category partly overlap?\n2. How does the number of topic tokens impact the overall performance?\n2. In the VAE, what is the intuition of using BERT as the encoder and GPT2 as the decoder? Can an encoder-decoder pre-trained model work?\n", "paper_topic_and_main_contributions": "This paper proposes an event disambiguation method equipped with two designs to enhance entity predictions' coherence. The authors use an unsupervised variational autoencoder (VAE) to extract latent topic vectors of context sentences, which are then used as extra inputs. Meanwhile, they incorporate an external category memory, enabling the method to retrieve relevant categories for undecided mentions, hence improving coherence at the category level. The method achieves competitive performance on popular ED benchmarks.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tztzQhr9xH", "reviewer": "Reviewer_Wj22", "summary": "", "questions": "q1: from eq1, it seems that z is a sequence of hidden vector z1...zn, and thus each sentence is encoded in a sequence of hidden zs rather than only one vector z. But it contradicts with line 250. What is true here? \n\nq2: Eqs in sec 3.4 show that E2 is computed by a LayerNorm and a Transformer. How could it be? \n\nq3: Line 404, it is quite unclear what \"oracle category guidance\" is. Could the authors explain it in a clearer way? \n\nq5: Line 848, the model sample k sentences and \"different sampling strategies bring negligible improvement\". Why is sampling still used?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their insightful comments and valuable suggestions. We address each of the concerns below, with particular emphasis on fixing the readability issue.\n\n### Readability Issue\n\nq1: from eq1, it seems that z is a sequence of hidden vector z1...zn, and thus each sentence is encoded in a sequence of hidden zs rather than only one vector z. But it contradicts with line 250. What is true here?\n\na1: The true case is that $k$ topic sentences are encoded into topic representations that fit onto the start of the main model’s input as soft prompt tokens. We apologize for the mistake made in Section 3.3.1, where `input token sequence` confuses readers a lot. A more appropriate term would be `tokens of a topic sentence`.\n\n---\n\nq2: Eqs in sec 3.4 show that E2 is computed by a LayerNorm and a Transformer. How could it be?\n\na2: \n\nWe apologize for the indexing error in Equation 3.4. The corrected third and fourth lines of Equation Group 3.4 are as follows, where $ \\mathbf{E}^{1 \\prime} $ represents the intermediate entity hidden states after aggregating category memory:\n\n$ \\mathbf{E}^{1 \\prime}=\\operatorname{LayerNorm}\\left(\\mathbf{H}+\\mathbf{E}^1\\right) $\n\n$ \\mathbf{T}^2, \\mathbf{W}^2, \\mathbf{E}^2=\\text{Transformer}_{N}\\left(\\mathbf{T}^1, \\mathbf{W}^1, \\mathbf{E}^{1 \\prime}\\right)$\n\n---\n\nq3: Line 404, it is quite unclear what \"oracle category guidance\" is. Could the authors explain it in a clearer way?\n\na3: We would be happy to provide a more intuitive explanation.\n\nDuring the inference process, as the entity masks are predicted one at a time, there are no resolved entities at the beginning of inference. As a result, the intermediate representations corresponding to all entities must engage in attention operations with the external category memory layer to obtain the retrieved category representations as described in the second line of Section 3.4. \n\nHowever, once an entity has been finalized in the prediction, we will directly use the true category vector of this known entity as the attention coefficient, which aligns more closely with the supervision during the training process. \n\n---\n\nq5: Line 848, the model sample k sentences and \"different sampling strategies bring negligible improvement\". Why is sampling still used?\n\na5: When handling too long documents with `CoherentED`, one has to select $k$ topic sentences out of all $N$ sentences. Despite the performance consistency among different sampling strategies, sampling topic sentences **out of** the model’s context window is more intuitive with our motivation, that is to capture global topic coherence. \n\n### Missing Yamada (2022) Results\n\nWe double-checked the Yamada (2022) work and found that they reported **In-KB accuracy** on AIDA-test-B rather than micro F1 score we generally use. We will append reproduced Yamada (2022)’s AIDA F1 score w/o fine-tuning on AIDA onto Table 1 in future manuscript.\n\n| Method | AIDA | MSNBC | AQUAINT | ACE2004 | CWEB | WIKI | AVG-5 | AVG-6 |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Yamada et al. (2022) | - | 96.3 | 93.5 | 91.9 | 78.9 | 89.1 | - | 89.9 |\n| Reproduced Yamada et al. (2022) | 88.7 | 96.1 | 94.6 | 92.3 | 76.9 | 88.7 | 89.7 | 89.6 |\n| Our CoherentED$_{\\text{large}}$ | 89.4 | 96.3 | 94.6 | 93.4 | 81.1 | 90.6 | 90.9 | 91.2 |\n\n### Hyperparameter Tuning and Dev Set\n\nWe apologize for the oversight of not introducing the strategy for hyper-parameter tuning. As we mentioned in Line 903-905, we do not run massive hyper parameter searches. We randomly held out 5% articles of Wikipedia 2022-09-01 dump and chose best-performing hyperparameter setting. The final checkpoint was trained with the full volume of the dump and the chosen hyperparameter setting.\n\nThank you for your valuable feedback and we hope these could address most of your concerns!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper points out two problems of current systems and fixes them with the two proposed components. The motivation is thus clear, and the solutions are backed by SoTA results on average across 6 popular datasets.", "reasons_to_reject": "First of all, the paper is quite difficult to understand. Please see the *questions* section for what should be clarified. \n\nSecond, there can be some problems in the experiments. \n* AIDA dataset is a standard for EL/ED evaluation in the literature. However, the paper does not present experiments for finetuning the proposed model on AIDA. \n* The paper doesn't fully show Yamada (2022) results -- lacking performance on AIDA. Note that in Yamada (2022) show on table 1 the performance of their model w/o finetuning. \n* The proposed model is trained on wikipedia only. The paper doesn't mention about any dev set, and thus it is unclear how hyper-params are tuned.", "questions_to_authors": "q1: from eq1, it seems that z is a sequence of hidden vector z1...zn, and thus each sentence is encoded in a sequence of hidden zs rather than only one vector z. But it contradicts with line 250. What is true here? \n\nq2: Eqs in sec 3.4 show that E2 is computed by a LayerNorm and a Transformer. How could it be? \n\nq3: Line 404, it is quite unclear what \"oracle category guidance\" is. Could the authors explain it in a clearer way? \n\nq5: Line 848, the model sample k sentences and \"different sampling strategies bring negligible improvement\". Why is sampling still used?", "paper_topic_and_main_contributions": "The paper is an extension of Yamada (2022) work with two new component: topic embeddings from neighbour sentences and entity category embeddings. For topic modelling, the paper uses a VAE whose encoder is BERT and decoder is GPT-2. The paper shows that with the two new components, new SoTA is achieved on average across 6 datasets (AIDA, MSNBC,...) without finetuning on AIDA training set.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "k4MMbaI5zh", "reviewer": "Reviewer_AHEW", "summary": "", "questions": "None", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We highly appreciate the reviewer's time and effort in providing feedback on our manuscript. And we would like to respond to the concerns raised, especially on the efficiency issue.\n\n### Novelty Issue\n\nWe would not deny that some individual components of CoherentED have precedents in previous works, such as the VAE architecture from Optimus and masked entity modeling from LUKE. These render our work less creative. \n\nHowever, the novelty of our framework lies in their unique combination of design motivation, with each of designs focusing on specific problems. For instance, the VAE design provides soft topic representations for later main model to condition on, providing global topic coherence. We also integrate a novel external category memory bank for intermediate entity representations to attend on, along with an oracle category label guidance strategy, which proves to be effective from evaluation results. `CoherentED` synthesizes these elements into a singular, optimized framework that exhibits synergy and improved performance over its constituent parts.\n\n### Runtime Efficiency\n\nWe would like to present the runtime efficiency comparison table on K50 consisting of 50 documents and 150 unresolved mentions. \n\n| Method | Basemodel | K50 Evaluation Took |\n| --- | --- | --- |\n| De Cao et al. (2021) | BART-large | 196.3 s |\n| Yamada et al. (2022) | BERT-large | 11.6 s |\n| Barba et al. (2022) | longformer-large-4096 | 5.72 s |\n| Our CoherentED$_{\\text{large}}$ | Customized BERT-large | 15.9 s |\n\nCompared to the baseline solutions, `CoherentED` does suffer in terms of inference speed, but the overall efficiency is acceptable. Importantly, using the `CoherentED` solution not only yields more accurate entity disambiguation results, but the soft topic tokens produced by the VAE Encoder module and the category prediction obtained by querying the Category Memory can also serve as valuable by-products for downstream applications, such as for interpretability analysis.\n\n### Missing References\n\nWe greatly appreciate your mention of related works concerning autoencoder-based topic models, especially the numerical evaluation of topic clustering in Bianchi et al., which is highly relevant to the further investigation of the source of performance improvement suggested by Reviewer LkD8. Works related to topic modeling will be added to Section 2 *Related Works* in our next manuscript.\n\nMeng et al. focus more on unsupervised topic discovery using pre-trained language models.  While they did not seek to variational training objective, the strong and intuitive regularization loss terms helped the model to learn compact topic representations that can be verified by qualitative results and t-SNE visualization (which we also used in our work). We would add this work to Section 2 *Related Works* and build a qualitative topic probe to assess the topic modeling ability of our VAE architecture.\n\nHope these would address all your concerns of our work! And thanks again for the constructive comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Neat and reasonable model design: The use of a VAE for topic embedding extraction and category memory augmentation make sense to me. They are conceptually simple and clear.\n* Clear presentation: The paper is clear in its motivation and relevance to related work. The method part is introduced in detail clearly.\n* Effectiveness: The proposed method is effective and outperforms compared baselines. There are also ablation studies that support the influence of each module along with visualizations of learned topic embeddings.", "reasons_to_reject": "* Novelty: While the combination of existing methods forms a new framework CoherentED, the overall novelty of the paper appears to be not very strong. Specifically, the VAE architecture consisting of a BERT encoder and a GPT decoder is directly borrowed from Optimus (Li et al. 2020), and the masked entity-style training of the entity extractor is largely similar to LUKE (Yamada et al. 2020) and the ED model in Yamada et al. 2022.\n* Lack of efficiency analysis: Since the CoherentED framework introduces an additional VAE for topic embedding extraction and category memory augmentation, it'll be necessary to analyze the latency brought by the extra modules (considering the baselines don't have these).", "questions_to_authors": "None", "paper_topic_and_main_contributions": "This paper studies the entity disambiguation (ED) task and proposes a new framework CoherentED aiming to enhance high-level topical coherence in entity predictions. The method consists of the following modules:\n* A variational autoencoder (VAE) that is used to extract latent topic vectors given input sentences. This ensures that topic coherence is preserved.\n* Category memory embedding table that is inserted into the intermediate Transformer layers, enabling the system to retrieve relevant categories for undecided mentions when appropriate.\n\nThe authors conduct evaluations on several ED tasks and demonstrate that CoherentED generally outperforms baselines by around 1.3 F1 points.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "None"}, "extra_text": {"Missing_References": "I believe the authors should also discuss similar usages of autoencoder architectures for topic/category embedding learning, such as the following two papers:\n* Bianchi et al. “Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence.” ACL 2020\n* Meng et al. “Topic Discovery via Latent Space Clustering of Pretrained Language Model Representations.” WWW 2022"}}]

Author rebuttal:

Meta-review: This paper introduces CoherentED, an entity disambiguation system that incorporates topic embedding of context sentences extracted by VAE and an external category memory to retrieve relevant categories for undecided mentions. The approach design is reasonable and effective though the key components of CoherentED are borrowed from existing studies. Experimental results demonstrate promising improvements on 6 benchmark datasets. One main concern from most reviewers is the experiment analysis: more discussions about the performance on long documents, supervised performance on AIDA, latency, etc. are needed.
Decision: Accept-Findings