A Read-and-Select Framework for Zero-shot Entity Linking
Authors: Zhenran Xu
Yulin Chen
Baotian Hu
Min Zhang
Venue: EMNLP 2023 Findings (2023)
Keywords: entity linking
zero-shot learning

Abstract: Zero-shot entity linking (EL) aims at aligning entity mentions to unseen entities to challenge the generalization ability. Previous methods largely focus on the candidate retrieval stage and ignore the essential candidate ranking stage, which disambiguates among entities and makes the final linking prediction. In this paper, we propose a read-and-select (ReS) framework by modeling the main components of entity disambiguation, i.e., mention-entity matching and cross-entity comparison. First, for each candidate, the reading module leverages mention context to output mention-aware entity representations, enabling mention-entity matching. Then, in the selecting module, we frame the choice of candidates as a sequence labeling problem, and all candidate representations are fused together to enable cross-entity comparison. Our method achieves the state-of-the-art performance on the established zero-shot EL dataset ZESHEL with a 2.55\% micro-average accuracy gain, with no need for laborious multi-phase pre-training used in most of the previous work, showing the effectiveness of both mention-entity and cross-entity interaction.

A Read-and-Select Framework for Zero-shot Entity Linking
Zhenran Xu, Yulin Chen, Baotian Hu and Min Zhang Harbin Institute of Technology (Shenzhen), Shenzhen, China (xuzhenran, 280110528jestu. hit. fxt..ddu.co cn (hubaotian, zhangmin2021 Jehit. edu...u.cc. cn
Abstract
Mention Context
Professor Henry Jones is Indiana Jones's father. He appears in the Indiana Jones and the Last Crusade theme.
Zero-shot entity linking (EL) aims at align- ing entity mentions to unseen entities chal- lenge the generalization ability. Previous meth- ods largely focus the candidate retrieval stage and ignore the essential candidate rank- ing stage, which disambiguates among enti- ties and makes the final linking prediction. In this paper, we propose a read-and-select (ReS) framework by modeling the main components of entity disambiguation, i.e., mention-entity matching and cross-entity comparison. First, for each candidate, the reading module lever- ages mention context to output mention-aware entity representations, enabling mention-entity matching. Then, in the selecting module, frame the choice of candidates as sequence labeling problem, and all candidate representa- tions are fused together to enable cross-entity comparison. Our method achieves the state- of-the-art performance on the established zero- shot EL dataset ZESHEL with 2.55% miero- average accuracy gain, with no need for labo- rious multi-phase pre-training used in most of the previous work, showing the effectiveness both mention-entity and cross-entity inter- action. Code is available at https: //t/ thub. com/HITsz- MeeTTsszTMGGRRRR Read- Beeladaand-see meleee.ct.
Indiana Jones (minifigure)
Indiana Jones (theme)
Indiana Jones is minifigure released 2008 as lead character of the Indiana Jones Theme. Indiana Jones is both professor and adventurer.
Indiana Jones is theme introduced in 2008 and discontinued the following year, and based off the Indiana Jones movie franchise Lucasfilm.
Prediction of cross-encoder
Gold entity
Figure 1: An example of ZESHEL, the zero-shot entity linking dataset. The cross-encoder often encounters confusion when presented with candidates that share lexical similarities with the mention.
Previous EL systems have achieved high perfor- mance when a large set of mentions and target enti- ties is available for training. However, such labeled data may be limited in some specialized domains, such as biomedical and legal cases. Therefore, we need to develop EL systems that can generalize to unseen entity sets across different domains, high- lighting the importance of zero-shot EL (Sil al., 2012; Vyas and Ballesteros, 2021).
1 Introduction
Entity Linking (EL) is the task aligning entity mentions in document to their referent entity in a knowledge base (KB). Considering the example in Figure 1, given the sentence "Professor Henry Jones Sr is Indiana Jones's father ..." ...", the men- tion Indiana Jones should be linked to the entity Indiana Jones (minifigure). As bridge that con- neets mentions in unstructured text and entities in structured KBs, EL plays a key role in a variety of tasks, including KB population (Ji and Noth- man, 2016), semantic search (Blanco et al., 2015), summarization (Dong et al., 2022), etc.
Zero-shot EL task is proposed by Logeswaran et al. (2019) along with the ZESHEL dataset. There are two key properties in the zero-shot setting: (1) labeled mentions for the target domain are unavail- able; (2) mentions and entities only defined through textual descriptions, without additional sources (e.g.. alias tables and frequency) which many previous EL systems rely on. Most zero-shot EL systems consist of two stages (Wu et al., 2020): (1) candidate retrieval, where a small set of candi-
Corresponding author.

---

dates are efficiently retrieved from KB; (2) candi- date ranking, where candidates are ranked to find the most probable one. So far, many methods have been proposed the first-stage retrieval (Sun et al., 2022; Sui et al., 2022; Wu et al., 2022). However, the candidate ranking stage, which makes the final prediction, is important yet less discussed. This paper focuses on the candidate ranking stage.
phase pre-training used in most of previous work. showing strong generalization ability. Further ablation study and case study demonstrate the ef- fectiveness of cross-entity interaction. Our contributions are summarized as follows:
We propose read-and-select (ReS) frame- work for candidate ranking stage of the zero- shot EL task, explicitly modeling the main components of disambiguation, i.c., mention- entity matching and cross-entity comparison.
The cross-encoder, introduced by (Logeswaran et al., 2019) is a widely-used candidate ranking model that facilitates mention-entity matching. It takes as input the concatenation of the mention context and cach entity description, and outputs whether or not the mention refers to the concate- nated entity. However, since each entity is encoded independently with the mention, the cross-encoded lacks the capability to compare all candidates at once and select the most appropriate one. Figure I shows an error case of the cross-encoder. When ranking candidates that share a high degree of lexi- cal similarity with the mention, the cross-encoder tends to assign high scores to both candidates. It may even incorrectly rank entity higher than the correct one. Given the mention "Indiana Jones", the wrong Indiana Jones (theme) is rated higher than the gold Indiana Jones (minifigure)) highlight- ing the necessity for cross-entity comparison.
We propose a new framing of candidate rank- ing as sequence labeling problem.
We achieve the sfate-of-the-art result on the established ZESHEL dataset, showing the ef- fectiveness of both mention-entity and cross- entity interaction.
2 Related Work
Entity linking (EL) bridges the gap between knowl- edge and downstream tasks (Wang al., 2023; Dong et al., 2022; Li al., 2022). There have been great achievements building general EL systems with Wikipedia as the corresponding knowledge base (KB). Among them, the bi-encoder has been particularly successful: Two encoders are trained to learn representations in shared space for men- tions and entities (Gillick et al., 2019). However, the actual disambiguation is only expressed via a dot product of independently computed vectors, ne- glecting mention-entity interaction. To this end, Wu et al. (2020) attempt to add a candidate rank- ing stage via stacking a cross- cross-encoder after the bi- encoder. Recent years have seen more Transformer- based ranking models obtaining promising perfor- mance gains (Barba et al., 2022; De Cao et al.. 2021), ), but most of proposed works are built the assumption that the entity set shared among the train and test sets (Sun al., 2022). In many prac- tical cases, the train and test sets may come from different domain distributions and disjoint entity sets, emphasizing the necessity of zero-shot EL. Zero-shot EL has recently attracted great inter- est from researchers. To enable progress on this task, Logeswaran et al. (2019) propose dataset called ZESHEL, where mentions in the test set are linked to unseen entities without in-domain labeled data. Following this work, a number of methods that operate on ZESHEL have been pro- posed, but they only focus on the retrieval stage by
Motivated by the above example, we propose a read-and- -nd--eld-eettttt (ReS) framework, explicitly model- ing both mention-entity matching and cross-entity comparison. The ReS framework consists of two modules: (1) In the reading module, we prepend prefix tokens (Li and Liang. 2021) to the concatena- tion entity description and mention context, and obtain mention-aware aware entity representations, en- abling mention-entity matching, (2) In the selecting module, the choice of candidates is framed as se- quence labeling problem, and all entity representa- tions are fused together to enable cross-entity com- parison. Unlike previous pipeline approaches that contain independently trained models (Wu al., 2020), our reading module and selecting module share parameters, and the whole model is trained end-to-end.
We evaluate our approach on the widely-used ZESHEL dataset (Logeswaran et al., 2019). Based on the experimental results, our ReS framework achieves the state-of-the-art performance with 2.55% micro-average accuracy improvement. Be- sides its performance advantage, ReS comes with another benefit: It does not need laborious multi-

---

Reading Module
Selecting Module
Mention Context
Encoder
Sequence labels
xXX w/ xxX XXX
Prefix
Entity;
Mention Context
Encoder
Encoder
Concat
Prefix
Entity;
Mention Context
Encoder
Prefix
Entity,
Mention Context
Encoder
Mention
Candidate tokens
tokens
Prefix
Entity,
Mention
Encoder
Illustration of our read-and -ndeec (RcS) framework. Our reading module encodes mention context and prefix tokens mention-aware entity representations. The above token embeddings fused together and fed into the selecting module, where the entity is predicted through sequence labeling. The parameters are shared between the encoders in the reading selecting module.
Figure 2:
modifying the bi-encoder architecture, such as in- corporating triplets in knowledge graphs (Ristoski al., 2021), adding ultra-fine entity typing super- vision (Sui al., 2022), creating multi-view entity representations (Ma al., 2021), and mining hard negatives for training (Zhang and Stratos, 2021; Sun et al., 2022). None of the above works con- siders the candidate ranking stage. However, the ranking stage is essential as it makes the final link- ing prediction among candidates, hence the need for further discussion about the ranking stage.
interaction respectively. First, we formally present the task formulation in Section 3.1. Next, we in- troduce reading module in Section 3.2. Finally, introduce selecting module and end-to-end training objective in Section 3.3.
Task Formulation
We first formulate the general entity linking (EL) follows: given an entity mention m in document and a knowledge base (KB) consisting of N enti- tics, i.c., & {e1, e2, eN }. the task is to identify the entity e E & that m refers to. The goal is to ob- tain an EL model on a train set of M mention-entity pairs D'I'rann correctly labels mentions in test set Destt Drain and Destt are typically assumed to be from the same domain.
The few works focusing on the ranking stage all follow the cross-encoder formulation. Fur- ther improvements include extending to long doc- uments (Tang et al., 2021; Yao et al., 2020) and different pre-training strategies (Logeswaran et al., 2019). In contrast, here propose sequence labeling formulation, where model receives the mention, its context and all mention-aware entity representations as input, and marks out the the cor- responding entity tokens. Note that this differs from the cross-encoder where each entity is inde- pendently encoded with mention context. With our framework, mention-entity matching and cross- entity comparison can be explicitly modeled.
{(m,, ei)l 1,.., M), which
In this paper, we focus on the candidate ranking stage of zero-shot EL. formulated as follows: In the zero-shot setting, DIrain and Dest contain N and Nigt sub-datasets from different domains re- spectively, i.e., Draán = (D = 1,.., N. and D est = [Digalé i = 1,..., Negt). Note that the entity sets in KBs (i.e., (E) 1,... Nerc). {Eiylé i - 1,.., Nege }) corresponding to the sub- datasets are disjoint. The entities and mentions are expressed as textual descriptions. In the candidate ranking stage, given the mention m and its k candi- date entities (denoted Cnd(m) {ei, ..., ex)) from the first-stage candidate retrieval, the ranking model chooses the most probable entity Cnd(m) as the linking result.
3 Methodology
In this section, describe the read-and-select (ReS) framework. our proposed method for zero- shot EL. As shown in Figure 2, the framework consists of the reading module and the selecting module, enabling eentiy and erossy

---

3.2 Reading
module. The output of the last layer m can be denoted as follows:
The reading module aims to produce mention m'ss representation and mention-aware representations of candidate entities (nd(mn), in preparation for the input of the selecting module.
H Tm = TTHH"
p**
= [h A Pi Pi.
PLp
(4)
For the mention m, the imput representation Tm is the word-piee pord-pieces of the mention and its surrounding context:
where h E Ri P Rldd and both nota- tions represent a row the matrix Hm.e, Through self-attention mechanism (Vaswani et al., 2017) the encoder, token-level interaction among candi- dates is conducted.
[CLS] ctxt, [START] m [END] ctxt, [SEP]
where ctxt, and ctxt, are context before and after the mention m respectively. [START] and [END] are special tokens to tag the mention. Then the men- tion representation (denoted as H") is the token embeddings from the last layer of the Transformer- based encoder T:
We frame the choice of candidates sequence labeling problem. The model learns to mark tokens of the corresponding entity correct, and mark other tokens wrong. The prediction score of the i-th token of candidate ej. denoted as 8 is calculated through a classification head (i.e., a linear layer W) with the sigmoid activation function:
H = (TT)) E bmxd
(1)
8/ Sigmoid Wp,"
(5)
where Lm the number of word- pord-pieces in Tm and d is the hidden dimension.
where 5 E [0.0, 1.0), and a higher score means that the token p,º is more likely to be correct. Optimization. For every mention m, we use the gold entity e as the positive example, and use N - 1 entities in Cnd(m) (e is not included) as negative examples. Suppose the candidate ej is the gold entity, all of its representation tokens (i.e., p ..... pł,) should be labeled correct, and tokens of other entities should be labeled wrong. We opti- mize the encoder with binary cross entropy loss:
For candidate entity the input representation is the word-pieces of its description. To obtain mention-aware entity representations, we prepend PREFIX (Li and Liang, 2021) of length L, to the concatenation of entity description and mention context, and then feed into the encoder T. The output the last layer can be computed as follows, enabling mention-entity matching:
He_ T(PRRFIX; Tei Tmm.)
Lp N Σ
We use the prefix's representation in He the cention-aware entity representation, i.e.,
L. = Lp x N
log(s?)
(6)
1=1
+ (1 - s )log - ()
ee_ [hi, ... ht,] ) Rex
(3)
where S takes the value if the p token should be labeled correct, otherwise 0.
We write h E I 1 d to denote the i-th row of the
matrix He
As the encoder T in the reading and selecting module share parameters, our framework can be optimized end-to-end. Note that the concatenation order of candidate representations is random during training.
3.3 Selecting
We propose a selecting module with new framing candidate ranking as a sequence labeling prob- lem. The model attends to mention context and all candidates together, and thus enables cross-entity comparison.
We obtain the final score for each candidate through maximum pooling over all of its tokens:
Inference.
With the mention representations and mention- aware entity representations from the reading mod- ule, the selecting module concatenates them all, and feeds it into the encoder T. Note that the en- coder here is the same as the one in the reading
nh = max max((5)) s.....))))
2,1)
(7)
We use the highest score to choose the best candi- date as the final linking result.

---

4 Experiment
Domains
Entities
Mentions
Training
this section, we assess the effectiveness of ReS on candidate ranking of zero-shot EL. We first in- troduce the experimental setup we consider, i.e., the data (Section 4.1), evaluation metric (Section 4.2), technical details (Section 4.3), and baselines in comparison (Section 4.4). Then we present the main result of ReS, its category-specific results, and the ablation study Section 4.5. More analy- sis about the cross-entity comparison can found in the impact of candidate number (Section 4.6) and case study (Section 4.7).
American Football
31929
3898
Doctor Who
40281
8334
Fallout
16992
3286
Final Fantasy
14044
6041
Military
104520
13063
Pro Wrestling
10133
1392
StarWars
87056
11824
World of Warcraft
27677
1437
Validation
Coronation Street
17809
1464
Muppets
21344
2028
4.1 Dataset
Ice Hockey
28684
2233
We conduct our experiments on the widely-used zero-shot EL dataset ZESHEL, which is con- structed by Logeswaran et al. (2019) and built with documents from Wikia' Table shows the overall statistics of the dataset. In this dataset, multiple en- tity sets are available for training, with task perfor- mance measured for which no labeled data is available. ZESHEL contains 16 specialized Wikia domains, partitioned into domains for training. 4 for validation and for test. The training set has a total of 49,275 labeled mentions while the validation and test sets both have 10,000 unseen mentions.
Elder Scrolls
21712
4275
Test
Forgotten Realms
15603
1200
Lego
10076
1199
Star Trek
34430
4227
YuGiOh
10031
3374
disjoint set of test entity sets
Table 1: Statisties of the zero-shot entity linking dataset 7ESHEL.
4.3 Implementation Details
ReS is implemented with PyTorch 1.10.0 (Paszke et al., 2019). The encoder in the reading and se- lecting module share parameters, initialized with RoBERTa- basee parameters (Liu et al., 2019). The number of parameters for ReS is roughly 124M.
4.2 Evaluation Metric
Following previous work (Wu al., 2020; Tang et al., 2021), we use the normalized accuracy as the evaluation metric. The normalized EL performance is defined as the performance evaluated on the sub- set of test instances for which the gold entity is among the top-k candidates during the candidate retrieval stage. We use the dataset along with top- 64 candidate sets retrieved by BM252. The number of test instances is 1000, 974, 2785, 2053 for such subsets of the Forgotten Realms, Lego, Star Trek and YuGiOh domain respectively, resulting in a top-64 recall of 68% on the test sets. We compute average performance across a set of domains by both micro- and macro- averaging. Performance is defined as the accuracy of the single-best identified entity (top-1 accuracy).
ReS is trained two NVIDIA 80G 1 GPUs. We use Adam optimizer (Kingma and Ba, 2015) with weight decay set to 0.01 for experiments. Batch size is set at 4. The length of PREFIX I.p is set at 3. We search learning rate among [5e- 6.255 based on the validation set. The best- performing lcarning rate is 4c-5, We search the number of training candidates (i.e., N in Equa- tion 6) in [10,20,40,56] and the final number is set to 56 due to memory constraint.
For each mention context and each entity descrip- tion in the reading module, length of the input at 256. We finetune for 4 epochs and choose the best checkpoint based on the micro- averaged normalized accuracy on the validation set. Finetuning ReS takes hours per epoch.
the maximum
't tps: /www. wikia. con
"The candidate sets are from the official ZFSHEL Github repository: tttpttbt.tttettee//ee/eeeee..ett www://ajaaugen//eeehe..o

---

Method
Forgotten Realms Lego
Star Trek YuGiOh Macro Acc. Micro Acc.
Baseline (Logeswaran et al., 2019)
77.05
BLINK (Wu et al., 2020)
76.58
BLINK (Wu et al., 2020)
87.20
75.26
79.61
69.56
77.90
77.07
GENRE* (De Cao al., 2021)
55.20
42.71
55.76
34.68
47.09
47.06
ExtEnD (Barba et al., 2022)
79.62
65.20
73.21
60.01
69.51
68.57
E-repeat (Yao et al., 2020)
79.64
Uni-MPR (Tang et al., 2021)
87.25
78.57
80.56
67.31
78,42
76.65
Bi-MPR (Tang et al., 2021)
89.60
80.50
81.04
68.74
79.97
77.85
ReS (ours)
88.10
78.44
81.69
75.84
81.02
80.40
ReS (w/o selecting) (ours)
87.10
77.10
80.57
73.22
79.47
78.79
Table Normalized accuracy of our ReS framework compared with previous state-of of ahe methods on test set of ZESHEL.. Bold denotes the best results. means our implementation. 4_" means not reported in the cited paper.
4.4 Baselines
an inter-paragraph attention mechanism. addition, Uni-MPR also adopts DAP, and ap- plies "Whole Entity Masking (WEM)) strat- egy in the masked language modeling (MLM) pre-training.
To evaluate the performance of ReS, we compare it with the following 7 state-of-the-art EL systems that represent diverse array of approaches.
Logeswaran et al. (2019) propose base- line method based cross-encoder. In Ta- ble 2, report its best-performing variant, i.e., the cross-encoder with Nomain-adaptive pre-training (DAP), and denote this model as Baseline.
Bi-MPR (Tang et al., 2021) built on the above Uni-MPR. adds another attention module to aggregate different paragraphs of the mention document. Same Uni-MPR, Bi-MPR also uses WEM strategy in DAP.
BLINK (Wu et al., 2020) uses a cross-encoder based on BERT-base, which only finetunes on the training set of ZESHEL, without any pre- training phases. For a fair comparison with ReS, we implement cross-encoder based on RoBERTa- base,, denoted BLINK*
4.5 Results
4.5.1 Main Results
We compare ReS with previous mate-of-the-ane models in Section 4.4, and list the performance in Table 2. ReS outperforms all other models on ZESHEL, and improves previous best reported results by 1.05% macro-averaged accuracy and 2.55% micro-averaged accuracy, showing the ef- fectiveness of our overall framework.
GENRE (De Cao al., 2021) is autore- gressive method base on generating the title of the corresponding entity with constrained beam search.
Besides its performance advantage, ReS comes with another benefit: it does need laborious multi-phase pre-training. As stated in Section 4 4.4, Yao et al. (2020) and Tang et al. (2021) both use domain adaptive pre-training (DAP) introduced by Logeswaran et al. (2019). Specifically, the model first pre-traiee pre-trains with texts in all domains (including source and target domains) with masked language modeling (MLM). Then for each target domain that the model is going be applied on, extra MLM pre-training stage is added before finetuning, using only the data the target domain. This means that Yao et al. (2020) and Tang et al. (2021) use different checkpoints for different test domains. However,
ExtEnD (Barba et al., 2022) proposes new framing entity disambiguation as text ex- traction task;
E-repeat (Yao et al., 2020) initializes larger position embeddings by repeating the small one from BERT-base, allowing reading more information in context. The berf-pero variant also uses DAP.
Uni-MPR (Tang et al., 2021) encodes men- tion and each paragraph of an entity descrip- tion, and aggregates these encodings through

---

ReS only needs one checkpoint for all domains and does not require any pre-training phase.
Method
HO
MC
AS
L0
Baseline
87.64
77.27
75.89
71.46
4.5.2 Domain-specific Results
BLINK
94.30
75.40
79.95
73.50
Uni-MPR
91.43
79.07
75.60
73.53
We break down the overall performance into dif- ferent test domains. As Table shows, compared with BLINK (better than the BLINK paper's re- ported result), our ReS framework outperforms by 0.90, 3.18, 2.08, 6.28 accuracy points on the For- gotten Realms, Lego, Star Trek, YuGiOh domains respectively. Specifically, the improvement in the YuGiOh domain is the most significant. We hypoth- esize the reason is that the test domain "YuGiOh" is closely related to the train domains "Star Wars" and "Final Fantasy" since they all belong to the super-domain of comics. By reading related con- texts and entity descriptions during training, ReS can effectively generalize to similar domains.
Bi-MPR
92.84
81.93
77.37
73.88
ReS
94.42
81.29
77.80
76.51
ReS (w/o selecting)
92.72
78.30
79.00
75.50
Table 3: Accuracy the category-specifie subsets in- cluding High Overlap (HO), Multiple Categories (MC), Ambiguous Substring (AS), Low Overlap (LO). "*" means our implementation.
4.5.4 Ablation Study
For the ablation of the selecting module, we con- sider variant of ReS: remove the selecting module altogether, use the mention-aware entity representa- tions (i.e., prefix tokens) from the reading module to obtain candidates' scores, thus without cross- entity interaction. Table shows that removing the selecting module causes performance drop across all test domains, leading to a 1.61 % micro-averaged accuracy drop. Based on Table 3, compared with ReS (w/o selecting), ReS improves the most in the Multiple Categories subset. This indicates that cross-entity interaction is helpful in disambiguat- ing lexically similar entities, which is in line with our motivation for fine- one-grained comparison among candidates illustrated in the case in Figure More cases will be discussed in Section 4.7.
4.5.3 Category-specifit Results
Logeswaran al. (2019) categorize the mentions based on token overlap between mentions and the corresponding entity title as follows:
High Overlap (HO): mention string is identi- cal to its gold entity title.
Multiple Categories (MC): The gold entity title is mention text followed by disambigua- tion phrase (e.g., in Figure 1, mention string: "Indiana Jones", entity title: "Indiana Jones (minifiguree)).
In addition, for the ablation of the reading mod- ule, we can compare the performance of BLINK* and ReS (w/o selecting): For mention- tity representations, BLINK* uses the [CLS] token while ReS uses the PREFIX tokens. According Table 2, ReS (w/o selecting) outperforms BLINK* by an overall of 1.72% micro- hicro-averaged accuracy. Based on fine-gratd ane-grained results of 4 categories Table 3, our prefix token embeddings achieve bet- ter performance on Multiple Categories and Low Overlap subsets, the aforementioned two relatively challenging subsets. These experimental results suggest that our reading module can better aggre- gate information from mention context and entity description, and thus create a better mention- -aare entity representation.
Ambiguous substring (AS): mention is sub- string of its gold entity title (e.g., mention string: "Agent", entity title: "The Agent").
All other mentions are categorized as Low Overlap (LO).
According to the fact that the performance of candidate retrieval models Multiple Categories and Low Overlap subsets much lower than the other two subsets (Sui et al., 2022), we conjecture that mentions in MC and LO are more difficult and ambiguous, and require more complex reasoning. As Table 3 shows, ReS achieves the state-of-the the- art performance in the LO subset. Compared with BLINK* the improvement the most notable in the MC subset with 5.89% gain. This indicates that our ReS framework has the reasoning ability to compare candidates and choose the best one.
4.6 Scaling the Number of Candidates
In Figure 3 (left), we report normalized accu- racy with respect to the number of test candidates.

---

1s ATLAT souryyy Ieptsement
Bei KE-ad HLACATIAN
No Nommlizee Aicurrccy
Unrormalized Accuracy
Improver Tane
Improvern memee
22
12-
18
Number candidates
Number candidates
Figure 3: Micro-averas averaged normalized accuracy (left) and unnormalized accuracy (right) of our ReS framework and
BLINK* on test sets as function of the number of test candidates.
Mention Context
ReS (w/o selecting)
ReS
Turkey: Turkey a type LEGO food. The pieces are orange- browgn are made out a rugged plastic material show that has been cooked, is made up of Part 33048 two of Part 33057.
Part 33057: Part 33057 a piece shaped a turkey leg. can be held Minifigure's hand ways. It supposed connect Part 33048 to assemble turkey.
can hold onto objects with the turkey leg. LEGO Digital De- signer name TROLL LEFT HAND.
Part 33013 is made resemble cake with fruit and chocolate only appeared three sets, two Scala and Mickey Mouse one Mickey Mouse set, it was used a spare part.
Mickey Mouse (Theme): Mickey Mouse is a LEGO theme based around Disney's 's Mickey Mouse and friends. The theme began and ended in 2000, and consisted of five sets.
This part
Mickey Mouse (figure): Mickey Mouse a figure from Mickey Mouse and the Disney's Baby Mickey theme. Hc appeared in all eight the themes
the
Table 4: Examples of top I candidate predicted by ReS and its variant without selecting. Bold in mention contexts denotes mentions. The titles and descriptions of gold entities are bold. The titles entities are italics.
Choosing among more candidates poses a chal- lenge to candidate ranking models. As candidate number increases, the accuracy of both models de- creases, but the accuracy of ReS decreases slower than BLINK leading to increasing gap be- tween their performance.
text. However, with eross-entity comparison in the selecting module, ReS can make comprehen- sive judgements with contextual information and all candidates, and disambiguate better such ambiguous cases.
For evaluating end-to-end EL performance, we use unnormalized accuracy, which is computed on the entire test set. In Figure (right), we can find that ReS consistently outperforms BLINK* for all candidate numbers. The improvement of ReS becomes larger the candidate number increases.
Conclusion
In this work, we focus the candidate ranking stage in zero-shot entity linking (EL). To disam- biguate better among entities which have high lexi- cal similarity with the mention, we propose a read- and-select (ReS) framework, which first produces mention-aware entity representations in the read- ing module, and then applies a sequence labeling paradigm on the fusion of all candidate representa- tions in the selecting module, thus modeling both mention-entin and cross-entity interaction. Experi- ments show that ReS achieves the statc-of-the-ar result ZESHEL, need of any laborious pre- training strategy, showing its effective generaliza- tion and disambiguation ability. Future work may expand this framework to take longer contexts and descriptions into consideration.
The above two findings demonstrate that ReS is better than BLINK at aggregating information from more candidates, showing the effectiveness of cross-entity interaction.
4.7
Case Study
We list the candidate predicted by ReS and its variant without selecting in Table 4 for qualitative analysis. The examples are in Low Overlap and Multiple Categories subsets respectively. From these examples, We can infer that models with only mention-entity interaction (e.g., ReS without selecting) tend to confuse candidates which both have high lexical similarity with the mention con-

---

Limitations
Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference Learning Representations, San Diego, CA.
Despite outperforming previous methods in macro and micro accuracy, our model does face limita- tions, chiefly due to the information loss when we restrict the length of mention context and en- tity descriptions to 256 tokens. The evidence for mention-entity matching could potentially reside in any paragraph within the entity document, rather than solely within the initial 256 tokens. This makes mention-awar aware entity representations less comprehensive, thereby impacting the interaction among candidates. Tackling the length constraint remains an intriguing avenue for future research.
Haoran Li, Yanggiu Song, and Lixin Fan. 2022. You don't know my favorite color: Preventing dialogue representations from revealing speakers' private per- sonas. arXiv preprint arXiv:2205. 10228.
Xiang Lisa Li and Percy I Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Pupers), pages 4582- 4597, Online. Association for Computational Lin- guistics.
i
Acknowledgments
Yinhan Liu, Myle Oit, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907.1
We thank Zifei Shan for discussions and the valu- able feedback. This work is jointly supported by grants: Natural Science Foundation of China (No. 62006061, 82171475), Strategic Emerging Industry Development Special Funds of Shenzhen 1UXCY1/2020010011140333333
Lajanugen Logeswaran, Ming- Ming-Wei Chang, Kenton Lee. Kristina Toutanova, Jacob Devlin, and Honglak Lee. 2019. Zero-shot entity linking by reading entity de- scriptions. In Proceedings of the 57th Annual Meet- ing of the Association for Compulational Linguistics, pages 3449-3460, Florence, Italy. Association for Computational Linguistics.
References
Edoardo Barba, Luigi Procopio, and Roberto Navigli. 2022. ExtEnD: Extractive entity disambiguation. In
Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics Volume 1:
Xinyin Ma, Yong Jiang, Nguyen Bach, Tao Wang. Zhongqiang Huang, Fei Huang, and Weiming Lu, 2021. MuVER: Improving first-stage entity retrieval with multi-view entity representations. In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2617-2624, Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
Long Papers), pages 2478-2488 2488, Dublin, Ireland. As- sociation for Computational Linguistics.
Roi Blanco, Giuseppe Ottaviano, and Edgar Meij. 2015. Fast and space-efficient entity linking for queries. In Proceedings of the Eighth ACM International Confer- ence on Web Search and Data Mining. WSDM 15, page 179-188. New York, NY, USA. Association for Computing Machinery.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style. high-performance deep learning library. Advances in neural information processing systems, 32.
Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2021. Autoregressive entity retrieval. In International Conference on Learning Representa- tions.
Yue Dong, John Wicting, and Pat Verga. 2022. Faithful the document the world? mitigating hallu- cinations via entity-linked knowledge abstractive summarization. arXiv preprint arXiv:2204. 13761..
Petar Ristoski, Zhizhong Lin, and Qunzhi Zhou. 2021. Kg-zeshel: Knowledge graph-enhanced zero-shot entity linking. In Proceedings of the 11th on Knowl- edge Capture Conference, K-CAP '21, page 49-56. New York, NY, USA. Association for Computing Machinery.
Daniel Gillick, Sayali Kulkarni, Larry Lansing, Alessan- dro Presta, Jason Baldridge. Eugene Ie, and Diego Garcia-Olas 2019. Learning dense representations for entity retrieval. In Proceedings of the 23rd Con- ference on Computational Natural Language Learn- ing (CoNLL), pages 528-537, Hong Kong, China. Association for Computational Linguisties.
Avirup Sil, Ernest Cronin, Penghai Nie, Yinfei Yang. Ana-Maria Popescu, and Alexander Yates. 2012. Linking named entities to any database. In Proceed- ings of the 2012 Joint Conference on Empirical Meth- ods in Natural Language Processing and Computa- rional Natural Language Learning, pages 116-127, Jeju Island, Korea. Association for Computational Linguistics.
Heng Ji and Joel Nothman. 2016. Overview of tac- kbp2016 tri- lingull edl and impact on end-to-end kbp. Theory and Applications of Categories.

---

Xuhui Sui. Ying Zhang, Kehui Song, Baohang Zhou, Guoqing Zhao, Xin Wei, and Xiaojie Yuan. 2022. Im- proving zero-shot entity linking candidate generation with ultra-fine entity type information. In Proceed- ings of the 29th International Conference on Com- putational Linguistics, pages 2429-2437 Gyeongju, Republic of Korea. International Committee on Com- putational Linguistics.
Wenzheng Zhang and Karl Stratos. 2021. Understand- ing hard negatives in noise contrastive estimation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 1090- 1101, Online, Association for Computa- tional Linguistics.
Kai Sun, Richong Zhang, Samuel Mensah, Yongyi Mao, and Xudong Liu. 2022. A transformational biencoder with in-domain negative sampling for zero-shot entity linking. In Findings of the Association for Compu- kational Linguisties: ACI. 2022, pages 1449-1458, Dublin, Ireland. Association for Computational Lin- guistics.
Hongyin Tang. Xingwu Sun, Beihong Jin, and Fuzheng Zhang. 2021. A bidirectional multi-paragraph read- ing model for zero-shot entity linking. Proceedings of the AAAI Conference on Artificial Intelligence, 35(15) 13889-13897.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Yogarshi Vyas and Miguel Ballesteros. 2021. Linking entities to unseen knowledge bases with arbitrary schemas. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 834 834,844, Online. Association for Computational Linguistics.
Weiqi Wang, Tianqing Fang. Wenxuan Ding, Baixuan Xu, Xin Liu, Yangqiu Song, and Antoine Bosse- lut. 2023. Car: Conceptualizaatoon-aammeee soner for zero-shot commonsense question answer- ing. arXiv preprint urXiv:2305. 14869.
rea-
Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. 2020. Scalable zero- shot entity linking with dense entity retrieval. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3397-6401, Online, Association for Computa- tional Linguistics.
Taiqiang WU, Xingyu Bai, Weigang Guo, Weijie Liu, Siheng Li, and Yujiu Yang. 2022. Modeling fine- grained information via knowledge-aware hierarchi- cal graph for zero-shot entity retrieval, arXiv preprint arXiv:2211.1099)
Zonghai Yao, Liangliang Cao, and Huapu Pan. 2020. Zero-shot entity linking with efficient long range se- quence modeling. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2517-2522, Online. Association for Computational Linguistics.


Reviews:
[{"review_id": "xttBcA8yg8", "reviewer": "Reviewer_kXSc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- (+) Simple, yet meaningful idea to improve candidate ranking for zero-shot EL.\n- (+) Thorough comparison with previous work.\n- (+) Competitive, state-of-the-art results.\n", "reasons_to_reject": "- (-) Single, focused contribution: the paper is on the wordy side and could have been (imho...) easily presented as a short paper. I do not see enough experiments or analysis to justify a full-blown full paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a framework for zero-shot entity linking using mention-entity matching and candidate ranking as sequence labeling as key new contributions.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0gbiWiLTzi", "reviewer": "Reviewer_cEiX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for the detailed and constructive review! We appreciate your positive feedback about our paper's motivation and novelty. Here, we address your main concerns:\n\n**Q1: The paper lacks comparative analysis with robust baselines, such as Bi-MPR.** \n\nThanks for the insightful suggestion. We have introduced the difference of our method in Line 197-212, and compared the domain-specific and category-specific results with baselines (such as Bi-MPR) in Table 2 and Table 3. In the final version, we will ensure to highlight the discussion of the distinction between our method and baselines.\n\n**Q2: The concept of integrating entities in the encoder is not entirely novel and has been previously proposed in Fusion-in-decoder.**\n\nOur method differs significantly from Fusion-in-decoder (FID) in both **motivation** and **implementation**:\n\n- FID aims to **aggregate evidence** from multiple passages, while our method focuses on cross-entity interation to **make comparsions** among entities and capture their fine-grained difference.\n- FID simply concatenates **the whole sequence embeddings** of all passages, while our method employs **a prefix prompt** to obtain mention-aware entity representations before concatenation.\n\nIn addition, we have tried to implement FID for zero-shot entity linking in our preliminary study, and encountered with the **out-of-memory (OOM)** problem with two 80G A100 GPUs, due to the long length of entity descriptions and mention contexts. Therefore, the FID's fusion strategy is not applicable for this task.  \n\n**Q3: The scope of the task and dataset is somewhat limited, with a singular focus on the ZESHEL dataset.**\n\nAs we focus on the problem of zero-shot entity linking, to the best of our knowledge, ZESHEL is the only well-established and widely-used benchmark for this task.\n\nThanks for the suggestion of adding experiments on additional datasets. We have implemented our method on AIDA-YAGO (with the same experimental setting as ExtEnD [1]). The following table shows the results. Our method outperforms the state-of-the-art ExtEnD and exhibits strong performance in an *in-domain* setting.\n\n|    | AIDA Accuracy |\n|:---------:|--------------:|\n|   GENRE   |88.6|\n|   ExtEnD  |          90.0 |\n|    ours   |          **91.5** |\n\n*The results of GENRE and ExtEnD are taken from [1].\n\nWe hope the above response and the general response have addressed your concerns, and we are grateful for your help in improving the paper. Please let us know if you have any further comments! \n\nReference:\n[1] Edoardo Barba, Luigi Procopio, Roberto Navigli. 2022. ExtEnD: Extractive Entity Disambiguation. https://aclanthology.org/2022.acl-long.177/", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper enhances the candidate reranking phase of entity disambiguation, achieve new state-of-the-art performance on the ZESHEL dataset.\n- The paper underscores the critical role of cross-entity comparison in the process, a factor that has been largely overlooked in prior research.", "reasons_to_reject": "- The paper lacks comparative analysis with robust baselines, such as Bi-MPR. A detailed discussion highlighting the distinctions between this work and these baselines would have been beneficial.\n- The concept of integrating entities in the encoder, although presented with variations, is not entirely novel and has been previously proposed in Fusion-in-decoder (https://arxiv.org/pdf/2007.01282.pdf).\n- The scope of the task and dataset is somewhat limited, with a singular focus on the ZESHEL dataset. The paper could have been strengthened by reporting results on additional entity linking datasets, such as AIDA-YAGO.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces a novel framework, termed Read-and-Select (ReS), designed to enhance the candidate ranking phase in zero-shot entity linking tasks. The ReS framework consists of two modules: a reading module and a selection module. The reading module generates mention-aware entity representations for each candidate entity, facilitating effective mention-entity matching. The selection module, on the other hand, amalgamates all candidate representations and produces sequence labels, thereby enabling a comprehensive cross-entity comparison. The proposed framework demonstrates superior reranking performance on the ZESHEL dataset.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "s1AevGihln", "reviewer": "Reviewer_WPJY", "summary": "", "questions": "1. The T in Equation 1 and Equation 2 are the same Transformer-based encoder?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 1, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and constructive review! We address your concerns as follows:\n\n**Q1: Poor motivation**\n\nThe cross-encoder focuses on **mention-entity interaction** to perform semantic matching between **a mention** and **an entity**, while our method attends to **all entities at one time** in the selecting module. We focus on **entity-entity interaction** to make comparisons among entities and capture their fine-grained difference.\n\n**Q2: Poor novelty of methodology**\n\nAs stated in Line 110-112, our work is partly motivated by the promising cross-entity interaction. Though ExtEnD has explored such interaction and achieved state-of-the-art performance in general domain, **its performance drops under the zero-shot setting (shown in Table 2)**, significantly worse than the cross-encoder baseline and our method.\n\nIn ExtEnD, the entity is only represented by **its name**, while our method use the reading module to obtain **mention-aware entity representations**, enabling more effective cross-entity comparison and more accurate prediction.\n\n**Q3: Unfair comparison**\n\n**Regarding the encoder:** \n\nPrevious works based on BERT-base (Baseline, E-repeat, Uni-MPR and Bi-MPR) all utilize some continual pre-training strategies (e.g., domain-adaptive pre-training with in-domain corpus). However, our method does not need laborious multi-phase pre-training by simply applying RoBERTa-base. Our target is to learn how to compare candidates according to the their descriptions, and generalize across domains without the domain-specific corpus.\n\nIn addition, as stated in Line 424-427, for fair comparison, we reproduced the cross-encoder based on RoBERTa-base, denoted as BLINK* in Table 2.\n\n**Regarding the number of candidates:**\n\nThere might be a misunderstanding. As stated in Line 398-401, the number of **training** candidates is **56** due to memory constraint. The number of candidates during **inference** is **64**, same as the experimental setting in previous works.\n\n**Q4: Wrong baseline results**\n\nWe double checked the BLINK paper and E-repeat paper, and confirmed that their reported results are computed by **macro-averaging**:\n\n- In the Table 2 of the BLINK paper: \"Average performance across a set of worlds is computed by macro-averaging.\".\n- In Section 4.1 of the E-repeat paper: \"Average performance across a set of domains is computed by macro-averaging.\"\n\n\n\n**Q5: Poor figures**\n\nFigure 1 shows an error case of the cross-encoder, which motivates the fine-grained comparison among candidates. Thanks for your advice, and we will present the motivation more clearly in the next version.\n\n**Q6: The T in Equation 1 and Equation 2 are the same Transformer-based encoder?** Yes.\n\nWe hope the above details and the general response have addressed your concerns. We are greatful for your help in improving the paper. If there are any additional questions or concerns, we are happy to engage in further discussions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper tries to capture cross-entity comparison information for zero-shot entity linking task.\n2. The method is easy to follow.", "reasons_to_reject": "1. Poor figures (minor). Figures in this paper are not clear. I can not obtain the effectiveness of capturing fine-grained cross-entity interaction among candidates in comparison in Figure 1.\n2. Poor motivation (major). The cross-encoder architecture is not \"ignoring cross-entity comparison\". It also \"attends to all candidates at once\" to obtain the final matching scores. Of course, it may be not so fine-grained.\n3. Poor novelty of methodology (major). The idea of capturing fine-grained cross-entity interaction among candidates has already been proposed in entity linking (ExtEnD: Extractive entity disambiguation).\n4. Unfair comparison (major). As far as I know, all previous zero-shot entity linking candidate ranking works use BERT-base parameters rather than RoBERTa-base and top 64 candidates rather than top 56. These may result in unfair comparison in the experimental results.\n5. Wrong baseline results (minor). As far as I know, the results in BLINK and E-repeat are Micro Acc. rather than Macro Acc.", "questions_to_authors": "1. The T in Equation 1 and Equation 2 are the same Transformer-based encoder?", "paper_topic_and_main_contributions": "This paper focuses the zero-shot entity linking task and proposes a framework to capture fine-grained cross-entity comparison information among candidates to improve the performance.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a method for reranking entity candidates for zero-shot Entity Linking.

All three reviewers acknowledge the importance of the cross-entity comparison for candidate ranking and Reviewers kXSc and cEiX also recognise the significant improvement to the state of the art by this work. 

Reviewers kXSc and cEiX pointed out different ways in which the scope work and the manuscript itself are limited. The authors have responded positively to these points by providing new results and suggesting structural changes to the paper that would shift more original content (rather than background information) into the main paper. The issue regarding the scope pointed out by Reviewer cEiX (reranking in zero-shot with a single dataset) remains, but I think this mostly affects the excitement for rather than the soundness of the work.

After an extensive conversation with Reviewer WPJY, we were able to dill down to a few specific issues that they identified in the current work (poor motivation, low novelty, unfair comparisons). I agree with the reviewer on the latter - the change of encoder from BERT to RoBERTa means that previous works (other than the re-implemented BLINK) are at a disadvantage (and that an ablation study with BERT as an encoder would be a plus), but I don't think it's detrimental to the soundness of the work. Regarding the other two issues, my opinion is that they are not as severe as the reviewer presented them in their review and discussion and that neither impacts the soundness of the work.
Decision: Accept-Findings