Revisiting Sparse Retrieval for Few-shot Entity Linking
Authors: Yulin Chen
Zhenran Xu
Baotian Hu
Min Zhang
Venue: EMNLP 2023 Main (2023)
Keywords: entity linking
sparse retrieval

Abstract: Entity linking aims to link ambiguous mentions to their corresponding entities in a knowledge base.
One of the key challenges comes from insufficient labeled data for specific domains. 
Although dense retrievers have achieved excellent performance on several benchmarks, their performance decreases significantly when only a limited amount of in-domain labeled data is available.
In such few-shot setting, we revisit the sparse retrieval method, and propose an ELECTRA-based keyword extractor to denoise the mention context and construct a better query expression.
For training the extractor, we propose a distant supervision method to automatically generate training data based on overlapping tokens between mention contexts and entity descriptions.
Experimental results on the ZESHEL dataset demonstrate that the proposed method outperforms state-of-the-art models by a significant margin across all test domains, showing the effectiveness of keyword-enhanced sparse retrieval.

Revisiting Sparse Retrieval for Few-shot Entity Linking
Yulin Chen", Zhenran Xu", Baotian Hut and Min Zhang Harbin Institute of Technology (Shenzhen), Shenzhen, China {200110528, xuzhenran]@stu. hit. mit.edu.de cn (hubaotian, zhangmin2021 J@hit. edu...duccco cn
Abstract
2022). Using bi-encoders and Approximate Near- est Neighbors (ANN) search has become the stan- dard approach for initial entity retrieval, overcom- ing the long-standing vocabulary mismatch prob- lem and showing promising performance gains. Although the bi-encoder has shown strong in- domain performance, they typically require train- ing on sufficient labeled data to perform well in a specific domain (Ren et al., 2022). However, such labeled data may be limited or expensive in new and specialized domains. As reported by Li et al. (2022b), in the absence of sufficient training data, the recall of bi-encoder significantly decreases. performing even worse than unsupervised sparse retrievers (e.g.. BM25 (Robertson and Zaragoza, 2009)). Motivated by the above finding, this work, instead of resorting to dense representations in the "semantic space", we go back to the "lex- ical space" and explore PLM-augmented sparse representations for few-shot EL.
Entity linking aims link ambiguous mentions their corresponding entities in knowledge base. One of the key challenges comes from insufficient labeled data for specific domains. Although dense retrievers have achieved excel- lent performance on several benchmarks, their performance decreases significantly when only a limited amount of in-domain labeled data available. In such few-shot setting, we re- visit the sparse retrieval method, and propose an ELECTRA- keyword extractor to de- noise the mention context and construct bet- ter query expression. For training the extractor, we propose a distant supervision method to automatically generate training data based on overlapping tokens between mention contexts and entity descriptions. Experimental results the ZESHEL dataset demonstrate that the proposed method outperforms state-of-the-art models by significant margin across all test domains, showing the effectiveness of keyword- enhanced sparse retrieval. Code is avail- able at https://gi thub. com/HITsz-TMG/ RetsseeReeiiivllleee
When applying BM25 to the first-stage retrieval, previous work only employs the mention string query (Wu et al., 2020). However, this can be insuf- ficient due to under-speeified mentions (c.g., "his embodiments" in Figure 1), requiring additional context formulate a more comprehensive query. While an intuitive solution could be incorporating all context into the query, this approach is subopti- mal for two reasons: (i) it does not create mention- specific queries, i.c., for all mentions in a docu- ment, their queries and retrieved entities are the same; (ii) it may introduce noise into queries (Mao et al., 2022), since not all words in the context are necessary for understanding the mention.
EL.
1 Introduction
Entity linking (EL) aligns entity mentions in docu- ments with the corresponding entities in a knowl- edge hase (KB), which is a crucial component of information extraction (Ji and Nothman, 2016). Typically, EL systems follow "retrieve and re- rank" pipeline (Logeswaran et al., 2019): Candi- date Retrieval, where small set of candidates efficiently retrieved from a large number of enti- ties, and Candidate Ranking, where candidates are ranked to find the most probable one. With the rapid development of pre-trained language models (PLM), EL has witnessed radical paradigm shift towards dense retrieval (Ma et al., 2021; 1; Sun et al.,
This paper introduces a keyword extractor to denoise the context for BM25 retrieval. The extrac- tion is modeled as binary token classification, iden- tfying which tokens in context should be added to the mention string. We employ discriminative PLM, specifically ELECTRA (Clark et al., 2020), as the basis of our keyword extractor, and leverage
"Both authors contributed equally this work. 'Corresponding author.

---

its discriminator head to predict score for each token. The top-k tokens with the highest scores are then added into the final query.
models, the bi-encoder has largely mitigated this problem, but its effectiveness depends on extensive annotation data (Ren et al., 2022). Without suffi- cient labeled data in specific domains, the recall of the bi-encoder drastically decreases (Li et al., 2022b), hence the need for further research about few-shot EL.
Training the extractor requires binary labels to distinguish whether token is a keyword. The key- words are expected to be related to the mention and necessary for understanding it. We propose distant supervision method to automatically gen- erate training data, based on overlapping words between the mention context and the description of the corresponding entity. These overlapping words are then ranked based on their BM25 score, which measures the relevance of the word to the entity description. We select the top-k words with the highest scores as keywords.
Few-shot Eniity Linking is recently proposed by Li et al. (2022b). They split the data in a specific domain of ZESHEL (Logeswaran et al., 2019) and provide few examples to train. They address the data scarcity problem with synthetic mention-entity pairs, and design meta-lcarning mechanism assign different weights to synthetic data. With this training strategy, the performance of bi-encoder is better than simply training with in- sufficient data. However, it still cannot surpass the performance of the unsupervised sparse retrieval method (i.e., BM25 in Table 2). As dense retrievers (e.g.. the bi-encoder) and generative retrievers (e.g., GENRE (Cao et al., 2021)) are data-hungry, we go back to the "lexical space" and resort to sparse retrievers for few-shot EL solutions.
Following the few-shot experimental setting in Li al. (2022b), We evaluate our approach on the ZESHEL dataset (Logeswaran et al., 2019). The results highlight our approach's s superior per- formance across all test domains, with an average of 15.07% recall @ 64 improvement over the best dense retriever result. In addition to its superior per- formance, our method inherits the desirable proper- ties of sparse representations such efficiency of inverted indexes and interpretability.
3 Methodology
The contributions of this work are threefold:
When applying BM25 to the first-stage retrieval of EL, previous work directly uses mention string query. We propose a keyword extractor to denoise the context, and add the extracted keywords into the query. Then the expanded query used for BM25 retrieval. Section 3.1, we introduce our distant supervision method to generate keyword labels. In Section 3.2, we present the architecture, training and inference of our keyword extractor. Preprocessing. BM25 implementation gener- ally comes with stopword removal (Yang et al., 2017). In this work, we remove the words which oceur in more than 20% of entities' description documents, resulting in removal of uninformative words (such as "we", "the", etc.).
We are the first to explore PLM-augmented sparse retrieval in entity linking, especially in scenarios of insufficient data.
We propose keyword extractor to denoise the mention context, and use the keywords to formulate the query for sparse retrieval, ac- companied by distant supervision method to generate training data.
We achieve the state-of-the the-artt performance across all test domains of ZESHEL, outper- forming previous dense retrieval methods by large margin.
2
Related work
3.1 Distant Supervision
Entity linking (EL) bridges the gap between knowl- edge and downstream tasks (Wang et al., 2023; Dong et al., 2022; Li et al., 2022a). Recent entity linking (EL) methods follow two-stage "retrieve and re-rank" approach (Wu al., 2020), and our work focuses on the first-stage retrieval. Traditional retrieval systems based on lexical matching strug- gle with vocabulary mismatch issues (Formal et al., 2021). With the advent of pre- treitrained language
Given a mention m, its context M and the de- scription document E of its corresponding entity, the distant supervision method aims to gather key- words in the context M related to the descrip- tion E. As shown in Figure 1(a), the overlapping words form a preliminary keyword set (denoted as K = {W1, W2,.... ()x-))) i.e.,
k = words(M) words(E)
(1)

---

"Carrier" anime-only archetype of monsters used by Aporia embodiments...
Overlapping words
Mention Context
supported by Meklord...
monster
top-k words keywords
meklord anime yugioh
ELECTRA
Three Pure Nobles: capable wielding the all powerful
Entity Description
"Meklord Emperars" and absorbing Synchro Monster...
archetype
monsters
embodiments
by
Meklord
ordered by BM25 score
[a] Distant Supervisiin
(b) Keyword Extractor
Figure I: Overview of our method. The distant supervision method (left) generates weak keyword labels based on overlapping words. Then the keyword extractor (right) is trained on these generated keyword labels.
where words( is the set of words in sequence The words in KC* are further ranked according to the BM25 score between each word w; and the document E, calculated as follows:
Domains
#Train
#Dev
#Test
Forgotten Realms
50
50
1100
Lego
50
50
1099
Star Trek
50
50
4127
YuGiOh
50
50
3274
s(W;, E) (w:) f(w,, E) (k1 +
Table 1: Statistics of the few-shot entity linking dataset ZESHEL.
E f(w,, E) k1 (1 b avgdl (2)
where ID;( represents the inverse document frequency of word w/, f(w;, E) denotes the fre- quency of word wy appearing in description E, |E] is the length of the description D, avgdl is the av- erage document length in entity descriptions. The hyperparameters in and b are adjustable values, where we set kj 1.5 and b 0.75. We select the top-k words with the highest scores as the final keyword set IC for the mention m..
with a discriminator head (i.e., a linear layer W) and the sigmoid activation function:
8 Sigmoid W)
(4)
where h; E R is i-th of the matrix H. & E [0.0, 1.0), and a higher score means the token is more likely to be keyword.
Optimization. In Section 3.1, for each mention m and its context M, we have obtained the key- word set K. The tokens of every word in K. should be labeled as keywords. We optimize the extractor with binary cross entropy loss, below:
3.2 Keyword Extractor
Our keyword extractor aims to denoise the context through sequence labeling. Since discriminative pre-trained models can be easily adapted tasks in- volving mulli-token classification (Xia et al., 2022), we use ELECTRA as the backbone of our keyword extractor, as shown in Figure 1(b). For the mention in its context M, the input Tm is the word-pieces of the highlighted mention and its context:
1 Σ s.lo((;;) (1- s;)log(1 -8)) (5)
where 8; takes the value the i-th token should be labeled as a keyword, otherwise O. Inference. The keyword extractor predicts a score & for every token. The top-k distinct tokens with the highest scores are selected for the keyword set R. The final query for BM25 combines the extracted keywords with the mention
[CLS] Mj [START] m [END] M, [SEP]
where Mt and M, are context before and after the mention m respectively. [START] and [END] are special tokens to tag the mention. The token em- beddings from the last layer of the extractor T' can be denoted as follows:
= words(m) KC
(6)
The top-n retrieved entities by BM25 are the result of our method.
H = T(Tm) E LLxd
(3)
4
Experiment
4.1
Dataset and Evaluation Metric
where the number of word-pieces in Tm and d is the hidden dimension. Finally, the predicted score of the i-th token, denoted as &, is calculated
For fair comparison with et al. (2022b), we follow their few-shot experimental settings, i.e.,

---

Method
Forgotten Realms Lego
Star Trek
YuGiOh Macro Average Mícro Average
BLINK (Wu al., 2020)
35.27
52.68
21.57
35,00
36.13
31.28
MUVER (Ma et al., 2021)
37.27
66.15
23.89
56.38
45.92
41.34
GENRE (Cao et al., 2021)
67.27
50.32
55.75
29.14
50.62
47.37
DL.4FL (I.c and Titov, 2019)
66.71
75.12
59.91
59.20
65.24
62.19
MctaBINK (Li
2022b)
69.56
78.17
61.41
61.28
67.61
64.22
BM25 (mention words)
83.87
81.62
65.99
61.03
73.13
68.
BM25 (mention context)
75.22
74.19
59.20
71,45
70.02
66.93
Ours
88.53
85,48
74,81
79.75
82.14
79,29
Ours (w/o mention words)
80.36
77.53
62.03
74.40
73.58
70.12
Table 2: Recall @4 of our method compared with dense and generative retrievers (top) and sparse retrievers (bottom) on all test domains of ZESHEL. Bold denotes the best results.
Mention Context
Corresponding Entity Description
Birth of the Ultimate Tag is the forty seventh chapter of the Yu Gi Oh GX manga. It was first printed in Japanese in the V Jump magazine and in English in the Shonen Jump magazine. Both of which were printed in volume 7 of the Yu Gi Oh GX graphic novels afterwards...
Yu-Gi-Oh! GX-Volume King Atticus True Power!!, known The King s True Power!! Japanese version, is the seventh volume of the Yu -Gi-Oh! GX manga.
Anzu bought magazines with tourist information. The two met at Domino Station at 10:00 where Anzu was nervous at first. They first went to coffee shop for drinks...
Domino Coffee is a restaurant in Domino City in the anime and manga. ... Gardner and Yami Yugi went here among other places on their date.
Table 3: Examples of the extracted keywords and their overlap with the corresponding entity descriptions. Italic denotes mentions. red in contexts shows the extracted keywords. Red in entity descriptions represents words overlapping with the keyword set.
choose ZESHEL (Logeswaran et al., 2019) as the dataset, split the data as shown in Table and use top-64 recall (i.e., Recall @ 64) as the evaluation metric.
performs even worse, indicating the importance of context denoising. The YuGiOh domain has the lowest recall of BM25 (mention words), showing a large vocabulary mismatch between mention string and entity documents. However, with mention- related context, our method addresses this issuc effectively, resulting in an impressive 18.72% per- formance improvement in the YuGiOh domain.
4.2 Baselines
To evaluate our method, we compare it with the state-of-the-art sparse retriever (i.e., BM25 (Robert- son and Zaragoza, 2009)), dense retrievers (i.c., BLINK (Wu et al.. 2020), MUVER (Ma et al., 2021), MetaBLINK (Li et al., 2022b), DL4EL (Le and Titov, 2019)), and the generative retriever GENRE (Cao al., 2021). More details of the baselines can be found in Appendix A. Implemen- tation details are in Appendix B.
Ablation study in Table 2, i.e. results of BM25 (mention words) and Ours (w/o mention words). demonstrates that the mention string and the key- words are both important to the final query. Delet- ing either of them in BM25's query will result a significant performance drop. To further explore the contribution of keywords, we present the case study in Table 3. From these cases, we can find that the extracted keywords are semantically related to the mention, excluding the noise in context.
4.3 Results
Main results. Table 2 shows that our model outper- forms all dense and generative retrieval methods across all domains, and achieves a 15.07% micro- averaged Recall 64 improvement over the best reported result (i.c., MetaBLINK).
Conclusion
In this work, we focus on the first-stage retrieval in few-shot entity linking (EL). Given the significant performance drop dense retrievers for few-shot scenarios, we revisit the sparse retrieval method and enhance its performance with a keyword ex- tractor. We also propose a distant supervision ap-
Comparison among sparse retrievers in Ta- ble highlights the importance of mention-related context. BM25 (mention words) is apparently not the optimal solution since it ignores all context; BM25 (mention context) leverages all context and

---

proach for automated training data generation. Our extensive experiments demonstrate our method's superior performance, with case studies showing denoised context and mention-related keywords. Future work may focus on more PLM-augmented sparse retrieval solutions for EL, expanding the query to words outside the context.
Heng Ji and Joel Nothman. 2016. Overview of TAC- KBP2016 tri tri-lingual EDL and its impact on end-to- end KBP. In Proceedings of the 2016 Text Analy- sis Conference, TAC 2016, Gaithersburg, Maryland, USA, November 14-15, 2016. NIST.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Limitations
Despite surpassing the previous soate-of-the-aee per- formance by significant margin, our method ex- hibits certain limitations, primarily related to the constraint on the keywords. The formulated query effectively denoises the context, but is still limited to the scope of mention context. The vocabulary mismatch between the mention context and the en- tity description cannot be alleviated in our method. Expanding the query to words outside the context could be an interesting research direction.
Phong Le and Ivan Titov. 2019. Distant learning for entity linking with automatic noise detection. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4081- 4090, Florence, Italy. Association for Computational Linguistics.
Haoran Li, Yangqiu Song, and Lixin Fan. 2022a. You don't 't know my favorite color: Preventing dialogue representations from revealing speakers' private per- sonas. arXiv preprint arXiv:2205. 10228.
Xiuxing Li, Zhenyu Li, Zhengyan Zhang, Ning Liu, Haitao Yuan, Wei Zhang, Zhiyuan Liu, and Jiany- ong Wang. 2022b. Effective few-shot named entity linking by meta-learning. In 2022 IEEE 38th Inter- national Conference Data Engineering (ICDE), pages 178-191, Los Alamitos, CA, USA. IEEE Com- puter Society.
A Acknowledgments
We thank Zifei Shan for discussions and the valu- able feedback. This work is jointly supported by grants: Natural Science Foundation of China (No. 62006061, 82171475), Strategic Emerging Industry Development Special Funds of Shenzhen (NNJJCY100201001114333336
Lajanugen Logeswaran, Ming-Woc Chang, Kenton Lee, Kristina Toutanova, Jacob Devlin, and Honglak Lee. 2019. Zero-shot entity linking by reading entity de- scriptions. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 3449-3460, Florence, Italy. Association for Computational Linguistics.
References
Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2021. Autoregressive entity retrieval. In International Conference on Learning Representa- tions.
Xinyin Ma. Yong Jiang, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, and Weiming Lu. 2021. MUVER: Improving first-stage entity retrieval with multi-view entity representations. In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2617-2624, Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
Kevin Clark, Minh-Thang Luong, Quoe V. Le, and Christopher D. Manning. 2020. Electra: Pre-train Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representa- tions.
Kelong Mao, Zhicheng Dou, and Hongjin Qian. 2022. Curriculum contrastive context denoising for few- shot conversational dense retrieval. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Reirieval, SIGIR '22, page 176-186, New York, NY, USA. As- sociation for Computing Machinery.
Yue Dong, John Wieting, and Pat Verga. 2022. Faithful to the document or to the world? mitigating hal- lucinations via -utity-linked knowledge in abstrac- tive summarization. Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1067-1082, Abu Dhabi, United Arab Emirates. Asso- ciation for Computational Linguisties.
Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Qifei Wu, Yuchen Ding, Hua Wu, Haifeng Wang. and Ji-Rong Wen. 2022. A thorough examination on zero-shot dense retrieval. CoRR, abs/2204.12733
Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2021. Splade: Sparse lexical and expan- sion model for first stage ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21, page 2288-2292 2292, New York, NY, USA. Association for Computing Machinery.
Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and be- yond. Found. Trends Inf. Reir, 3(4):333-389

---

Kai Sun, Richong Zhang. Samuel Mensah, Yongyi Mao, and Xudong Liu. 2022. A transformational biencoder with in-domain negative sampling for zero-shot entity linking. Findings of the Association for Compu- lational Linguistics: ACL 2022, pages 1449-1458, Dublin, Ireland. Association for Computational Lin- guistics.
MetaBLINK (Li et al., 2022b) proposes a weak supervision method to generate mention- entity pairs. These pairs are then used for training BLINK with a meta-learning mecha- nism.
DL4EL (Le and Titov, 2019) is a denoising method based on KL divergence to force the model to select high-quantity data. It can be combined with MetaBLINK''s generated data to train BLINK.
Weiqi Wang, Tianqing Fang. Baixuan Xu, Chun Yi Louis Bo, Yangqiu Song, and Lei Chen. 2023. Cat: A contextualized conceptualization and instanti- ation framework for commonsense reasoning. arXiv preprint arXiv:2305.04800
Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian
B Implementation Details
Riedel, and Luke Zettlemoyer. 2020. Scalable zero- shot entity linking with dense entity retrieval. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6397-6407, Online. Association for Computa- tional Linguistics.
Our model is implemented with PyTorch 1.10.0. The number of the model parameters is roughly 109M. All experiments are carried out on sin- gle NVIDIA A100 GPU. We use Adam opti- mizer (Kingma and Ba, 2015) with weight decay 0.01, learning rate to 2e-5. The batch size is and the number of extracted keywords (denoted k) is 32, For the convenience of com- parison, the number of final retrieved entitics (de- noted as n) is 64. The mention context length is set to 128, with each left and right context having a length of 64. For each domain, we train for total of 10 epochs, and choose the best checkpoint based the performance of development set. The whole training process takes about 1.5 minutes for each domain.
Mengzhou Xia, Mikel Artetxe, Jingfei Du, Dangi Chen, Veselin Stoyanov. 2022. Prompting EL ELECTRA: Few-shot learning with discriminative pre-trained models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1 1351-11361, Abu Dhabi, United Arab Emi- rates. Association for Computational Linguistics.
Peilin Yang, Hui Fang, and Jimmy Lin. 2017. Anserini: Enabling the use of lucene for information retrieval research. In Proceedings of the 40th International ACM SIGIR Conference on Research and Devel- opment in Information Retrieval, SIGIR '17, page 1253-1256, New York, NY, USA. Association for Computing Machinery.
In Table 2, the results of MUVER and GENRE are our reproduction based on their official Github repositories. These reproduction results, together with results of sparse retrievers, are from 5 runs of our implementation with different random seeds.
Baseline details
To evaluate the performance of our method, we compare with the following state-of-the-art retriev- ers that represent diverse array of approaches.
BM25 (Robertson and Zaragoza, 2009), a tra- ditional and strong sparse retrieval method. We consider two variants: BM25 (mention words), which simply uses the mention string as query, and BM25 (mention context), which uses all context query.
BLINK (Wu et al., 2020) adopts the bi- encoder architecture for dense retrieval.
MUVER (Ma al., 2021) extends the bi- encoder to multi-view entity representations and approximates the optimal view for men- tions via heuristic searching method.
GENRE (Cao al., 2021) an autoregressive retrieval method base on generating the title of candidates with constrained beam search.


Reviews:
[{"review_id": "C3JbwC9JUd", "reviewer": "Reviewer_2Pme", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and constructive feedback! We appreciate your positive feedback about our technical contributions and strong performance. \n\n**Regarding the experimental setup,** our intention was to maintain consistency with prior research efforts, specifically the work of Li et al. (2022) [1], which focused on four specialized domains within ZESHEL. This allows for a direct comparison of results.\n\nTo further validate the efficacy of our method, we conduct experiments in another specialized domain, i.e. **biomedical**. We utilize the NCBI Disease Corpus [2], a widely-used biomedical entity linking dataset. Employing the same few-shot setting, we randomly select 50 samples from the training set and 50 samples from the development set. The following table shows the recall@k results on the test set. Notably, our method consistently outperforms the strong baseline, BM25.\n\n\n|        | R@10 | R@20 | R@40 | R@50 | R@64 |\n|--------|------|------|------|------|------|\n| ours   | **62.86**| **67.83**| **71.85**| **72.70**| **73.23**|\n| BM25   | 62.33| 65.07| 70.26| 70.90| 71.74|\n| BLINK  | 11.98| 16.04| 20.10| 26.04| 28.02|\n\n\nPlease let us know if you have any additional concerns after you read our response.\n\nReferences:\n\n[1] Xiuxing Li, Zhenyu Li, Zhengyan Zhang, Ning Liu, Haitao Yuan, Wei Zhang, Zhiyuan Liu, Jianyong Wang. 2022. Effective Few-Shot Named Entity Linking by Meta-Learning. https://arxiv.org/abs/2207.05280\n\n[2] Rezarta Islamaj Doğan, Robert Leaman, Zhiyong Lu. 2014. NCBI Disease Corpus: A Resource for Disease Name Recognition and Concept Normalization. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3951655/", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed method is interesting and makes a lot of sense to me. The presentation is very clear and easy to follow. The improvement over other methods is big in the experiments.", "reasons_to_reject": "There is only one dataset, but I'll not consider this a big issue for a short paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a PLM-augmented sparse retriever and shows it performs much better than BM25 and dense retrievers in the setting of few-shot entity linking.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zBzUG3Urm8", "reviewer": "Reviewer_BHTM", "summary": "", "questions": "1. How much of the left and right context is considered for each mention? Is it the entire text before and after?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful review and valuable feedback! We appreciate your positive feedback and would like to address your concerns:\n \n- **The keyword extraction method seems to be expensive**: While it is true that the token prediction process costs some computational overhead during the inference phase, we would like to emphasize that the time dedicated to token prediction is relatively **small** compared to the overall inference time. For example, in the \"Star Trek\" domain, token prediction only accounts for approximately 5% of the total inference time. \n\n- **The evaluation is done on a subset of a single data set**: About the experimental setup, our intention was to ensure consistency with prior work by Li et al. (2022) [1], enabling a direct comparison of results. To further demonstrate the effectiveness of our method on other specialized domains, we conduct experiments on NCBI Disease Corpus [2], a widely-used **biomedical** entity linking dataset, under the same few-shot setting as Li et al. (2022). The following table presents the recall@k results on the test set. Our method consistently outperforms the strong baseline, BM25, and the dense retriever, BLINK.\n\n|        | R@10 | R@20 | R@40 | R@50 | R@64 |\n|--------|------|------|------|------|------|\n| ours   | **62.86**| **67.83**| **71.85**| **72.70**| **73.23**|\n| BM25   | 62.33| 65.07| 70.26| 70.90| 71.74|\n| BLINK  | 11.98| 16.04| 20.10| 26.04| 28.02|\n\n\n- **How much of the left and right context is considered for each mention**: The length of mention context is set to 128, while both the left and right contexts have a length of 64. We appreciate your suggestion and will add this in our implementation detail.\n\nIf there are any additional questions or concerns, we are happy to engage in further discussions.\n\nReferences:\n\n[1] Xiuxing Li, Zhenyu Li, Zhengyan Zhang, Ning Liu, Haitao Yuan, Wei Zhang, Zhiyuan Liu, Jianyong Wang. 2022. Effective Few-Shot Named Entity Linking by Meta-Learning. https://arxiv.org/abs/2207.05280\n\n[2] Rezarta Islamaj Doğan, Robert Leaman, Zhiyong Lu. 2014. NCBI Disease Corpus: A Resource for Disease Name Recognition and Concept Normalization. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3951655/", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The proposed method is effective and shows a significant boost in performance.", "reasons_to_reject": "1. The keyword extraction method seems to be expensive as it needs to predict a score for every token. This may increase the latency by a substantial amount as compared to BM25 and dense retrieval.\n2. The evaluation is done on a subset of a single data set. More extensive experiments on multiple data sets would help to demonstrate whether this method is effective across data sets/domains.", "questions_to_authors": "1. How much of the left and right context is considered for each mention? Is it the entire text before and after?", "paper_topic_and_main_contributions": "The paper proposes a way to improve the recall of candidate retrieval stage of entity linking. In addition to the surface text of mentions, it also uses denoised context for candidate retrieval. The context is denoised by extracting most relevant keywords to the mention. A distant supervision approach is applied to extract potential keywords for each mention by using word overlap between the mention’s surface text and entity descriptions. The evaluation is done on a subset of the ZeShEL data set with a few-shot settings. The experimental results suggest significant improvements over traditional BM25 and more recent dense retriever approaches.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "N0SEYcbQlB", "reviewer": "Reviewer_paUf", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and valuable feedback! We greatly appreciate your positive feedback about the motivation and technical contributions of our work.\n\n- **Some concepts need to be explained to ease the reading**: Thank you for the constructive suggestion. In the final version, we will ensure to add more comprehensive explanations about ELECTRA and the bi-encoder. Furthermore, we will emphasize the performance drop of the bi-encoder in data scarcity scenarios to highlight our motivation.\n- **Is the proposed model also effective on normal EL datasets**: Theoretically, our proposed model also works in the full-shot setting on normal EL datasets. In our preliminary study, we found that with an increase in training data, the performance of our method improves. We would like to emphasize that our work addresses data scarcity scenarios. Future work may extend the application of PLM-augmented sparse retrievers to the full-shot setting. \n\nIf you have any further questions or suggestions, please feel free to let us know.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "S1. The paper is well written and clearly organized. \n\nS2. The motivation is clear and the claim is well supported. \n", "reasons_to_reject": "W1. Some concepts need to be explained to ease the reading. For instance, it is unclear what is the bi-encoder and why it depends on the training data. Besides, it is unclear what ELECTRA is and how it works. \n\nW2. Is the proposed model also effective on normal EL datasets? \n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work addresses the retrieval of candidates problem in few-shot entity linking task. It offers a PLM based keyword extractor to construct a refined context and uses BM25 retrieval model to obtain the candidate entities. Experimental results validate that the proposed model can achieve state-of-the-art performance. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a method for enhancing the recall of the candidate retrieval phase in the entity linking process. It introduces a keyword extractor based on Pre-trained Language Models (PLM) to create a more refined context and utilizes the BM25 retrieval model for obtaining candidate entities.
Decision: Accept-Main