Learning to Rank Context for Named Entity Recognition Using a Synthetic Dataset
Authors: Arthur Amalvy
Vincent Labatut
Richard Dufour
Venue: EMNLP 2023 Main (2023)
Keywords: ner
transformers
context retrieval

Abstract: While recent pre-trained transformer-based models can perform named entity recognition (NER) with great accuracy, their limited range remains an issue when applied to long documents such as whole novels. To alleviate this issue, a solution is to retrieve relevant context at the document level. Unfortunately, the lack of supervision for such a task means one has to settle for unsupervised approaches. Instead, we propose to generate a synthetic context retrieval training dataset using Alpaca, an instruction-tuned large language model (LLM). Using this dataset, we train a neural context retriever based on a BERT model that is able to find relevant context for NER. We show that our method outperforms several retrieval baselines for the NER task on an English literary dataset composed of the first chapter of 40 books.

Learning to Rank Context for Named Entity Recognition Using a Synthetic Dataset
Arthur Amalvy
Vincent Labatut
Laboratoire Informatique d' Avignon arthur. a fr
Laboratoire Informatique d' Aviignn vincent wawatuteuuni--viigoono fr
Richard Dufour
Laboratoire des Sciences du Numérique de Nantes richard. derrdddduuuurrueee www.uuuuuuuiii-aaetee fr
Abstract
et al. (2022) for survey). Still, long sequences remain challenging (Tay et al., 2021).
While recent pre-trained transformer-base models can perform named entity recognition (NER) with great accuracy, their limited range remains an issue when applied to long docu- ments such whole novels. To alleviate this issue, solution is to retrieve relevant context at the document level. Unfortunately, the lack of supervision for such a task means one may have to settle for unsupervised approaches. In- stead, we propose to generate a synthetic con- text retrieval training dataset using Alpaca, instruction-ture tunne large language model (LLM). Using this dataset, we train neural context retriever based on a BERT model that is able to find relevant context for NER. We show that method outperforms several unsuper- vised retrieval baselines for the NER task on an English literary dataset composed the first chapter of 40 books, and that it performs par with re-rankers trained on manually annotated data, or even better.
Retrieval-basee methods circumvent these issues, by retrieving relevant context from the doc- ument and concatenating it the input. Unfor- tunately, no context retrieval dataset available for NER, and manually annotating such a dataset would be costly. This lack of data prevents from using a specialized supervised context retrieval method.
Mcanwhile, instruction learning has very re- cently seen an increasing focus from the NLP com- munity (Renze et al., 2023). Instructions-tunce models are able to solve variety of tasks (classi- fication, regression, recommendation..) provided they can be formulated text generation prob- lems. Inspired by the progress of these mod- els, we propose to generate synthetic context retrieval dataset tailored to the NER task using Alpaca (Taori et al., 2023), a recently released instruction-tuned large language model (LLM). Us- ing this synthetic dataset, we train a neural con- text retriever that ranks contexts according to their relevance to given input text for the NER task. Since applying this retriever on whole document costly, we instead use it re-ranker, by first retrieving set of candidate contexts using sim- ple retrieval heuristics and then keeping the best sentences among these.
1 Introduction
Named entity recognition (NER), as a fundamental building block of other natural language process- ing (NLP) tasks, has been the subject of a lot of re- searchers' attention. While pre-trained transformer- based models are able to solve the task with very high F1-score (Devlin et al., 2019; Yamada al., 2020), they still suffer from range limitation caused by the quadratic complexity of the attention mechanism in the input sequence length. When ap- plied to long documents such as entire novels, this range limitation prevents models from using global document- level context, since documents cannot be processed as a whole. However, lecument-level context is useful for entity disambiguation: for NER, lacking access to this context results in per- formance loss (Amalvy al.. 2023). To try and solve the range limitation of transformers, num- ber of authors recently explored "efficient trans- formers", with a sub-quadratic complexity (see Tay
In this article, we study the application of our proposed method on a corpus of literary novels, since the length of novels make them particularly prone to the range limitation of transformers. We first describe the generation process of our syn- thetic retrieval dataset. We then evaluate the in- fluence of the neural context retriever trained this dataset in terms of NER performance, by com- paring unsupervised retrieval methods. We also compare our re-ranker with other supervised re- re-rankers trained on a manually annotated con- text retrieval dataset unrelated to literary NER. We

---

explore whether the number of parameters of the LLM used for generating our synthetic context re- trieval dataset has an influence on the trained neural retriever by comparing Alpaca-7b and Alpaca- 13b, and the influence the number of candidate sen- tences to retrieve before re-ranking. Additionally, we study the influence of the context window (i.e. context retrieval range) the performance of the baseline retrieval heuristics and of our neural con- retriever. We release our synthetic dataset and our code under free license'.
Long document
Retrieval heuristic
Candidate
Input scutennce
couslead.
satlenass
Neural re-mankin
(CLS) Minas Tirth was impressive [SEP] Mines Tirih the Capiial of Gondor [SEP]
Jepur sentence with auginentedd
2 Related Work
Named entity recognition
2.1 Re-Rankers
B-LOC I-LOC
NER Annotations the input sentence
In the field of information retrieval, re-rankers used to complement more traditional retrievers such as BM25 (Robertson, 1994). In that setting, given a query and a set of passages, a traditional retriever is first used to retrieve a certain number of candidate passages. Then, a more computationally expensive re-ranker such as MonoBERT (Nogueira and Cho, 2020) Mono'T5 (Nogueira et al., 2020) retrieves the top-k passages among these candi- dates. While supervised re-rankers offer good per- formance, they necessitate training data, but anno- tation can be costly. To avoid this issue, we propose a re-ranker tailored to the NER task that is trained on a synthetic retrieval dataset.
Figure I: Overview of our neural re-ranker performing context retrieval.
2.3 Instruction-Follovii Large Language Models
Instruction-followinng LLMs are trained to output text by following user instructions. Multi-task learning (Wei et al., 2022; Sanh et al., 2022; Muen- nighoff et al., 2023) and fine-tuning on instructions datasets (Taori al., 2023; Chiang et al., 2023) are two common training paradigms for these models. One of the goals of these training methods is obtain good zero-shot performance for a variety of tasks, making fowlooiinn models very versatile.
2.2 Named Entity Recognition and Context Retrieval
Different external context retrieval techniques have been leveraged for NER (Luo et al., 2015; Wang et al., 2021; Zhang al., 2022). Comparatively, few studies focus on document-level context re- trieval, which can be used even when no exter- nal resources are available. Luoma and Pyysalo (2020) introduce majority voting to combine predic- tions made with different contexts, but their study is restricted to neighboring sentences and ignores lecument-level context.
Our synthetic dataset generation task calls for models producing longer outputs than multi-task learning based models such BloomZ mTO, which biased in favor of short outputs (Muen- nighoff et al., 2023). We therefore select Al- paca (Taori al., 2023), an instruction-tuned model based on Llama (Touvron et 2023).
3 Method
In this section, we start by introducing the document-level context retrieval task for NER. Then, since no dataset exists for this task, we de- tail how we generate synthetic context retrieval dataset using an coorructioo-ollowinng LLM in Sec- tion 3.2. We describe the neural context retriever that train using this dataset in Section 3.3. Fig- ure provides an overview of the process we pro- pose to retrieve context with our neural retriever. Finally, we provide details on our dataset in Sec- tion 3.4.
Recently, Amalvy et al. (2023) explore the role of global document-level context in NER. How- ever, their study only considers simple unsuper- vised retrieval heuristics due the lack of supervi- sion of the context retrieval task. By contrast, we propose to solve the supervision problem by gen- erating a synthetic context retrieval dataset using a LLM. This allows us to train a neural model that outperforms unsupervised heuristics.
hnttps://github... http://///..////////////////eeee

---

Description (all classes)
Prompt template '(INPUT SENTENCE}' In the preceding sentence, (ENTITY) is character. Invent a one-sentence description for this character, mentioning their name.
Example imput sentence [One-EyeJ's handicap in no way impairs his marvelous hindsight.
Example generated context One-Eye is a wise and mysterious character with a penchant for coming up with invaluable insights after the fact. Prompt template Invent single sentence depicting the character '(ENTITY)' performing an action, mentioning their name.
Action (PER only)
Example input sentence "It's my stomach, [Croaker]."
Example generated context Croaker was whistling a jaunty tune as he strolled through the Park.
Movement (LOC only)
Prompt template Invent single sentence depicting a character of your invention going to {ENTITY). You must mention the name of the character.
Example input sentence Lighining from clear sky smote the [Necropoli- tan Hill].
Example generated context The gothic vampire Count Necropolis tured Necropolitan Hill, his ancient stomping grounds.
Table 1: Prompts templates and examples of positive context retrieval samples generated by Alpaca-7b.
Positive examples swapping
Example input sentence We left in pretty good time and came after nightfall Klausenburgh
Example context Forley was adventurous and daring individual who was never afraid take risks
Negative sampling
Example input sentence 1 am afraid that have been tempted into too great length about the Italian Catherine; but in truth she has been my favourite.
Example context said Alice, as she swam about, trying to find her way out.
Table 2: Examples of negative context retrieval samples generated by positive examples swapping negative sampling.
3.1 Document-Level Context Retrieval
3.2 Context Retrieval Dataset Generation
Unfortunately, no context retrieval dataset exists for the task of retrieving relevant context for NER at the document level. Without supervision, we cannot train a supervised model to solve this task and are restricted to unsupervised retrieval methods. Therefore, we set out to generate synthetic dataset using Alpaca (Taori et al., 2023).
We define the document-level context retrieval problem as follows: given sentence 8; and its enclosing document D, we must retrieve S,, a set of relevant sentences in D; We define relevance as being helpful for predicting the entities in s; by allowing entity class disambiguation. After retriev- ing S, we concatenate its sentences to s, in order form a list. 5,, while keeping the relative ordering of sentences D,.
We define a context retrieval dataset as a of 3-tuples (s;, Sj. y), where:
s; is the input sentence (the sentence for which we wish retrieve context).
After context retrieval, we compute NER labels for sentence s; by inputting S into a NER model and keeping only the labels related to 8;.
Sj is the retrieved context sentence.
y is the relevance of 8, with respect to s;:

---

either 1 if sj is relevant, or 0 otherwise.
not contain an entity from 8;. Note that all context sentences sj contain an entity. We use this additional generation technique because we found that generating examples using only negative sampling led to the model overfitting on contexts containing entities (since negative sampling does not guarantee that the retrieved context contains an entity).
Depending on whether the example is positive (y 1) or negative (y/ = 0), we design different generation methods.
Positive Examples For each NER class in our dataset (PER (person), LOC (location) and ORG (or- ganization)), we empirically observe which types of sentences can help for disambiguation with other classes. We determine that the following categories sentences are useful:
Generated Dataset We generate two context re- trieval datasets: one with Alpaca-7b, and one with Alpaca- 133. They contain respectively 2,716 and 2,722 examples? Interestingly, the models display knowledge of some entities from the dataset when generating samples, probably thanks their pre- training. For example, when asked to generate a context sentence regarding Gandalf the Grey from The Lords of the Rings, an Alpaca model incorpo- rates the fact that he is a wizard in the generated context even though it does not have this informa- tion from the input text.
For all classes (PER, LOC and ORG): descrip- tions of the entity explicitly mentioning it (Description).
For the PER class only: sentences describing the PER entity performing an action (Action).
For the LOC class only: sentences describ- ing PER entity going to the LOC entity (Movement).
We design a prompt for each of these types of sentences that, given an imput sentence and an en- tity, can instruct Alpaca to generate sentence of this type. Table show these prompts and some example sentences generated by Alpaca-7b. Given a sentence s; from our NER training dataset, we select an entity from 8; and generate 8, by instruct- ing Alpaca with one of our prompts. This allows us to obtain a positive context retrieval example (8), S5. 1). We repeat this process on the whole NER training dataset. To avoid overfitting on the most frequent entities, generate exactly ample per unique entity string in the dataset. We filter sentences 8j that do not contain the target entity string from the input sentence 8,.
3.3 Neural Context Retriever
For our neural context retriever, we use a BERT model (Devlin et al., 2019) followed by regres- sion head. For given sentence and a candidate context, the retriever outputs the estimated rele- vance of the candidate context between 0 and 1. Because applying our model the whole docu- ment is computationally costly, use our retriever as re-ranker: we first retrieve some candidate contexts using simple heuristics (cf. Section 4.2) before re-ran re-ranking these sentences.
3.4 Dataset
Negative Examples We use two different tech- niques to generate negative examples:
We use the English NER literary dataset constituted by Dekker al. (2019) and then corrected and en- hanced by Amalvy et al. (2023). We chose this dataset specifically for the length of its documents. is composed of the first chapter of 40 novels, and has 3 NER classes: Person name (PER), Location (LoC) and Organization (ORG). We split each doc- ument in sentences. To perform context retrieval the entire novels, also collect the full text of each novel.
Negative sampling: given sentence 8; from document D,, sample irrelevant sen- tence by randomly selecting a sentence from another document Dj. Negative sampling gen- erates some contexts that contain entities, and some contexts that do not.
Positive examples swapping: given an exist- ing positive example (si, Syz 1), we generate negative example (si, Sk: 0) by replacing 8) with a context 8 from another positive exam- ple. Since we generate a single positive ex- ample per entity string, 85 has a high chance
Thhe number examples between the two datasets is different bocause of the filtering process when generating positive examples. This filtering occurred very rarely, as the vast majority of generated sentences included the target entity.

---

4 Experiments
4.3 Re-ranker Baselines
We compare our system with the following trivial re-rankers baselines:
4.1 Unsupervised Retrieval Baselines
We compare the performance of our neural context retriever with the following unsupervised baselines from Amalvy et al. (2023):
random re-ranker: We retrieve sen- tences with all the unsupervised heuristics (as described in Section 4.2), and then randomly select l sentences from these candidates.
The model performs the NER task on each sentence separately, without ad- ditional context.
retrieval:
bucket random re-ranker: Same above, except we select k/4 sentences for each heuristic.
Retrieves sentences that are just before and after the input sentence.
surrounding:
These baselines allow to observe the real ben- efits of re-rankers. We also compare our retriever with supervised re-rankers trained on the MSMarco passage retrieval dataset (Bajaj et al., 2018):
bm25: Retrieves the most similar sentences according to BM25 (Robertson, 1994).
bm25+monobert: We retrieve 16 sentences using BM25, and then use the MonoBERT- large (Nogueira and Cho, 2020) re-ranker to retrieve the most relevant sentences. This configuration simulates a classic information retrieval setup.
samenoun: Retrieves & sentences that contain at least noun in common with the input sen- tence. We identify nouns using NLTK (Bird et al., 2009) done by Amalvy et al. (2023).
4.2 Inference
all+monobert: We retrieve 4n sentences us- ing all the unsupervised heuristies (as de- scribed in Section 4.2), and then use the MonoBERT-large re-ranker to retrieve the k most relevant sentences. This configuration is the closest to our neural retriever, so we use it to compare the influence of the training dataset.
Contrary to Amalvy et al. (2023), we retrieve con- text from the whole document instead of the first chapter only. We use our neural context retriever re-ranker. For each sentence 8; of the dataset, we first retrieve a total of 4n candidate context sentences using the following heuristies:
sentences using the bm25 heuristic.
bm25+monot5: Same as bm25+monobert, but using a MonoT5 reranker (Nogueira et al., 2020).
sentences using the samenoun heuristic.
The sentences that are just before the current
all+monotss Same as all+monobert, but ing MonoT5 re-ranker.
sentence.
4.4 Training
The sentences that are just after the current sentence.
We train the neural context retriever on the syn- thetic dataset we generated as described in Sec- tion 3.2 for 3 epochs with a learning rate x 10 5. To study the importance the LLM size that we use when generating our synthetic dataset, we generate our context retrieval dataset with two differently sized versions of the same model; as de- tailed in Section 3.2: Alpaca-7B (7 billion parame- ters) and Alpaca- (13 billion parameters) (Taori al., 2023).
Note that the candidate sentence be re- trieved by different heuristics: avoid redundancy, we filter out repeated candidates. We experiment with the following values of n {4, 8, 12, 16, 24).
After retrieving 4n candidate contexts, we com- pute their estimated relevance with respect to the input sentence using neural context retriever. We then concatenate the top-k contexts to the input sentence s; before performing NER prediction. We report results for number of retrieved sentences from to 83.
NNote that the samenoun heuristic is an exception, as it may retrieve fewer than & sentences. This is because some input sentences have fewer than contexts in the document with at least one common noun.

---

+ alpaca-7b 7b
I
pacca-a 13b
8
12
16
24
0.92
0.91
2 3
7 8
2 3
6
7
8
2 3
7 8
2 3
6 7 8
2 3
6 7 8
Number of retrieved sentences k
Figure 2: Efffec of the number candidate sentences 4n on the performance of our neural context retriever, trained using dataset generated with Alpaca-7b or Alpaca-13b.
4.5 Named Entity Recognition Model
Number of candidate contexts We find that 4 is too low, probably because not enough relevant candidate contexts are retrieved to obtain good performance. Increasing n leads to better performance for high values of the number of re- trieved sentences However, the best performance obtained for values of between and 5.
In all of our experiments, we use a pretrained BERT model (Devlin et al., 2019) followed by a classifi- cation head fine-tuned for 2 epochs with a learning rate of 2 X 10-5, We use the bert-base-cased checkpoint from the huggingface transformers library (Wolf al., 2020).
LLM model size We find that the size of the Al- paca model (7b versus 13b) has very little influence on the final NER performance for all surveyed val- ues of n.
4.6 Evaluation
We compute the F1-score of our different configurations using the default mode of the seqeval (Nakayama, 2018) library for repro- ducibility. We perform all experiments 5-folds, and report the mean of each metric on the folds. To compare the different retrieval configurations fairly, we train a single NER model and compare results using this model and different retrieval methods. For retricval methods that are not deterministic (some re-rankers and the samenoun heuristic*), we report the mean of 3 runs account for variations between runs.
In the rest of this article, we discuss the best configuration found in this experiment unless specified otherwise: the neural retriever trained using Alpaca-7b and with = and A = 3. We also keep the same configuration for the supervised baselines. With that configuration, our neural re- ranker has a Fl of 98. 01 on the evaluation portion of our synthetic context retrieval dataset.
5.2 Comparison with Unsupervised Retrieval Methods
Figure 3 shows the Fl of our NER model when using re-ranker different unsupervised trieval methods. Retrieval with any method is beneficial (except when retrieving sentences or more with surrounding or bm25), but our neu- ral re-ranker generally outperforms all the simple heuristics. Specifically, it beats the no retrieval configuration by around 1 Fl point, with its peak performance being when the number of retrieved sentences k 3.
5 Results
5.1
Neural Re-Ranker Configuration Selection
this section, we set out to find the optimal config- uration of our neural context retriever. We survey the number of candidate contexts 4n to retrieve be- fore re-ranking, and the size of the model. Figure 2 shows the NER performance of the neural retriever trained with either Alpaca-7b or Alpaca-131 for different values of
Per-book results Figure 4 shows the Fl score per book for our retrieval method and the unsupervised retrieval methods, for single retrieved sentence. Our neural context retriever is better or on-par with
'The samenoun method is not deterministic since it ran- domly retrieves among candidate sentences that have a com- mon noun with the input sentence.

---

Input sentence
Candidate contexts
Score
"The Ministry of Truth Minitrue, in Newspeak (Newspeak was the official lan- guage of Oceania."
"The Ministry of Truth, which concerned itself with news, entertainment, education, and the fine arts."
1.0
"Winston made for the stairs."
0.0
"The eldest of these, and Bilbo's 's favourite, was young Frodo Baggins."
"Then he disappeared inside with Bilbo, and the door was shut." "
1.0
f I need a holiday, very long holiday, as have told you before.
0.0
Table 3: Example predictions of the neural context retriever.
5.4 Size of the Context Window
no retrieval
samenoun
surrounding
neural (our)
To understand the role of the size of the context window, we compare between retrieving context in the first chapter of each book only as (Amalvy et al., 2023) and retrieving context on the entire book. Figure 6 shows the performance of the bm25, samenoun and neural configurations depending on the context window. For the neural configu- ration, since the number of retrieved sentences 4n may have influence the results, report performance with values of in {4, 8, 12).
bm25
0.925
0.920
0.915
0.910
0.905
6
8
Number retrieved sentences
bm25 Retrieving a few sentences from the cur- rent chapter seems more beneficial to the bm25 heuristic. However, performance decreases sharply when retrieving more than 2 sentences in the same chapter. This effect is weaker when retrieving con- text in the whole book, which seems indicative of saturation issue, where the number of helpful sen- tences that can be retrieved using bm25 in single chapter low.
Figure 3: NER Fl score comparison between our neural retriever and unsupervised retrieval methods.
other configurations 50% of the novels (20 out 40). Performance varies lot depending on the novel, with the highest improvement of the neu- ral method over the no retrieval configuration being 8.1 Fl points for The Black Company.
The performance of the samenoun heuristic seems to follow the same pattern as bm25. Retrieving few sentences from the current chap- ter seems better than doing so in the full document. This might be because the current chapter has higher chance of containing sentences talking about an entity of the input sentence. Meanwhile, is better to retrieve contexts in the whole book for val- ues > 4, possibly because such a number retrieved sentences means the heuristic has high enough chance of retrieving relevant sentence at the book level.
samenoun
5.3 Comparison with Re-Rankers
We compare the Fl of our neural re-ranker with other re-rankers trained MSMarco (Bajaj et al., 2018) in Figure All the supervised methods outperforms the random re-rankers, showing that they can indeed retrieve useful context. Our model outperforms the other supervised methods, albeit slightly. While results are close, this is still interest- ing we used generated data only while the other supervised approaches were trained with gold anno- tated data. We note that using multiple pre-retrieval heuristics (the all+ configurations and our neural re-ranker) is generally better than using only BM225, which tends to show that retrieving sentences with BM25 alone may lead to missing important context sentences.
neural The context window seems critical for the performance of the neural retriever. Retrieving con- text only in the current chapter suffers from a large performance drop compared to retrieving context in the whole book. This is true even for different

---

no retrieval
bm25
neural (our)
surrounding
samenoun
1.0
F1 score
0.8
1984
AGameOrThroud
AStudy AStudyInScarie In
AliceIn oaderlande land.
AssassinaA Aesasinsspprront pprentiio
Prick
Bra New World
DaviaClopperton
Dracula
Elentris
Emma
Frankenstedd
CardensOFThehh Oohenaooooooooo
Harry Harry Potter
Huckleberry becklbbrryyPe inn.
MokyllAndllydd
Magician
Mistborn
Moby
Ollver
WrideAndPrejuut
StormFront
TheBlack ComakCComann
TheBladel sesee
TheCallOrThee Peelorrhhheee Wild Wild
The ColloooooooMnnni Ofdaniic
ThoCount ONMonte Cristo
Thelooloosbipoo hww.owswbwohhhheeeeeoeooo webipoothhennnng
TheGunslinge
TheLive O FLodoLLooo
TheN Of Wind
Thel'aintedde Lan
The ThrTheeeMMuee MuskeMukketeot
TheWayO WaWWaOKKiigg OfKingg
Way OIS hoShddwws
Wel.ive five New
TheWheel Of Time
Tinker www.Teooooooeeeeeereeeeoo SoldeieSSpprr
Uysses
Vanity
The Way Wel Wel
Figure 4: NER Fl Score per book for different retrieval methods, with number of retrieved sentences
retrieval +mooobrt
all monot5
that newly formed input. Future works may expand on this article by exploring whether taking these interactions into account can have positive impact on performance.
bucket random rerauker
all monobert
random reranker
bm25 monot5
neural (our)
0.925
We also note that our proposed method, gen- erating retrieval dataset using instructions- tuned LLM, could be leveraged for other NLP tasks where global document-level retrieval is ful. The only pre needed to generate such dataset is the knowledge of which types of context sam- ples can help to improve the performance for a local task, in order to generate positive samples. Future works may therefore focus on applying this principle on different tasks.
0.920
0.915
2
5
Number of retrieved sentences
k
Figure 5: NER F1 score comparison between our re- triever and different re-ranking methods.
Limitations
values of the number of retrieved sentences 4n.
7.1 Candidate Contexts' Interactions
6 Conclusion
Our proposed neural context retriever does not consider the relations between retrieved sentences. This can lead to at least two issues:
In this article, we proposed to train neural context retriever tailored for NER on synthetic context re- trieval dataset. Our retrieval system can be used to enhance the performance of NER model, achiev- ing a gain of around 1 Fl point over a raw NER model and beating unsupervised retrieval heuristics. We also demonstrated results on par or better than re-rankers trained using manually annotated data.
When the model is asked to retrieve several context sentences, can return sentences with redundant information. This makes the input of the NER task larger, and therefore the in- ference slower, for no benefit.
As stated in Section 7, limitation of our neu- ral context retriever is that the relevance of each candidate context sentence estimated separately. To account for potential interactions, we could it- cratively add each candidate to the input sentence and rank the remaining candidates with respect to
If an input sentence contains two entities that need disambiguation, the model might retrieve several examples related to one the entities only.

---

book
chapter
bm225
salnenoun
neural (our)
0.92
0.920
0.92
E1
P1
0.915
0.90
0.90
0.910
7
2 3
5
1
2
3
4
5
6
8
Number retrieved sentences lc
Figure 6: NER performance of different retrieval methods for different context window sizes as function of the number of retrieved sentences For the neural configuration, we report performance for values of {4, 8, 12].
7.2 Computational Costs
1. Stoica, and E. P. Xing. 2023. Vicuna: An Open- Source Chatbot Impressing GPT-4 with 90%* Chat- GPT Quality.
Our context retrieval approach extends the range transformer- based models by avoiding the quadratic complexity of attention in the sequence length. However, retrieval in itself still bears a computational cost that increases linearly in the number of sentences of the document in which it is performed. Additionally, retrieving more context sentences also increases the computational cost of predicting NER labels.
N. Dekker, T. Kuhn, and M. van Erp. 2019. Evalu- ating named entity recognition tools for extracting social networks from novels. PeerJ Computer Sci- ence, 5:e
Devlin, M. Chang, K. Lee, and K. Toutanova. 2019. BERT: Pre-training of deep bidirectional transform- ers for language understanding. In Conference of the North American Chapter the Association for Computational Linguistics: Human Language Tech- nologies, volume 1, pages 4171 1-16..
7.3 Biases
LLMs are prone to reproducing biases found in the training data (Sheng al., 2019). The origi- nal Llama models, which are used as the basis for fine-tuning Alpaca models, display a range of bi- ases (about gender, religion, age...) (Touvron et al., 2023) that could bias the context retrieval dataset. This, in turn, could bias the neural context retriever model trained on that dataset.
G. Luo, X. Huang, C. Lin, and Z. Nie. 2015. Joint entity recognition and disambiguation. In 2015 Con- ference on Empiricul Methods in Natural Language Processing, pages 879-888.
Luoma and S. Pyysalo. 2020. Exploring cross- sentence contexts for named entity recognition with BERT. 28th International Conference on Compu- tational Linguistics, pages 904-914.
N. Muennighoff, T. Wang, L. Sutawika, A. Roberts,
S. Biderman, T. Scao, M. S. Bari. S. Shen, Z. X. Yong. H. Schoelkopf, X. Tang, D. Radev, A. F. Aji. K. Almubarak, S. Albanie, Z. Alyafeai, A. Webson, E. Raff, and C. Raffel. 2023. Crosslingual gener- alization through multitask finetuning. In 61st An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1,5911- 16111.
References
Amalvy, Labatut, R. Dufour, 2023. The role of global and local context in named entity recogni- tion. In 61st Annual Meeting of the Association for Computational Linguistics.
P. Bajaj. D. Campos, N. Craswell, L. Deng, J. Gao. X. Liu, R. Majumder, McNamara, B. Mitra, T. Nguyen, M. Rosenberg, X. Song, A. Stoica, S. Ti- wary, and T. Wang. 2018. Ms marco: A human generated machine reading comprehension dataset. arXiv, cs.CL:1611 (99268.
H. Nakayama. 2018. seqeval: A python framework for sequence labeling evaluation.
R. Nogueira and K. Cho. 2020. Passage re-ranking with bert. arXiv,
S. Bird, E. Loper, and Klein, 2009. Natural Language Processing with Python. O'Reilly Media Inc.
R. Nogueira, Z. Jiang, Pradeep, and Lin. 2020. Document ranking with a pretrained sequence-to- sequence model. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 708-718.
W. Chiang. Z. Li, Z. Lin, Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez,

---

L. Renze, Z. Kai, and Y. Wenpeng. 2023. Is prompt all you need? no. a comprehensive and broader view of instruction learning. arXiv, cs.CI.2303. 10475
Davison, Shleifer, P. von Platen, Ma, Y. Jer- nite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Con- ference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45.
S. E. W. Robertson. 1994. Some simple effective ap- proximations to the 2-poisson model for probabilistic weighted retrieval. In SIGIR '94, pages 232-241.
Yamada, A. Asai, H. Shindo, H. Takeda, and Y. Mat- sumoto. 2020. LUKE; Deep contextualized entity representations with entity-aware self-attention. In Conference Empirical Methods in Natural Lan- guage Processing, pages 6442-644 22-44
V. Sanh, A. Webson, Raffel, S. H. Bach, L.. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, T. L. Scao, A. Raja, M. Dey, M. S. Bari, C. Xu, U. Thakker, S. S. Sharma, E. Szczechla, T. Kim, G. Chhablani, N. Nayak, D. Datta, J. Chang. M. Jiang, H. Wang. M. Manica, S. Shen, Z. X. Yong, H. Pandey, R. Baw- den, T. Wang. T. Neeraj, J. Rozen, A. Sharma, A. San- tilli, T. Fevry, J. A. Frics, R. Techan T. Bers, S. Bi- derman, L. Gao, T. Wolf, and A. M. Rush. 2022. Multitask prompted training enables zero-shot task generalization. 10th Internation Conference on Learning Representations.
X. Zhang. Y. Jiang, X. Wang. X. Hu, Y. Sun, P. Xie, and M. Zhang. 2022. Domain-specifte NER via retrieving correlated samples. In 29th International Conference on Computational Linguistics, pages 2398-2403
A Comparison with Unsupervised Retrieval Methods: Precision/Recall Breakdown
E. Sheng. K. Chang, P. Natarajan, and N. Peng. 2019. The woman worked as a babysitter: biases lan- guage generation. In 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3407-3412.
no retrieval
samenown
surrounding
neural (our)
bm25
R. Taori. I. Guirajani. T. Zhang. Y. Dubois, X. Li, C. Guestrin, Liang, and B. Hashimoto. 2023. Stanford alpaca: An instruction-followin Ilama model.
0.925
0.920
Precision
Y. Tay, M. Dehghami, Abnar, Y. Shen, D. Bahri, P. Pham, J. Rao, L. Yang, S. Ruder, and D. Metzler. 2021. Long range arena: A benchmark for efficient transformers. 10th International Conference on Learning Representations.
0.915
0.910
2
4
5
6
7
8
Y. Tay, M. Dehghani, D. Bahri, and D. Metzler. 2022. Efficient transformers: A survey, ACM Computing Survey, 55(6).
Number of retrieved sentences k
retrioval
samenoun
surrounding
neural (our)
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. 2023. Llama: Open and efficient foundation language models. arXiv, cs. CL:23022,13777
bm25
0.92
Recall
X. Wang, Y. Jiang, N. Bach, T. Wang, Z. Huang. F. Huang, and K. Tu. 2021. Improving named en- tity recognition by external context retrieving and cooperative learning. In 59th Annual Meeting of the Association for Computational Linguistics and 1 lth International Joint Conference on Natural Language Processing, volume 1, pages 1800-1812.
0.91
2
4
7
Number of retrieved sentences A
J. Wei. M. Bosma, V. Zhao, K. Guu, A. Wei Yu, B Lester,
Figure 7: NER Precision (top) and Recall (hottom) com- parison between neural retriever and unsupervised retrieval methods.
N. Du, M. Dai, and Q. Le. 2022. Finetuned language models are zero-shot learners. In 10th In- ternational Conference Learning Representations.
T. Wolf, L. Debut, V. Sanh, Chaumond, Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz,
As seen in the bottom plot of Figure 7, unsuper- vised retrieval methods and our neural re-ranker

---

all benefits recall (except surrounding when the number of retrieved sentences k 8), but, as ob- served in the top plot, they can all have a negative impact on precision depending on the value of


Reviews:
[{"review_id": "IsyHrSviu0", "reviewer": "Reviewer_YYw4", "summary": "", "questions": "Do you plan to share the proposed dataset with the public? If so, what would be the license of your dataset?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThank you for your review! We tried to answer to your remarks as best as we could below:\n\n> Do you plan to share the proposed dataset with the public? If so, what would be the license of your dataset?\n\nAs indicated in the article, we release the generated datasets. You can find these datasets in the anonymized repository under _runs/gen/genv3_ (Alpaca 7b) and _runs/gen/genv3\\_13b_ (Alpaca 13b). Datasets are under CC BY-NC 4.0 license. As pointed by reviewer NDUN, we also released code and instructions to replicate all of our experiments.\n\n\n> The proposed model outperformed the baselines; however, these baselines are not strong enough to justify the superiority of the proposed model. The author should compare the final NER performance with other methods to claim that the proposed method significantly improves performance in this area.\n\nWe performed additional experiments on other baselines to answer this concern. We compared our re-ranker to a pretrained MonoBERT re-ranker (Nogueira and Cho, 2019). We chose MonoBERT since we wanted to compare to a model with a similar architecture in order to study the difference of performance caused by the training dataset. Still, note that the version of MonoBERT we employed is based on BERT large (huggingface checkpoint: `castorini/monobert-large-msmarco`), while our model is based on BERT base.\n\nWe experimented with two variants of a MonoBERT based system:\n\n- bm25 -> MonoBERT, to replicate a classic BM25 + reranker baseline\n- [bm25, samenoun, surrounding] -> MonoBERT with $n=8$, to compare with the neural setup of our article (as in paragraph Inference of Section 4.1)\n\nWe obtained the following mean results on 3 runs (F1-score):\n\n| System             /                 k    |     1 |     2 |     3 |     4 |     5 |     6 |     7 |     8 |\n| ----------------------------------------- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |\n| bm25 -> MonoBERT                          | 92.18 | 91.91 | 91.93 | 91.98 | 91.90 | 91.58 | 91.65 | 91.44 |\n| [bm25, samenoun, surrounding] -> MonoBERT | 92.23 | 92.16 | 92.27 | 92.20 | 92.16 | 91.87 | 91.56 | 91.47 |\n| our                                       | 92.34 | 92.30 | 92.45 | 92.36 | 92.30 | 92.19 | 92.10 | 91.72 |\n\nOverall, our model obtains better performance for all values of $k$, even with MonoBERT having more parameters than our model. Our hypothesis is that the performance differences come from our retrieval dataset being specifically suited for NER and for the domain at hand. We will include these results in a revised version of the article.\n\n\nPlease feel free to let us know if you have further questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Data augmentation with LLM is an interesting/promising direction that can be applied to many NLP downstream. Good to see research that applies it to the NER domain.\n", "reasons_to_reject": "The authors propose a method to generate a trainset from LLM; however, there is no analysis of the quality of the generated samples. \n\nThe proposed model outperformed the baselines; however, these baselines are not strong enough to justify the superiority of the proposed model. The author should compare the final NER performance with other methods to claim that the proposed method significantly improves performance in this area.\n", "questions_to_authors": "Do you plan to share the proposed dataset with the public? If so, what would be the license of your dataset?", "paper_topic_and_main_contributions": "This research focuses on retrieving context for the NER task. In particular, it leverages the large language (LLM) model (i.e., alpaca) to generate supervision to train the context retrieval model. The proposed neural context retriever with several simple baselines.\n", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KyC7XjFvSs", "reviewer": "Reviewer_1ozW", "summary": "", "questions": "1) Why is there no comparison with other retrieval systems, like using a simple BM25 + T5 re-ranker or unsupervised retrieval systems?\n2) What model is used for the final NER task? Is it also the BERT model?\n3) Why did you write a long paper when you have to use up two pages for simple examples or redundant graphs? Even Amalvy et al. 2023 is a short paper.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThank you for your review! We tried to answer to your remarks as best as we could below:\n\n> Q1: Why is there no comparison with other retrieval systems, like using a simple BM25 + T5 re-ranker or unsupervised retrieval systems?\n\nWhile we included some baselines in our article, we performed additional experiments to address this concern. We compared our re-ranker to a pretrained MonoBERT re-ranker (Nogueira and Cho, 2019). We chose MonoBERT since we wanted to compare to a model with a similar architecture in order to study the difference of performance caused by the training dataset. Still, note that the version of MonoBERT we employed is based on BERT large (huggingface checkpoint: `castorini/monobert-large-msmarco`), while our model is based on BERT base.\n\nWe experimented with two variants of a MonoBERT based system:\n\n- bm25 -> MonoBERT, to replicate a classic BM25 + reranker baseline\n- [bm25, samenoun, surrounding] -> MonoBERT with $n=8$, to compare with the neural setup of our article (as in paragraph Inference of Section 4.1)\n\nWe obtained the following mean results on 3 runs (F1-score):\n\n| System             /                 k    |     1 |     2 |     3 |     4 |     5 |     6 |     7 |     8 |\n| ----------------------------------------- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |\n| bm25 -> MonoBERT                          | 92.18 | 91.91 | 91.93 | 91.98 | 91.90 | 91.58 | 91.65 | 91.44 |\n| [bm25, samenoun, surrounding] -> MonoBERT | 92.23 | 92.16 | 92.27 | 92.20 | 92.16 | 91.87 | 91.56 | 91.47 |\n| our                                       | 92.34 | 92.30 | 92.45 | 92.36 | 92.30 | 92.19 | 92.10 | 91.72 |\n\nOverall, our model obtains better performance for all values of $k$, even with MonoBERT having more parameters than our model. Our hypothesis is that the performance differences come from our retrieval dataset being specifically suited for NER and for the domain at hand. We will include these results in the a revised version of the article.\n\n\n> Q2: What model is used for the final NER task? Is it also the BERT model?\n\nOur model consists of a BERT encoder followed by a classification layer that we finetune on the NER task (see Section 4.2).\n\n\n> Q3: Why did you write a long paper when you have to use up two pages for simple examples or redundant graphs? Even Amalvy et al. 2023 is a short paper.\n\nWe were not trying to transform a short paper into a long paper by adding redundant graphs. We chose to include example tables because we think it benefits the reader's comprehension of the article, since the dataset generation is not straightforward. Additionally, removing these figures would not transform the current paper into a short one.\n\n\n> There should be another subsection in the Related works section to describe other re-rankers like T5 re-ranker etc.\n\nWe agree this warrants a subsection in the related works, and we will include it in a revised version of the article.\n\n\n> The authors should also discuss about this work: Improving the performance of Transformer Context Encoders for NER\n\nAs we understand it, this article proposes several enhancements to NER systems (data augmentation, usage of a character-level encoder), but is not concerned with retrieval or the influence of long-range context in NER.\n\n\n> The authors should include an ethics statement section in the paper.\n\nTo our understanding the ethics statement is not mandatory, and we did not see a reason to include one. Do you think there are specific concerns about our work that we should address in such a section?\n\n\n> The authors should explain explicitly that \"no retrieval\" means the NER model is not given any context and that it is a simple baseline (line 261).\n\nWe corrected the article and added it explicitly.\n\n\nPlease feel free to let us know if you have further questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) Novel idea of training a supervised context retriever for NER using synthetic data that the authors generated.\n2) The authors showed that their retriever outperforms other unsupervised retrievers like BM25.", "reasons_to_reject": "1) The paper seems to simply fine tuned a BERT model and used it as a re-ranker. The authors didn't even compare their results with other re-ranker models. (Update: authors compared the results with mono-BERT and T5)\n2) This paper should be written as a short paper only, the authors appears to have included example tables and redundant graphs on page 3 and 7 so as to meet the long paper requirements.\n3) As the authors are using a simple retrieval heuristics and then using their retriever as a re-ranker it will always be dependent on heuristics first and hence it is essentially a simple retriever plus re-ranker system like BM25+T5 re-ranker.", "questions_to_authors": "1) Why is there no comparison with other retrieval systems, like using a simple BM25 + T5 re-ranker or unsupervised retrieval systems?\n2) What model is used for the final NER task? Is it also the BERT model?\n3) Why did you write a long paper when you have to use up two pages for simple examples or redundant graphs? Even Amalvy et al. 2023 is a short paper.", "paper_topic_and_main_contributions": "The paper proposes a way to do supervised training of retrievers (pre-trained transformer-based) using relevant context from long documents for the NER task. Their main contributions are :\n1) Proposed to generate a synthetic context retrieval training dataset using Alpaca.\n2) Proposed to use this synthetic dataset for supervised training of neural context retriever based on the BERT model.\n3) The authors showed that their retriever outperforms other unsupervised retrievers like BM25.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Justification_for_Ethical_Concerns": "-"}, "extra_text": {"Missing_References": "There should be another subsection in the Related works section to describe other re-rankers like T5 re-ranker etc.\n\nT5 re-ranker reference:\n@misc{hui2022retrieval,\n      title={Retrieval Augmentation for T5 Re-ranker using External Sources}, \n      author={Kai Hui and Tao Chen and Zhen Qin and Honglei Zhuang and Fernando Diaz and Mike Bendersky and Don Metzler},\n      year={2022},\n      eprint={2210.05145},\n      archivePrefix={arXiv},\n      primaryClass={cs.IR}\n}\n\nThe authors should also discuss about this work, \n@article{Chawla2021ImprovingTP,\n  title={Improving the performance of Transformer Context Encoders for NER},\n  author={Avi Chawla and Nidhi Mulay and Vikas Bishnoi and Gaurav Dhama},\n  journal={2021 IEEE 24th International Conference on Information Fusion (FUSION)},\n  year={2021},\n  pages={1-8},\n  url={https://api.semanticscholar.org/CorpusID:244841717}\n}", "Typos_Grammar_Style_and_Presentation_Improvements": "1) The authors should include an ethics statement section in the paper. \n2) The authors should explain explicitly that \"no retrieval\" means the NER model is not given any context and that it is a simple baseline (line 261)."}}, {"review_id": "KscIWNCRrr", "reviewer": "Reviewer_SivA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThank you for your review! We tried to answer to your remarks as best as we could below:\n\n> The proposed method employs the neural retriever as a re-ranker to find top-k sentences from a candidate set. However, all the baselines have only a retrieval process without re-ranking. I suggest adding baselines that are equipped with a re-ranking stage, e.g., BM25 + TextRank.\n\nWe performed additional experiments on other baselines to answer this concern. We compared our re-ranker to a pretrained MonoBERT re-ranker (Nogueira and Cho, 2019). We chose MonoBERT since we wanted to compare to a model with a similar architecture in order to study the difference of performance caused by the training dataset. Still, note that the version of MonoBERT we employed is based on BERT large (huggingface checkpoint: `castorini/monobert-large-msmarco`), while our model is based on BERT base.\n\nWe experimented with two variants of a MonoBERT based system:\n\n- bm25 -> MonoBERT, to replicate a classic BM25 + reranker baseline\n- [bm25, samenoun, surrounding] -> MonoBERT with $n=8$, to compare with the neural setup of our article (as in paragraph Inference of Section 4.1)\n\nWe obtained the following mean results on 3 runs (F1-score):\n\n| System             /                 k    |     1 |     2 |     3 |     4 |     5 |     6 |     7 |     8 |\n| ----------------------------------------- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |\n| bm25 -> MonoBERT                          | 92.18 | 91.91 | 91.93 | 91.98 | 91.90 | 91.58 | 91.65 | 91.44 |\n| [bm25, samenoun, surrounding] -> MonoBERT | 92.23 | 92.16 | 92.27 | 92.20 | 92.16 | 91.87 | 91.56 | 91.47 |\n| our                                       | 92.34 | 92.30 | 92.45 | 92.36 | 92.30 | 92.19 | 92.10 | 91.72 |\n\nOverall, our model obtains better performance for all values of $k$, even with MonoBERT having more parameters than our model. Our hypothesis is that the performance differences come from our retrieval dataset being specifically suited for NER and for the domain at hand. We will include these results in a revised version of the article.\n\n\n> Since Vicuna could generate positive examples for building synthetic data, why cannot we just use the generated positive examples to augment NER tasks instead of retrieving them from a novel?\n\nAs mentioned in the article, we use the Alpaca LLM and not Vicuna, but your question still stands. This generation cannot be done at inference time because it requires the NER label of the entity to be correct. Since this label is unknown at inference time, a LLM could generate a context that mention the entity but resolves it to an incorrect type regarding the current novel. This is why we take advantage of the training set, which is labeled, to generate a context retrieval training dataset that we then use to train a re-ranker.\n\n\n> In the experiments, only BERT-base model is evaluated, which might have limited performance in NER tasks and result in a performance boost when augmented with retrieved sentences. I would like to know whether the performance can still be improved when a more powerful model is employed as the backbone, e.g., BERT-large or RoBERTa. As a result, more comparison analyses should be provided to probe the effectiveness of the approach.\n\nThis is a good point, and we are currently working on an experiment to include results with BERT-large in the article.\n\n\nPlease feel free to let us know if you have further questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe paper is well-written and easy to understand and follow.\n2.\tThe idea of using LLM to generate training data is well-motivated and is valuable for further exploration in many tasks. \n3.\tExperiments show the proposed approach can achieve better results than traditional methods.", "reasons_to_reject": "1.\tThe proposed method employs the neural retriever as a re-ranker to find top-k sentences from a candidate set. However, all the baselines have only a retrieval process without re-ranking. I suggest adding baselines that are equipped with a re-ranking stage, e.g., BM25 + TextRank.\n2.\tSince Vicuna could generate positive examples for building synthetic data, why cannot we just use the generated positive examples to augment NER tasks instead of retrieving them from a novel? \n3.\tIn the experiments, only BERT-base model is evaluated, which might have limited performance in NER tasks and result in a performance boost when augmented with retrieved sentences. I would like to know whether the performance can still be improved when a more powerful model is employed as the backbone, e.g., BERT-large or RoBERTa. As a result, more comparison analyses should be provided to probe the effectiveness of the approach.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a neural retriever that is trained on the synthetic dataset using Alpaca. The retriever could find relevant sentences from long novels to help the NER task. Specifically, the authors employ Vicuna to generate positive retrieving pairs through prompt instructions and use heuristic methods to form negative pairs. Then, the synthetic data is used to train the retriever, which could rank sentences in a small candidate set that is obtained from simple heuristic pre-retrieval. The top-k sentences are concatenated with the input sentence to finish the NER task. Experimental results show that the proposed method could outperform the unsupervised retrievers.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PIxZgMnA69", "reviewer": "Reviewer_NDUN", "summary": "", "questions": "* Can you elaborate on why training on negative sampling only leads to \"overfitting on context containing entities\", claimed on line 215?\n* In positive sampling, I don't understand the need to filter out sentences $s_j$ that do not contain the target entity (line 198). Why not continue generation until the target entity appears? Is this related to the fact that the datasets generated with Alpaca 7B and 13B are of different sizes? This fact is also unclear to me.\n* More details about training of the neural reranker should be given in the paper (line 276). Was the model evaluated intrinsically (on a portion of the synthetic dataset)?\n* How do you identify nouns in the \"samenoun\" setup? Do you use NLTK as done by Amalvy et al. (2023)?\n* I'm not sure if the claim on line 288 is true. My understanding is that the method of Amalvy et al. (2023) can perform global retrieval on the whole document and doesn't \"need NER annotations\". Extension to whole novels is therefore straightforward and unrelated to NER. Please elaborate on this to remove any confusion.\n* The hypothesis on line 406 should be tested empirically.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThank you for your review! We tried to answer to your remarks as best as we could below:\n\n\n> Dataset generation is described for specific entity types (PER, LOC, ORG). Because of the dependence on LLM prompts, it is not guaranteed that its extension to new types will result in similar performance gain.\n\nWe worked on ENAMEX entities, which are among the most common entities types, but we agree that it would be interesting to study the effect of the method on other entity types. However, these entity types should be chosen accordingly, since the method might be irrelevant for some of these (dates, amounts...).\n\n\n> It is not clear from the paper if the synthesized dataset allows for training of trainers that generalize to other NER tasks or if new datasets should be generated every time the approach is used. This is computationally expensive and would affect its adoption.\n\nThis is an interesting question, that would be an interesting extension to our work. However, we are concerned by the lack of long-range NER datasets for evaluation.\n\n\n> In order to better understand the importance of the reranker, a random baseline should be included which select $k$ random sentences from the $4n$. Also compare with randomly selecting sentences from each bucket of $n$.\n\nThis is a fair point. To address it, we performed your first suggested experiment with $n=8$. Here are the results we observed (f1, mean of 3 runs):\n\n| k |     1 |     2 |     3 |     4 |     5 |     6 |     7 |     8 |\n| - | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |\n|   | 91.57 | 91.64 | 91.74 | 91.72 | 91.67 | 91.69 | 91.56 | 91.51 |\n\nThe results are lower that using our reranker (or the MonoBERT reranker, see below) for all values of $k$, highlighting the added benefits of trained rerankers. We will include this experiment in a revised version of the article, as well as your second suggested experiment.\n\n\n> A missing baseline to assess the relevance of the dataset in my opinion is to compare with a neural reranker trained on any available similarity dataset.\n\nWe performed additional experiments on other baselines to answer this concern. We compared our re-ranker to a pretrained MonoBERT re-ranker (Nogueira and Cho, 2019). We chose MonoBERT since we wanted to compare to a model with a similar architecture in order to study the difference of performance caused by the training dataset. Still, note that the version of MonoBERT we employed is based on BERT large (huggingface checkpoint: `castorini/monobert-large-msmarco`), while our model is based on BERT base.\n\nWe experimented with two variants of a MonoBERT based system:\n\n- bm25 -> MonoBERT, to replicate a classic BM25 + reranker baseline\n- [bm25, samenoun, surrounding] -> MonoBERT with $n=8$, to compare with the neural setup of our article (as in paragraph Inference of Section 4.1)\n\nWe obtained the following mean results on 3 runs (F1-score):\n\n| System             /                 k    |     1 |     2 |     3 |     4 |     5 |     6 |     7 |     8 |\n| ----------------------------------------- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |\n| bm25 -> MonoBERT                          | 92.18 | 91.91 | 91.93 | 91.98 | 91.90 | 91.58 | 91.65 | 91.44 |\n| [bm25, samenoun, surrounding] -> MonoBERT | 92.23 | 92.16 | 92.27 | 92.20 | 92.16 | 91.87 | 91.56 | 91.47 |\n| our                                       | 92.34 | 92.30 | 92.45 | 92.36 | 92.30 | 92.19 | 92.10 | 91.72 |\n\nOverall, our model obtains better performance for all values of $k$, even with MonoBERT having more parameters than our model. Our hypothesis is that the performance differences come from our retrieval dataset being specifically suited for NER and for the domain at hand. We will include these results in a revised version of the article.\n\n\n> Qualitative analysis is needed to understand where the increase of 1 point of F1 is coming from and if it justifies the computational resources needed to implement the method.\n\nWe did include a few examples of retrieved contexts in Table 3. In order to go further, we are currently manually checking the cases where the re-ranker corrected a NER prediction that was erroneous when using one of the retrieval heuristics.\n\n\n> The proposed method to generate the dataset is completely unrelated to the document from which the prompt sentence is selected. It is also unrelated to the output of the retrieval heuristic on which the reranker operates. It might be better to use the LLMs to sort through the output of the heuristics. This creates a discrepancy between train and test distributions that might be harmful for the reranker. I understand though that performance gains in NER as a result of using this dataset indicates that the reranker is still able to function with the test distribution. An exploration of this question is a welcomed addition in my opinion.\n\nIf we understand your question correctly, you are asking if the discrepancy between the distribution of examples in the synthetic retrieval dataset and the distribution of sentences retrieved by the heuristics can have an impact on performance. As you point out, the results tend to show that the reranker is able to function on the distribution of retrieved sentences, but the discrepancy might still affect results negatively. This is an interesting question that warrants at least a mention in the article. An LLM might be able to sort through the output of the heuristics to produce retrieval examples, but we are concerned about the the correctness of the generated examples. We tested a similar approach where we generated positive examples by checking whether or not contexts retrieved by the heuristics could help the NER prediction, but the quality of the generated examples was subpar which made for a noisy retrieval dataset that did not lead to good retrieval performance in practice. \n\n\n> Can you elaborate on why training on negative sampling only leads to \"overfitting on context containing entities\", claimed on line 215?\n\nA large portion of contexts retrieved using negative sampling do not contain any entities (66.37\\% of the sentences of our NER dataset do not contain any entities). We observed that, due to this, the model would overfit and mostly predict that a context containing an entity was relevant (and a context with no entity irrelevant). Adding irrelevant contexts with entities fixed that issue.\n\n\n> In positive sampling, I don't understand the need to filter out sentences that do not contain the target entity (line 198). Why not continue generation until the target entity appears?\n\nWe could have adapted sampling to force the model to continue generating until the target entity is found (for example, by not allowing sampling the END token until the entity is generated). However, there is no guarantee that the model will ever generate the entity. This filtering only occurred in a very few cases, as the model generated a context containing the entity almost every time.\n\n\n> Is this related to the fact that the datasets generated with Alpaca 7B and 13B are of different sizes? This fact is also unclear to me.\n\nYou are right, the (small) difference in size between both datasets is due to the filtering process. We added this precision to the paper to increase clarity.\n\n\n> More details about training of the neural reranker should be given in the paper (line 276). Was the model evaluated intrinsically (on a portion of the synthetic dataset)?\n\nTo answer your question, we measured that the reranker has a F1 score of 98.01 on a synthetic test set generated 5-folds on the test portion of the NER dataset. The task on this synthetic train set may be easier than the task in real conditions, but the NER performance increase and retrieved contexts (Table 3) make us confident that the neural re-ranker is returning relevant contexts. We improved the article to add the reranker's score on the synthetic test set.\n\n\n> How do you identify nouns in the \"samenoun\" setup? Do you use NLTK as done by Amalvy et al. (2023)?\n\nWe used the same method as Amalvy et al. (2023). We added this fact in the article for clarity.\n\n\n> I'm not sure if the claim on line 288 is true. My understanding is that the method of Amalvy et al. (2023) can perform global retrieval on the whole document and doesn't \"need NER annotations\". Extension to whole novels is therefore straightforward and unrelated to NER. Please elaborate on this to remove any confusion.\n\nYou are right in saying that the method of Amalvy et al. can be extended to the whole document and do not need NER annotations. However, in their article, they only retrieved context on the first chapter of each book, which is the only chapter annotated in the book. By contrast, we applied their retrieval method on the whole book. We reformulated the sentence in the article in case it was unclear.\n\n\n> The hypothesis on line 406 should be tested empirically.\n\nWe agree that this in an interesting question. Since there are more retrieval candidates in the whole book, some heuristics might need to retrieve more sentences in the book than in the current chapter to find relevant candidates. This is something we are currently exploring by performing more experiments, as these are straightforward.\n\n\nPlease feel free to let us know if you have further questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Up to my knowledge, this is a novel use of LLMs in dataset generation for NER.\n* The provided anonymous repository is well organized which facilitates reproduction.", "reasons_to_reject": "* Dataset generation is described for specific entity types (PER, LOC, ORG). Because of the dependence on LLM prompts, it is not guaranteed that its extension to new types will result in similar performance gain.\n* It is not clear from the paper if the synthesized dataset allows for training of trainers that generalize to other NER tasks or if new datasets should be generated every time the approach is used. This is computationally expensive and would affect its adoption.\n* In order to better understand the importance of the reranker, a random baseline should be included which select $k$ random sentences from the $4n$. Also compare with randomly selecting $k/4$ sentences from each bucket of $n$.\n* A missing baseline to assess the relevance of the dataset in my opinion is to compare with a neural reranker trained on any available similarity dataset.\n* Qualitative analysis is needed to understand where the increase of 1 point of F1 is coming from and if it justifies the computational resources needed to implement the method.\n* The proposed method to generate the dataset is completely unrelated to the document from which the prompt sentence is selected. It is also unrelated to the output of the retrieval heuristic on which the reranker operates. It might be better to use the LLMs to sort through the output of the heuristics. This creates a discrepancy between train and test distributions that might be harmful for the reranker. I understand though that performance gains in NER as a result of using this dataset indicates that the reranker is still able to function with the test distribution. An exploration of this question is a welcomed addition in my opinion.\n* Some points in the method description need clarification (see the remarks below).", "questions_to_authors": "* Can you elaborate on why training on negative sampling only leads to \"overfitting on context containing entities\", claimed on line 215?\n* In positive sampling, I don't understand the need to filter out sentences $s_j$ that do not contain the target entity (line 198). Why not continue generation until the target entity appears? Is this related to the fact that the datasets generated with Alpaca 7B and 13B are of different sizes? This fact is also unclear to me.\n* More details about training of the neural reranker should be given in the paper (line 276). Was the model evaluated intrinsically (on a portion of the synthetic dataset)?\n* How do you identify nouns in the \"samenoun\" setup? Do you use NLTK as done by Amalvy et al. (2023)?\n* I'm not sure if the claim on line 288 is true. My understanding is that the method of Amalvy et al. (2023) can perform global retrieval on the whole document and doesn't \"need NER annotations\". Extension to whole novels is therefore straightforward and unrelated to NER. Please elaborate on this to remove any confusion.\n* The hypothesis on line 406 should be tested empirically.", "paper_topic_and_main_contributions": "This paper builds on the work of Amalvy et al. (2023) on retrieval-based context augmentation for document-level NER in novels. The authors propose to use instruction-following LLMs like Alpaca to generate sentences with given entities in them. Generation is guided by engineering specific prompts using a sentence including the given entity. The dataset, which also contains negative examples, is used to train a neural regression model that assigns a relevance score to two input sentences. This model is then used to rerank sentences retrieved by some heuristic from the input document in order to augment the context of the NER model. The authors show that this approach results in improvement of NER quality measure by 1 point increase in F1 on Dekker et al. (2019) dataset of novels.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: ***Quality***: Reviewers largely agreed that this work investigates a promising research topic and makes research contributions. Several reviews noted that the experimental evaluation did not sufficiently ablate performance improvements to differentiate improvements from data augmentation versus re-ranking. The authors performed additional experiments during the rebuttal period to address these issues. The authors provided sufficient materials to allow reproducibility of their findings.

***Clarity***: Reviewers found the ideas to be clear, although suggested some details and discussion be added to improve clarity.

***Originality***: The core idea of this paper (data augmentation using LLMs) is an area of active research, and is not novel. The application to contextual NER is an original application of this research approach.

***Significance***: Improvements from the proposed method on the NER task investigated were small, but the task area is mature and small improvements are still meaningful. This paper is of interest to the EMNLP community and provides a useful contribution. Overall, the paper did not generate strong excitement from the reviewers, so the impact to the community may be small.

***Pros***:
* Promising research direction for contextual NER
* Clear ideas and a novel research contribution
* Reproducible experiments supporting the claim 

***Cons***:
* Additional experiments necessary to more clearly support the method
* Extending to other datasets with different NER tags may be difficult
Decision: Accept-Main