Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The paper introduces an approach to generate a named entity linking dataset with a temporal dimension. It identifies various phenomena related to the problem, and the authors tackle them in the data generation process. As a result the method is used and a dataset is generated and provided, for 10 years of snapshots of English Wikipedia. Additionally, authors establish a baseline performance on this data set. 
strengths: In my opinion, the main strength of this paper is that it provides a well-structured procedure to create a temporal dataset, taking into account more detailed phenomena than previous data authors. The code is not available, but if it turned out that the code is abstract enough to be able to move beyond Wikipedia, that would make this paper an even stronger submission. 
weaknesses: The main weakness of the paper is that it delivers another Wikipedia-based dataset in a single language, which happens to be English. The English Wikipedia is the most extensively used data source in the most data rich language, what the authors could've done is at least run the method for top 5 Wikipedia languages, or select a lower-resourced language if they only had the budget and means to do one language - even just Belgian and Danish, to have the ability to verify the value of the method outside the most optimistic case of English Wikipedia. The authors mention cross-lingual phenomena as limitations, but truth be told, even a single language low resourced dataset added to this paper would make a great change.

correctness: The data set is constructed in a well-documented, sound way. Evaluation and experiment design are correct. The supplementary material is extensive.
clarity: The paper is well written.
relation to prior work: The proposed data set introduces mechanisms that control elements, which were not taken into consideration in previous work, such as context availability and dataset balance. 

I think it would also be very elegant to cite the efforts of people working on other languages in the same area and task, such as: MobIE by Henning et al., Caillaut et. al for French entity linking, Ogrodniczuk from 2020 for Polish entity linking. The last one was also performed on Wikipedia. Not all of those of course focus on temporal aspects, but remain valuable works in the field - if we want, as a community, to see more work on low-resourced languages we should cite approaches from non-English work in our fields. There's also a PhD thesis by Mendez from 2021 which does a good service to dataset surveying.


additional feedback: It would be very beneficial to the quality of the paper and its impact on the task, if the authors ran their approach on more languages in Wikipedia.
review_title: A useful data set that will become a good benchmark on the task, sadly only in English and needlessly so.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper presents an interesting dataset TempEL to explore how do evolving and emerging entities affect the entity linking task.  The proposed dataset includes 10 years snapshot of entities and various experiments were conducted to evaluate the impact of time information on the entity linking task. Experiments show that there is a consistent temporal decrease in performance for continual entities. 

One question for the result is in Table 2, why the performance of training/testing in the same year is lower than the performance of training year > testing year? This result is counterintuitive to me.  
strengths: This paper focuses on an interesting topic, the dynamic characteristic of entities is important but was less investigated. This paper constructed a large and time award entity linking dataset and examined the model performance in different time settings. 
weaknesses: The whole pipeline of constructing the dataset is not clear. For example, this paper didn't present the number of entities and mentions after each filtering step. Although the supplementary material provides stepwise numbers, a detailed filtering figure with important numbers would make this paper clear. 
correctness: Yes
clarity: Yes, but it would be better to add more details about the dataset construction.
relation to prior work: yes
additional feedback: None
review_title: Very interesting dataset with funny results.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper focuses on the problem of temporal shifts for entity linking. In short, as knowledge bases are continually modified–with entities being changed and added–entity linking models can grow stale, having been trained on “older” data. For instance, an entity model train on Wikipedia up til 2019 would have never seen sentences associated with the entity “Coronavirus.” This is a well known problem that has garnered significant interest. 

To promote better investigation of this problem, the authors design a benchmark–TempEL–around the idea of temporally shifting entity linking. Specifically, the authors extract 10 yearly snapshots of Wikipedia (2013-22), and for each snapshot, curate a train/val/test split containing entity mentions. This benchmark allows researchers to study how a model trained on one year may generalize to another. The authors describe their process for constructing this dataset, and conduct preliminary experiments to analyze performance. 

strengths: As far as I can tell, there doesn’t appear to be any other benchmark that enables this type of study. Thus, the contribution is clearly novel. 

There’s been a lot of focus on studying the performance of models on knowledge intensive tasks when that knowledge is (1) changing, or (2) less well known (i.e. “tail” entities). I was really glad to see that the authors recognized these directions, and explicitly built test sets to evaluate models along these dimensions. 


weaknesses: No large weaknesses come to mind. 

I think additional metrics–beyond just Accuracy@64–would be helpful. I understand why the authors use acc@64 (the Blink paper uses it), but a metric which captures performance more finely (i.e. even accuracy@10) would be interesting to see.
correctness: The construction of the dataset seems sound. I appreciate the care the authors took with regards to the details of the distribution of entities/mentions.

The experiments seem largely correct and well done. My questions are only minor. 

Isn’t the upper table (shared entities) in Table 2 a bit misleading? If I train a model on the 2013 split, the number of “unseen” entities relative to the 2013 split is going to be a lot more for the 2018 split than the 2013 split, right? My understanding is that the normalization of unseen entity count is occurring within a year–i.e. the number of unseen entities in the 2018 test relative to 2018 train is the same as the number in the 2013 test relative to the 2013 train. Then, it would make perfect sense why we see the degradation in performance for later years?


Is there a way to quantify the influence of yearly wikipedia pages (e.g. “2018 FIFA WorldCup”) on your metrics? You observe the model seems to have little trouble disambiguating these mentions. I can imagine slicing entities by whether or not their title contains a number. Additional slices (based on entity type) might also provide interesting insights. 

clarity: The paper is well written. I appreciate the detail the authors devoted to explaining the dataset construction process. 

One nit: I’m a bit unclear on where the entity descriptions come from. The paper mentions its anchored in time and that it comes from Wikipedia. Is it the first sentence of the entity’s Wiki page?
relation to prior work: Yes!
additional feedback: N/A
review_title: Nice dataset

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: The authors present TempEL: a new dataset for analyzing and evaluating Entity Linking models on time-evolving cases. 
The authors describe in detail the building approaches and quality control precautions to ensure a sound final dataset.
They also provide some basic experiments and results that validate their theory and shows the usability of the dataset.
strengths: The text is clear, well written and to the point.
The construction of the dataset and the quality check sections are very detailed and show a great effort to provide the cleanest dataset possible.
The benefit of the paper lies in providing a dataset that takes the entity linking evolution in time by intelligently using the Wikimedia system of writing, correction, verification and publication as a controllable proxy for real cases of entity mention.
Entity Linking and Named Entity Recognition are tasks of importance and required steps of many other tasks. An attempt to improve the study of their evolution in time is commendable.
weaknesses: To serve as a benchmark, the paper could have compared the obtained models (trained on their dataset) with other benchmarks or pre-trained/fine-tuned models from other investigations with their dataset in order to allow a direct comparison and an evident comparison between their work and other SotA work.
The choice of Accuracy@64 metric is not well justified and can be seen as arbitrary and not strict enough without a valid precedent or justification for its use.
correctness: The dataset building is transparent and well explained. The choices taken for its construction are justified and sound. 
The dataset is not presented as a benchmark although it could serve as one if it did not lack the required comparison with other related works and models.
clarity: The text is clear, well written and to the point. Figures and tables are clear and well explained in the caption.
relation to prior work: The paper clearly states that their main contribution lies in the time-evolving aspect of the dataset, which is absent from previous and related works. That statement is true, well exposed and justified.
additional feedback: We recommend the authors use the extra page post review to justify and correct the weaknesses discussed here.
review_title: Well constructed dataset with an interesting and understudied point of view on a useful and requiring task.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: Temporally evolving data streams are present in many application settings. Entity linking is a key application area that is under-explored in the temporal evolving entity setting. This paper presents a dataset, TempEL, that uses time-stratified snapshots of Wikipedia to create a benchmark dataset for evaluating entity linking performance given continually evolving mentions and entities.  The authors build 10 yearly snapshots from Wikipedia (2013-2022), creating snapshots that break down entities into (1) continual entities and (2) new entities. 
Using this dataset, the authors evaluate a baseline bi-encoder model (BLINK) to evaluate the impact of temporal drift on entity linking performance. 
strengths: - This is a very nice dataset paper that speaks to a realistic and important application setting (entity linking with continually evolving entities and KBs) 
- The continual vs. new entities is a nice framing
- All data is built on wikipedia, which is a great dataset for this setting.
- Provides a strong baseline (BLINK) that is evaluated to measure interesting performance differences between continual and new entities. 
- The dataset generation protocol is extremely detailed and clearly spelled out in supplemental materials. 
- The dataset is available now. 
- Related work is very comprehensive.
- COVID 19 provides a compelling natural experiment for examining temporal entity drift

weaknesses: - I think the Figure 2 pipeline figure could be clearer. At first glance it doesn't really make sense until read section 3.1 I think adding a "worked" input/output example at key steps of the workflow would help make the process more concrete. 

- Some notion of entity types / categories across datasets would be useful for ablation experiments. I'd guess new entities favor certain types (events like "COVID-19_pandemic_in_Portland,_Oregon") 

- (Minor) It would be nice to see multiple seeds/replicates for experiments (however since this comes at additional computational resources the absence is reasonable). 
correctness: Everything seems correct
clarity: - The paper is nicely written and easy to follow. The main area that could be improved is Figure 2, which I think is difficult to parse and doesn't really convey what each step is actually doing in the data workflow. 
relation to prior work: This dataset is a great addition to existing temporal and entity linking datasets. TempEL fills a nice gap in existing datasets by looking at both temporally evolving entities and mentions in a systematic evaluation framework. The manuscript provides a very comprehensive overview of existing entity linking and temporal datasets. 
additional feedback: No additional feedback.
review_title: A useful and well-constructed dataset

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper presents a dataset called TempEL. TempEL contains Wikipedia snapshots in 10 years and the entity mentions and KB entity descriptions are both drawn from the same Wikipedia text. The authors provide a detailed description on processing Wikipedia pages and sampling entities and mentions.  The authors then conducted an evaluation based on a dense retrieval model. 

strengths: The authors provide a detailed description on the data creation process, particularly on the part cleaning Wikipedia pages. 
The quality control in the creation process is discussed in detail.
The dataset is well analyzed through experiments and the authors report a few interesting findings on results. 

weaknesses: The main weakness of the paper is motivation. I do not find the paper well motivated to justify the need of such a dataset. In a typical entity linking setting, we have a document, where a few mentions are detected, then we try to search for candidate entities for each mention and try to link the mentions to their correct entities. The search for candidate entities can be conducted on any KB version.  As long as the KB is updated, then the search is able to return the most up to date entities as candidates. 

The second weakness: the dataset does not support collective linking. In a document, typically there are multiple mentions and the confident linking provide additional context for less confident linking, so all the mentions in a document can be collectively linked with confidence. This dataset can not be used to evaluate such models. Further, as the mentions are also from Wikipedia, this dataset does not simulate the linking task in practical setting. 

The third weakness is the quality control. It is good that the authors provide the description on how quality control is done. However, due to the quality control, the dataset does not reflect the true performance of a model in real setting (e.g., linking to wikipedia). In real setting, we would not avoid popular entities and it is not wrong to link an entity based on high mention frequency. The control on dataset distribution makes the dataset only for benchmarking purpose, and yet the benchmarking does not reflect the actual performance of a model in real setting.  

Thanks for the clarifications from author reply. 
correctness: The construction of the dataset is sound for the defined objectives in the paper. However, in my understanding, the dataset does not reflect any real problem settings. Hence the performance obtained on this dataset may not truly reflect a model's performance on real settings. An example setting is to link all mentions of names in a news article to Wikipedia entries. 
clarity: The paper is mostly well written. It would be very helpful if the authors provide some examples in the description. For example, the meaning of mention context. At least some samples of mention and mention context, and entity descriptions. I would consider Figure 1 provides such examples, before reading the supplementary material. 
relation to prior work: Section 2 briefly summarizes related datasets. This dataset is different from existing ones because of different task settings.  
additional feedback: Overall, I do not find the paper well motivated, and the task setting is fairly different from most entity linking tasks. 

Review updated to correct a typo: In real setting, we would **not** avoid popular entities and it is not wrong to link an entity based on high mention frequency.
review_title: The paper is not well motivated 

Meta-review: metareview: This paper describes a dataset for entity linking with entities that may change over time (both in how they are referred to as well as their KB description) as well as new, emerging entities. Baselines are evaluated on this dataset.

Reviewers appreciated the general thrust of the work, the resource itself, and the clear presentation. The reviewers were solidly in favor of acceptance overall.

One criticism is the focus on English, which I agree with. There were also criticisms of the Accuracy@64 metric (from BLINK), but these seem fixable in subsequent work if other authors want to use a different evaluation. Finally, reviewer FKvJ raised some issues about both the motivation/task setup as well as some particulars of the dataset construction. However, that reviewer and I found the replies from the authors generally satisfactory. I don't believe there is necessarily a "natural" distribution of entities to link, and choosing to focus on the temporal element is reasonable as a research direction (especially given the growing awareness of temporal divergence in model performance).
confidence: 5: The area chair is absolutely certain
Decision: Accept