Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This manuscript introduces a new benchmark named TweetNERD for NERD evaluation. The proposed dataset is based on over 340k tweets from a 10+ years time range and is publicly available with the proper licenses. 

The authors compare TweetNERD with four existing benchmarks from multiple aspects, including the number of unique entities, entities, mentions, and the number of tweets. The proposed dataset shows remarkable improvement over others. In this paper, the authors demonstrate the experimental settings of TweetNERD for NERD-related tasks. They are Named Entity Recognition (NER), Entity Linking with True Spans (EL), and End to End Entity Linking (End2End). The experimental results with recent models are present in this paper as well. The numbers indicate the benchmark is challenging for current solutions, and there's much to explore for this dataset. 
strengths: 1. The proposed dataset is considered a large-scale dataset that can adapt to a wide range of experimental settings. 
2. The paper introduces the experimental settings for three different tasks related to entity linking.
3. Compared to existing datasets, the TweetNERD benchmark has notable advantages.
4. The authors performed a set of experiments with recent models. The results indicate the benchmark is challenging for future research.
5. The authors utilize entropy-based sampling and create TweetNERD-ood and TweetNERD academia to improve the diversity of the datasets. 
weaknesses: 1. This paper has a limited description of existing work/ datasets in this domain. 
2. The paper can be improved with more detailed descriptions of models, methods, metric selections, insights on results, etc. This would make the paper easier to follow.
3. The present of figures and tables, especially for Figure 1, can be improved.

correctness: Yes. However, the evaluation methods, metrics, settings, and experiment designs may be further described. 
clarity: The general flow is good. However, the writing and presentation can be improved. The paper can be better organized for more details.
relation to prior work: The paper presents a comparison to existing work in Figure 1 but provides limited information on previous work. 
additional feedback: N.A.
review_title: A large-scale, tweets-based benchmark designed for Named Entity Recognition and Disambiguation (NERD) systems

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper introduces TweetNERD, a dataset for benchmarking Named Entity Recognition and Disambiguation (NERD) systems on Tweets. This is so far the largest and most temporally diverse open-sourced dataset benchmark for NERD on Tweets. Two subsets of the dataset, TweetNERD-OOD and TweetNERD-Academic are also provided. The former is for assessing out-of-domain performance. The latter consists of Tweets from a collection of existing academic benchmarks that have been re-annotated with the new annotation guidelines. The two subsets were evaluated for named entity Recognition, entity linking with true spans, and end-to-end entity linking tasks. Several NER/EL approaches were compared and the results are briefly reported in the paper. 
strengths: 1. The annotated dataset is released by Twitter. It's larger and more recent than existing ones. 

2. The dataset keeps raw annotation data and is flexible.

3. The limitations of the dataset are discussed. 

4. The paper is well-written and easy to read.

weaknesses: 1. The comparison to existing benchmarks needs more discussion.

2. I didn't find any description on how candidate entities were generated. 

3. The sampling is not uniform, partly due to creating TweetNERD-Academic. 

4. Evaluation was done only on the two subsets of TweetNERD, and only some brief results were reported. 
correctness: Are the claims made in the submission correct? Yes.

If the submission is a dataset, it is constructed in a sound way? Partially. The pre-2020 and post-2020 parts may differ in distribution. 

If it is a benchmark, are the evaluation methods and experiment design appropriate and performed correctly? Partially. The choice of competitors for some experiments needs more consideration.
clarity: Yes.
relation to prior work: Some statistics is shown, but more discussions would help explain the novelty. 
additional feedback: 1. The authors simply claimed that existing datasets have limited set of Tweets, are temporally biased, or are no longer valid because of deleted Tweets. I think the comparison to existing datasets needs more discussion, so users can understand what is really new here. For example, is the annotation setup is a new and better approach compared to those used in existing benchmarks?

2. Figure 1(b) mentioned candidates. How were generated?

3. The sampling is non-uniform. Most tweets are from 2020 to 2021, while older tweets correspond to the TweetNERD-Academic subset. I guess TweetNERD-Academic and its counterpart may have different distributions.

4. In page 4, the paragraph of "Entropy based sampling", from the description, I only see popularity is modeled in the phrase entropy, but don't understand how you model disambiguation difficulty here. Wikipedia provides disambiguation function if you search for entities with ambiguity, e.g., "Java". So why not use it?

5. In page 4, the paragraph of "TweetNERD-OOD", the description is unclear. Based on my understanding, it seems this is a subset of the high entropy bucket. 

6. In the experimental evaluation, it is encouraged to evaluate the methods on the entire TweetNERD rather than subsets. 

7. For the experiments on entity linking with true spans, as along as true spans are given, I think this becomes a disambiguation problem and you only need to decide which entity the mention refers to. So the choice of baseline approaches needs more consideration.
review_title: Review of Submssion ID 353

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: Summary:
This paper manually annotates 340K+ tweets across 2010-2021 to benchmark named entity recognition systems and evaluate existing models on three related tasks: Named Entity Recognition, Entity Linking with True Spans, and End to End Entity Linking.

Contributions:
besides the regular tweets, the paper also provides two subsets: TweetNERD-OOD and TweetNERD-Academic, to assess out-of-domain performance and temporal generalization of NER models, respectively.
strengths: The large-scale tweet dataset with manually annotated entity links enables further research on entity linking on social media. The performance of some recent methods does not outperform some classic methods on three evaluation tasks, which motivates further research for robust NERD models.
weaknesses: 1) The process of obtaining TweetNERD-OOD is not clearly described: "and then perform stratified sampling based on Tweet actions to divide these buckets into sub-buckets". Hence it's difficult to distinguish samples from this subset and those from regular subsets. Therefore, the implications of performance of investigated models on TweetNERD-OOD is not very obvious.

2) Analyses of performance from existing entity linking systems lack the potential reason for failures from some more recent models compared with earlier models. Distribution shift could be one reason: although some recent method shows more powerful in their respective evaluation dataset, they become less powerful when directly evaluating on the newly collected tweet dataset without fine-tuning.

3) In the experiments section, only the two subsets, TweetNERD-OOD and TweetNERD-Academic, are tested. However, the large remaining samples aren't investigated, hence their values are hard to tell: e.g., models training on remaining samples can obtain much better performance on the two evaluation subsets.

 

correctness: The claims made in this paper are correct.
clarity: The paper is well organized and clearly written in general.
relation to prior work: In introduction section, the paper discusses the major difference with existing datasets: 1) large-scale; 2) longer temporal coverage, 2) currently valid since deleted tweets from earlier times are not included
additional feedback: I have a few questions about the evaluation section:
1) only the two subsets are utilized for evaluation. How about the other samples (occupying the majority of the dataset)? Similar performance if we test existing models well-trained on their original datasets? Improved performance when training models on remaining subsets and testing on the two evaluation subsets?

2) As mentioned in the paper, samples from the academic subset have shown in the earlier released datasets, but are re-annotated with this paper's annotation guidelines. Is there any analysis on the annotation agreement of earlier and new linked entities? Would combinations from both sources can help refine the annotations? Or for the same entity mention, the link has changed due to different understanding along time?
review_title: This paper introduces a large-scale tweet datasets for benchmarking named entity recognition. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The paper introduces TweetNERD, which is a large dataset of 340K+ tweets, mostly collected across 2020-2021, and which can be used to benchmark NERD systems. The main contribution of the submission, thus, consists in the size of the dataset. Furthermore, all expert annotations have been kept in the dataset, which means that the dataset allows for further experimentation with ambiguous annotations or different levels of disagreement.
strengths: The main strengths of the dataset are mentioned above, as contributions: (1) size of the dataset and (2) disagreement-aware dataset.
weaknesses: Several weaknesses can be identified:
- Is is unclear what makes the proposed dataset better than existing one in terms of temporal bias (the dataset is much larger, but many tweets have been published way before 2020)
- How much have the annotations changed in the TweetNERD - Academic dataset compared to the original datasets? Why was the reannotation necessary?
- Were all annotators familiar with the task? Were there any instructions on how to deal or manage disagreement?
- Why computing inter-rater reliability as percentages instead of the more reliable ways that use proper IRR metrics (Krippendorff's alpha for instance)? There is very little discussion that acknowledges the reasons for disagreement and how they can impact the models that are trained and evaluated with this dataset.
- The dataset does not seem to be balanced, but only F1 scores are reported. Why is that and why is the F1 score the only needed metric?
- The results are discussed in very little to no detail.
correctness: The models are only evaluated based on the F1 score. 
clarity: The paper is written fairly well, but several parts could be rewritten in a more clear and concise manner. At times, the reader needs to re-read a particular section several times to grasp the meaning. 
relation to prior work: Prior work is well referenced. However, the authors could be more precise in the way they selected the baselines and which tweets to re-annotate. Furthermore, there is no clear description of how these tweets were initially annotated, why new annotation guidelines were necessary, how much the ground truth changed with regard to the original dataset, etc. 
Further comparisons in terms of inter-rater reliability is desirable. 
additional feedback: See questions in the weaknesses section.
review_title: Review for paper "TweetNERD - End to End Entity Linking Benchmark for Tweets"

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: In this paper, the authors present a large-scale dataset for entity recognition and linking for tweets. In terms of size, the dataset is much larger than existing datasets for the same problem settings. Based on the description, the dataset is properly annotated with clear instructions to the annotators. In this sense, the quality of annotations is high, and the authors release raw annotations in addition to the golden labels. The dataset also comes with two carefully constructed subsets targeting easy and challenging cases. The main weakness comes with the limited access and dynamics of the dataset. Following Twitter policy, only Twitter ID is released instead of the raw content. Downloading the dataset at a different time may get a different subset. This would make the results from different papers not comparable. I would then do not consider this dataset to be really a benchmark. The authors also do not mention the dynamics of Wikipedia, being edited every day.
strengths: The dataset is of large scale and the authors have considered many factors in constructing this dataset, from tweets sampling to the annotation guideline, and then to the construction of two subsets.
The authors conduct evaluations of a few models for three problems, NER, entity linking, and end-to-end entity linking. 
The authors choose to release raw annotations in addition to the golden labels. The raw annotation enables studies of other research problems on top of the entity linking problems. 
weaknesses: 1. The main weakness is the access to the data is limited by Twitter API terms and conditions. I do understand that all users of the dataset have to follow the rules, but the result is different versions of the dataset being used in different papers. As a result, model accuracies are not directly comparable. 

2. The authors do not consider dynamics of Wikipedia, which is always being edited. A specific version or dump shall be specified, to avoid minimize the differences used by different authors. Some entity linking models rely on inter-linking between wiki entries from Wikipedia, which is not included in the annotation. 

3. The context used during annotation (Lines 50 - 53) might not well match with what is recorded in the dataset. In other words, a model may not be able to access the full context for making the correct linking. Example is the temporal context of the tweet. 

4. The paper is in general not well organized and with lots of small issues in writing. For example, "annotation setup" and "inter-annotator agreement" shall be presented together to offer a complete picture of the annotation process. 

The authors have well addressed these comments. 
correctness: The dataset is constructed in a reasonable manner. The authors indeed discuss all considerations on the sampling and annotation details. Yet, there are some points can be made clearer. Are all 340K tweets annotated by the same 3 annotators? (Line 119). If not, how many annotators are there for these 340K tweets and on average how many tweets each annotator annotates?

In Table 4, results for two subsets are reported. It is more meaningful to report the results on the entire datasets. We will have a better understanding of model accuracy on this large dataset, and at the same time, we will know the difficulty levels of OOD and Academic with reference to the entire dataset. 
clarity: The paper is mostly well written but can be better organized as I mentioned earlier in the weakness. Some detailed comments on writing: 
1. The use of reference seems not correct. Table, Figure, Sections are not always capitalized in the text. 
2. Lines 109-112. In this example, the 95th Academy Award can be determined by the general temporal context, not necessarily mentioned in the tweets?
3. Table 3, do you have a specific definition for entity, phrase, and phrase-entity? 
relation to prior work: The paper well discusses the existing datasets. In terms of the evaluation, the models are not comprehensive, in Table 4. Nevertheless, this is not a key focus of this paper.
additional feedback: Please see the weakness and the detailed comments on clarity.
review_title: A carefully prepared dataset, but is unlikely a benchmark.

Meta-review: metareview: The authors release TweetNERD, a collection of 340k tweets from a 10+ years time range with the proper licenses for Named Entity Recognition (NER), Entity Linking with True Spans (EL), and End to End Entity Linking (End2End). All reviewers appreciate that this is a large-scale temporally diverse dataset with many advantages over existing benchmarks, useful for multiple tasks, and experiments show that it is challenging for current models. They also find the paper well-written and easy to read. The main shortcoming they find is that related work and existing benchmarks could be more exhaustively described, and the paper could be more clear in certain areas. Some reviewers are also concerned that since the dataset needs to be downloaded by the Twitter API, it could change over time, resulting in slightly incomparable results across groups. 
confidence: 5: The area chair is absolutely certain
Decision: Accept