Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: - The authors release METS-Cov, a dataset containing medical entities and targeted sentiments from COVID-19-related tweets. The medical entities include 4 medical entity types (Disease, Drug, Symptom, and Vaccine) and 3 general entity types (Person, Location, and Organization)
- The benchmark results for two tasks, Named Entity Recognition and Targeted Sentiment Analysis are given, arguing that there is vast room for improvement.
strengths: - They annotated both medical entities and the sentiment for certain entity types.
- The comprehensive benchmark results are given.
weaknesses: - This dataset is not convincing why labeling sentiment and medical entities in COVID-19-related tweets is important. There is no sufficient explanation for the future use of the dataset.
- For METS-CoV-TSA, there exists quite a serious class imbalance problem, but there is no analysis on that. And the number of samples is too small, as METS-CoV-TSA is filtered again from METS-CoV-NER, which is still not a huge dataset.
- The previous datasets, 23 million COVID-19-related tweets or COVIDSenti, seem to have much rich information.
correctness: The idea in the paper sometimes contradicts their own idea. For example, in Section 4.1, they are saying that “it is consistent with human performance” for the reason why COVID-TWITTER-BERT is confusing between disease and symptom. However, it seems that it is not a desirable dataset if the annotators are still confused with the topic. 
clarity: This paper is relatively well-written, but in some sections, the contexts are sometimes not related or confusing such as Section2 and Section4.
relation to prior work: They discuss that clinical social media texts remain under-explored and METS-CoV differs from the previous dataset.
additional feedback: - Additional to Table 1, it would have been better if there is an analysis of the unique number of tweets for each entity type.
- I wonder who are the annotators for labeling the medical entities, are they from a medical domain?
review_title: Relatively less persuasive datasets in terms of specialty and further use

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper introduces an annotated dataset, METS-CoV, of 10,000 tweets about COVID-19. The annotations include 4 types of named entities and the writer of the tweet’s perceived sentiment towards that named entity, either positive, negative, or neutral. The authors of the paper also tested many different models for their performance on type of named entity. Generally, the models are best at recognizing neutral entities.
strengths: •	Significance of the contribution: As the only targeted sentiment analysis dataset which is specifically oriented towards COVID-19 METS-CoV is a great and novel concept and a solid contribution to both the medical field and the field of sentiment analysis.
•	Relevance to the broader research community:  METS-CoV will be useful for both those researching sentiment analysis in general and those researching COVID-19 and attitudes around COVID-19.
•	Accessibility and accountability: METS-CoV is accessible on github and the authors have laid out a plan for its maintenance.
•	The annotation process is very clearly defined, well thought-out and triple-checked for inconsistencies. 

weaknesses: All four types of named entity types have large class imbalances between positive, negative, and neutral. For example, the “Person” category has 383 positive examples, 1,788 neutral examples and 1,038 negative examples. There are less than half as many positive examples as negative examples and a little over half as many negative examples as neutral ones. It can be very difficult or nearly impossible to curate a dataset like this without class imbalance, but it is still important to note that this dataset is very imbalanced which has caused both positive and negative targets to be more likely to be predicted to be neutral in some categories as shown in Figure 5.
correctness: As far I can tell the dataset is constructed in a sound way and the claims made in the submission are correct.
clarity: This paper is well written. 
relation to prior work: This dataset’s relation to prior work is discussed in detail in section 2 of the paper. 
additional feedback: Have you considered either over or under sampling to help the class imbalance when testing the performance of different models?
review_title: METS-CoV: A Slightly Imbalanced but Well-Annotated Dataset for COVID-19 Target Sentiment Analysis

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors collected a publicly available covid-19 dataset and filtered tweets. Post filtering, they had ~2 million tweets and randomly sampled 6,000 tweets and annotated the tweets for two different tasks i.e NER and TSA. Authors rigorously experimented with several machine learning models to demonstrate the performance of the 10,000 annotated tweets for the 2 tasks and presented the results. The following are the contributions from this paper 
1) An annotated dataset of 10,000 tweets 
2) Annotation guidelines 
3) Performance of the several machine learning models when trained on annotated dataset 
strengths: The dataset released by the authors is significant since there is a huge need for annotated datasets especially with an increase in Covid-19 research using social media data. The paper is mostly well written and easy to understand. The dataset is available via Github and all the code and config files are included in the repository. The dataset can definitely be used for broader NER applications as it is primarily targeted towards adding medical entities. 
weaknesses: This work does not directly describe how the work directly affects the NER community or what value does it add to the scientific community. The paper mentions the limitations of the other works in introduction section however it does not add whether this dataset addresses the limitations or not. Additionally there is no mention of how this dataset can be incorporated into downstream tasks. A brief discussion would further strengthen this paper. 
correctness: The claims made in this paper seem correct. However, I have trouble with the construction of dataset. 
1) Why did the authors use only Chen et al. dataset when there are several large covid-19 datasets (Eg:  Banda et al.)? Why not use all the datasets available. A rationale must be given why Chen et al. was the authors' choice of dataset 
2) Why tweets with URLs were removed? Authors can definitely remove the URLs and still keep the tweet text for analysis. However the authors chose to eliminate the tweet in total. 
3) When there are 3 pairs of annotators, why did the authors limit the sample size to 6,000 tweets? So what was the point of obtaining over 300 million tweets and pre-processing them to 2 million tweets and then only sample to 6000 tweets? 

clarity: The paper is well presented section wise. However, the paper must be proof read for grammatical errors and sentence constructions. 
relation to prior work: The authors discuss how the dataset is novel and how it differs from previous contributions
additional feedback: The paper has some scientifically significant contributions. In addition to the above things, if authors can provide clarity on the following items, it would strengthen the paper
1) Why F1 was considered as a metric in METS-COV-NER annotation and accuracy as a metric in METS-COV-TSA? Why not have a uniform metric. Usually accuracy shows inflated metrics when compared to F1 and this is also seen in Tables 3 and 5. Is there a specific reason why two different metrics were chosen?
2) Does length of the tweet  matter in either NER or TSA tasks? Fig 2 displays the distribution of tweets based on length and I do not understand the scientific significance of that. There is also limited text in the paper. If for sentiment analysis, the length of the paper matters, relevant papers must be cited to draw comparisons. 
3) There is only 1 paragraph on limitations. Limitations must be a different section and not included on Future work and conclusions. Further, limitations must be elaborated in detail. Why does the keyword based symptom collection bring imbalance? The paper's primary contribution is to create a dataset targeted to medical entities and hence why is the imbalance a limitation?
4) Several models perform poorly on METS-COV-TSA dataset when compared to METS-COV-NER dataset. Does this mean this dataset is not good for TSA tasks? A small discussion on performance drops across the models is needed.
review_title: This paper released a dataset primarily for medical entities and targeted sentiments on Covid-19 tweets. The annotation efforts are a strength to this paper while the dataset has few limitations which were not completely written in the paper. More relevant text must be added to the paper to add perspective to strengthen the paper. 

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper presents a dataset of covid related tweets, which can be used for two tasks: NER (10k tweets) and TSA (+5k tweets). The texts are tagged with 4 medical entity types and 3 general entity types, having some of them annotated with user sentiments. The wide range of models that were trained for benchmarking show room for improvement on both tasks, specially on TSA.

The main contribution of the paper is the aforementioned dataset (with the corresponding annotation guidelines), which the authors claim to be the first of its kind. As mentioned in the paper, it can be used to train domain-specific models to better understand the impact of a pandemic on people.
strengths: - First dataset of social media posts about COVID-19 annotated with both medical entities and their corresponding sentiment. 
- Provides a new dataset to the research community to evaluate SOTA models on the NER and TSA tasks.
- The benchmarking has been conducted with such a wide variety of models that show consistent results.
- Detailed description of the annotation process, which has been carefully customized for the specific scenario at hand.
- Full annotations guidelines provided as supplementary material.
- Well documented GitHub repository.
weaknesses: - The keywords used to filter medical-related tweets contain symptoms that are not exclusively present in covid patients (but it can be assumed that only retrieving tweets from pandemic times most of them will be covid related).
- The fact that disease names are used as keywords in the retrieval process inevitably gives rise to an imbalanced NER dataset, since one of the entities is precisely "disease". This is already noted in the text.
- Some of the lables (40% of METS-CoV-NER) are machine genereated (bert model trained on 6k manually annotated examples). 

- 
correctness: The dataset was constructed in a sound way, and the evaluation methods seem appropriate.
clarity: The text is very well written, with a clear and concise language used from beginning to end. The ideas presented are well explained and supported by tables and figures when necessary. Overall it was a pleasant read.
relation to prior work: Section 2 clearly discusses the main differences with previous work. After mentioning the most popular NER and TSA general-domain datasets, the authors talk about less known datasets that are closer in nature to METS-CoV.

In the case of NER, there is one dataset that tags covid related text but, as it is highlighted by the authors, it differs from this work because it uses scientific publications that use a very different language than user generated content from twitter. Moreover, the number of entities is much higher (75 vs 7).

Regarding TSA, there are plenty of datasets that provide the sentiment of tweets. The authors mention one that cover covid related tweets like METS-CoV, but it only gives the overall polarity of the text rather that targeting specific entities. 

This makes the claim of uniqueness true (unless some dataset that I am not aware of has been ignored).
additional feedback: I wonder why vaccine names (AstraZeneca, Pfizer, Moderna...) were not used as keywords in the filtering phase? Wouldn't this be more effective than general symptoms? Specially in the case of covid, the symptoms of which are common in many other diseases. Is there a reason for that decision?

Regarding Figure 2, maybe it would be more interesting to show the number of actual tokens (after tokenizing the text with i.e. bert) instead of words (split at spaces). In any case, since we are dealing with short pieces of text, it is clear that they would fit the model's context, but knowing the real distribution of tokens can help decide the value for the maximum sequence length (where to truncate). 

I believe the submission was supposed to be fully anonymized. In any case, I had no conflict with the authors/organization so it has not conditioned my review in any way.
review_title: Interesting new resource for domain-specific NER and TSA tasks, carefully crafted and and extensively evaluated with a variety of models.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper describes a dataset contains 10,000 tweets annotated with 7 types of entiteis, including drugs, symptoms, diseases, vaccines and so on. In addition, 4 types of entities are annotated with user sentiment.

The described dataset can be used to benchmark NER (Named entity recognition) and TSA (target sentiment analysis) models.

The authors test a bunch of existing NER and TSA models on the datasetn.

strengths: 1. The released dataset will be a useful resource for people who are interested in social computing and digital health.

2. The dataset seems to be well annotated (accoriding to Section 3.2 annotation process) and high inter-annotator agreement is reported.

3. The authors conduct extensive experiments of various models on the proposed dataset.

weaknesses: 1. reproducibility issue: the released dataset consists of a list of tweet ids and annotations, other researchers may not be able to download the actual text (note that this limitation may apply to any dataset using tweets)

2. the paper can be improved by adding more qualitative analysis and error analysis. For example, NER models fail to achieve good performance on Organizations, is it because limited context? or these organization names are written in a non-standard way? also see additional feedback 2 

correctness: The submmission is a dataset and it is constructed in a sound way
clarity: Yes, it is well written


relation to prior work: Yes, it is clearly discussed.
additional feedback: 1. it may be worthy explaining how to calculate F1 as the metric of inter-annotator aggreement and select annotators whose F1 greater than 80% (for example, do you use one annotator's annotations as gold labels, and calculate the F1 score of another annotator? or you have some gold test data, and use that to calculate the F1 of annotators?) 

2. suggest to report the number of unique entitiy mentions (symptoms, drugs, vaccines); especially because the tweets were filtered based on keyword matching. also suggest to report the proportion of unseen test entity mentions.

3. suggest to mention NER datasets from SMM4H shared tasks (https://aclanthology.org/W19-3203.pdf; https://aclanthology.org/2020.smm4h-1.4.pdf) which aim to identify the span of text in tweets corresponding to drugs and adverse drug events.

Note that I am saying you should cite these linked papers, as they are overview papers covering multiple tasks.
review_title: A dataset conatins tweets annotated with entity recognition and target entity sentiment

Meta-review: metareview: In this work the authors introduce METS-CoV, a dataset that contains 10,000 annotated tweets with 7 types of entities, including 4 medical entity types (Disease, Drug, Symptom, and Vaccine) and 3 general entity types (Person, Location, and Organization). Out of those entities: Person, Organization, Drug, and Vaccine are annotated with user sentiments, creating a unique dataset. 

Pros:
- Interesting intersection of medical entities and user sentiment as part of the dataset.
- Solid annotation procedure
- Through evaluation 

Cons: 
- The while it is the first medical entity + sentiment dataset, there other datasets with medical entities annotated that could have been used for evaluation (https://genominfo.org/m/journal/view.php?number=663)

The authors strengthened the paper during the discussion period for some reviewer concerns, others were not addressed completely. However, the score was improved or favorable in most cases.
confidence: 5: The area chair is absolutely certain
Decision: Accept