Named Entity Recognition in COVID-19 tweets with Entity Knowledge Augmentation
Authors: Zhangxuankang, Jiangming Liu


Reviewer paper summaries: This work investigates named entity recognition in COVID-19 related tweets using the METS-CoV dataset. The work proposes a data augmentation approach using LLMs that the authors claim is state of the art performance.

This work addresses the adaptation of NER system to COVID entities (vaccine, symptoms, drugs, diseases) with a method relying on prompts asking a GPT model to generate data related to COVID to provide additional entities and texts instances. The resulting dataset is provided to a standard LLM based NER system, which is evaluated on the domain benchmark (METS-CoV). Results show that small performance improvements are obtained, but not for all NE categories.
Summary of strengths: - Seeks to improve medical NLP related to the pandemic, which continues to be an important area of research. 
- Provides a good breakdown of performance across entity types
- Provides a figure with examples of the approach along with details and examples for the data augmentation that are easy to follow.

- Augmentation of a NER system to a specific domain
- Separate augmentation for entities and text instances, combined for learning
- Comparison to existing systems
Summary of weaknesses: - The results are not convincingly better than COVID-Twitter-BERT. The standard error margins appear to be large enough for the higher performance to just be caused by random chance. 
- Some of the papers main claims about improved performance and the necessity of using data augmentation seem unsupported because the baseline Twitter BERT model seems to perform within a range that could be accounted for by random chance.

- Data augmentation has already been studied in a number of studies
- The study heavily relies on ChatGPT
- The article does not explain how augmented entities are included in the trained the model
- The description of the NER task is unnecessarily too long for such a conference
Comments, suggestions and typos: line 45-47: I'm not quite convinced that NER really requires that extensive of medical knowledge of the pandemic. Maybe give an example of a complex example to back up the claim that this task requires extensive knowledge? 
 
 line 221:  Is WSLTM supposed to be LSTM? Not sure if this is a typo or a model I'm not familiar with. If this is the correct model name, please cite. 
 
line 222:  same with CCN. Is this CNN?

line 224: can achieves -> can achieve

line 246: as the size of the augmented instances increases

In general, I think the paper makes some hyperbolic claims about performance when the increase in scores is quite small and seems like it could be due to noise. A closer analysis of the types of errors made or what the content of the specific examples found  that differ between models would in my opinion be more informative and useful than trying to claim SOTA when you don't quite have a model that achieves SOTA. 

Typo line 290:  "Ethnic Statement" should be "Ethics Statement"

This study is interesting, but despite some merits regarding method and evaluation results, do not provide much additional results to previous similar studies. These data augmentation methods have been already presented in many works, I don't see any major original contribution in this paper. It would be nice to have a more detailed view of what kind of improvements are obtained concretely and how much it impacts studies in the domain (here biomedical and COVID). It would be nice to briefly explain how augmented entities are included in the trained model.
Overall assessment scores: 2.5

2.5
Metareview score: 2
Metareview: The paper proposes a model for recognizing named entities related to the pandemic on social media. The main idea is to investigate data augmentation using a closed LLM (ChatGPT). The performance is comparable to that of SOTA systems.
Summary of reasons to publish: - The paper is generally well-written and easy to follow.
- The experimental section is good.
- The application domain is relevant.