Unlocking Zero-Shot Biomedical Named Entity Recognition using LLM-Annotated Data
Authors: Rian Touchent, Éric Villemonte de la Clergerie


Reviewer paper summaries: This short paper presents an approach for medical NER which consists of two main components: (1) generating a large amount of domain specific training data using LLMs (Meta-Llama3-70B instruct) and (2) applying GliNER, an architecture specialized in zero-shot sequence tagging such as NER.

The training data generation is largely based on Bogdanov (2024) and the evaluation was performed on 5 different medical NER datasets for English and 3 for French. Results show that GliNER trained on the synthetic data obtains slightly better results than the baselines for English. Regarding French, best results are obtained with XLM-R and trained on the both en and fr synthetic data.
Summary of strengths: The paper addresses a very relevant topic, namely, how to address the lack of training data for an important task such as medical NER. It does test GliNER on a different specialized domain, with good results using automatically generated data for training.
Summary of weaknesses: This is a short paper, but still it is difficult to see what the contribution or novelty is. The method used to generate the training data is from some other paper, also the methods to perform the NER task. Thus, the only novelty is the training data itself. This could be OK if the comparisons and analyses were providing interesting insights. 

However, comparisons are not that comprehensive, as other well-known models could be included, for example specialized encoders such as Scibert or general ones like mDeBERTa has been tested with good results in these tasks. Also  text-to-text models such as Flan T5, Scifive for English and Medical mT5 (https://aclanthology.org/2024.lrec-main.974/)for English, Spanish, French and Italian have obtained very strong results on medical NER adding the possibility of multitask training. This last work also investigates crosslingual transfer with good results for several languages, including the ones investigated in this paper. 

Regarding crosslingual transfer, another interesting experiment is to compare with data-transfer, namely, translating automatically and projecting the annotations from the original training data (e.g. English) to the target language (e.g. French). There is quite a few papers investigating crosslingual transfer (https://aclanthology.org/2022.findings-emnlp.478/ and https://aclanthology.org/C18-1071/ and check their related work sections) also for the medical domain (https://arxiv.org/abs/2301.10527) and if interested, I think the authors could compare with some of the approaches included in those papers.

Finally, another alternative is to manually annotate a few samples and use few-shot approaches such as EntLM (https://aclanthology.org/2022.naacl-main.420.pdf).

A minor issue regarding the generation of the training data. There are also decoder-only models specialized to the medical domain such as BioMistral or PMC-Llama, which could be included in the experiments to generate the training data (and also to do the NER itself). I wonder if using specialized decoders is helpful to obtain better synthetic data.

Summarizing, the only real novelty is the training resource generated but the analysis and experimentation included could be more meaningful in a way that allows to draw interesting conclusions.
Comments, suggestions and typos: I think the objective of the paper is worthwhile, but more meaningful experimental comparison is required to draw interesting conclusions.
Overall assessment scores: 2