NERvous About My Health: Constructing a Bengali Medical Named Entity Recognition Dataset
Authors: Alvi Aveen Khan
Fida Kamal
Nuzhat Nower
Tasnim Ahmed
Sabbir Ahmed
Tareque Mohmud Chowdhury
Venue: EMNLP 2023 Findings (2023)
Keywords: Named Entity Recognition
Natural Language Processing
Consumer Health

Abstract: The ability to identify important entities in a text, known as Named Entity Recognition (NER), is useful in a large variety of downstream tasks in the biomedical domain. This is a considerably difficult task when working with Consumer Health Questions (CHQs), which consist of informal language used in day-to-day life by patients. These difficulties are amplified in the case of Bengali, which allows for a huge amount of flexibility in sentence structures and has significant variances in regional dialects. Unfortunately, the complexity of the language is not accurately reflected in the limited amount of available data, which makes it difficult to build a reliable decision-making system. To address the scarcity of data, this paper presents 'Bangla-HealthNER', a comprehensive dataset designed to identify named entities in health-related texts in the Bengali language. It consists of 31,783 samples sourced from a popular online public health platform, which allows it to capture the diverse range of linguistic styles and dialects used by native speakers from various regions in their day-to-day lives. The insight into this diversity in language will prove useful to any medical decision-making systems that are developed for use in real-world applications. To highlight the difficulty of the dataset, it has been benchmarked on state-of-the-art token classification models, where BanglishBERT achieved the highest performance with an F1-score of $56.13 \pm 0.75$%. The dataset and all relevant code used in this work have been made publicly available.

NERvous About My Health: Constructing a Bengali Medical Named Entity Recognition Dataset
Alvi Aveen Khan', Fida Kamal', Nuzhat Nower', Tasnim Ahmed Sabbir Ahmed', and Tareque Mohmud Chowdhury!
Islamic University of Technology "Queen's University (allvavven, fidakamal, nuzhatnower, sabbirahmed tartque))iiu--dhaaaa edu 2 tasnim. ahmed@queensu.ca
Abstract
rately answer question, the system must be ca- pable of identifying important entities in the text. This is even more critical when addressing health- related queries from the general public, known as Consumer Health Questions (CHQs). CHQs are often difficult to answer due to the informal lan- guage used (Mishra and Jain, 2016), and most pa- tients only provide vague description of their is- sues. This presents unique challenge to current systems, most of which were developed for use with medical documents that use formal language and concrete scientific terms to describe health is- sues (Carini et al., 2021). Being able to extract the correct medical knowledge required to answer the question from the vague descriptions contain- ing unstandardized medical terms used by the gen- eral public challenging task.
The ability to identify important entities in text, known Named Entity Recognition (NER), is useful in large variety of down- stream tasks the biomedical domain. This is considerably difficult task when work- ing with Consumer Health Questions (CHQs), which consist of informal language used in day-to-day life by patients. These difficul- ties are amplified the case Bengali, which allows for huge amount of flexibil- ity in sentence structures and has significant variances regional dialects. Unfortunately, the complexity of the language is not accu- rately reflected the limited amount of avail- able data, which makes it difficult build a reliable cerision-making system. To ad- dress the scarcity of data, this paper presents "Banhla-HealtNNERR; comprehensive dataset designed to identify named entities in health- related texts
The difficulties described are further exacer- bated in the Bengali language, which poses greater challenges compared to English for several rea- sons. Bengali allows for significantly larger num- ber of inflections, with 220 inflections existing in total compared to just 9 for English (Bhattacharya et al., 2005). This forces the models to learn a large number of words that are nearly identical but have slightly different meanings. For example, in the sentence 'I have fallen ill', the verb 'fallen' re- mains unchanged even if the subject is changed to 'You' or 'She'. Bengali, however, the same sen- tence ( ('anfee cार काςद I') will change the verb depending on the subject: 'अदς for the first per- son, "शाााा"" for the second-person, and "oाकर for the third-person. There is also greater flexi- bility regarding sentence structuring. Whereas the order of words is relatively rigid in English, and breaking the order makes sentences grammatically incorrect, the same is not true for Bengali. For ex- ample, 'I have headache" is correct in English, but 'headache have sounds odd and is unlikely to used. However, the Bengali versions of these sentences ( (भपासी вारा काश and "सर काश रााा re-
Bengali language. con- sists of 31,783 samples sourced from popular online public health platform, which allows it capture the diverse range linguistic styles and dialects used by native speakers from var- ious regions in their day-to-day lives. The in- sight into this diversity in language will prove useful to any medical dacision-making sys- tems that are developed for use in rorl-world applications. To highlight the difficulty of the dataset, it has been benchmarked on state-of- the-art token classification models, where Ban- glishBERT achieved the highest performance with an Fi-score of 56. 13 + 0.75%. The dataset and all relevant code used in this work have been made publicly available!
1 Introduction
A fundamental component of nearly all automated Question Answering (QA) systems is Named Entity Recognition (NER) model (Nadapana and Kommanti, 2022; Aliod et al., 2006). To accu-
'tttps: t/ub http.//ttucco///a/aa/ hoom/aliikhaan Hangla-Healthie

---

spectively) are both correct and commonly used. Furthermore, there large amount of diversity the language as there are numerous dialects that are so significantly different from each other that people from one region are often entirely unable to comprehend the language of another (Shahed, 1993). These attributes make Bengali an incredi- bly complex language with huge amount of vari- ation and intricacies.
Metric
Count
Sample count
31,783
Sentences
144,136
Percentage of entity tokens
23.47%
Avg. words per sample
175.81
Table Statistical metrics of Bangla- HealthNER
Unfortunately, the amount of data available for Bengali, especially in a medical context, is severely limited and fails to reflect the complex nature of the language. This makes it challeng- ing create system that accurately understands the language in the context of CHQs. To address the
formal language, differing greatly from the lan- guage people use in their day-to-day speech. The telemedicine dataset was compiled by manually transcribing telephone conversations with patients, while BanglaBioMed was created by extracting text from health articles published in a popular na- tional newspaper.
this issue, present BBanglaHHealthNe largest human-annotated Bengali medical NER dataset comprising 31,783 samples. The data has been collected from a public online health plat- form in Bangladesh, ensuring that it accurately captures the variety and complexity of the lan- guage used by native Bengali speakers in their day- to-day lives. The samples also consist of a large amount of text that combines Bengali and English, using words from both languages or transliterating words from one language using alphabets from the other, a common occurrence in informal conversa- tions known as 'code-switching.. The large vari- ety in language present in the dataset should allow models to gain thorough understanding of medi- cal terminology in informal contexts.
The small dataset sizes combined with their for- mal nature means that systems trained the ex- isting datasets will be unable to perform well in real-world cases. The introduction of Bangla- HealthNER addresses both of these limitations since is larger in size and also accommodates much more variety in language.
3
The Bangla- Dataset
3.1 Data Collection
The dataset was collected from a popular Ben- gali online medical health platform", which pro- vides publicly available health-related queries from users as well as answers by medical profes- sionals. Both the questions and answers were col- lected and are considered as separate samples in the dataset. Samples were collected via random sampling in order obtain an unbiased represen- tation of the data. An overview of various met- rics for the dataset and the different entities is pre- sented in Table 1 and Fig. 1, respectively. Com- pared to existing datasets, Bangla-HealthNERR not only has significantly more samples but also much longer samples, which mostly consist of multiple sentences. Lengthier training data will force sys- tems to learn retain contextual information for longer distances to be able accurately identify entities.
2 Literature Review
Although there a significant amount of data available for general Bengali NER (Haque et al.. 2023; Chowdhury et al., 2018; Shahgir et al., 2023), the availability of data specifically tai- lored for Bengali medical NER remains limited. Currently, there are two datasets publicly avail- able: one introduced by Islam et (2022), which we will be referring to in this paper as the 'Telemedicine Dataset', and the "BanglaBioMed' dataset (Sazzed, 2022). Both of these datasets have relatively small sizes, with the Telemedicine dataset consisting of 190 samples and nine en- tity labels and BanglaBioMed consisting of 100 sentences and four entity labels. This makes the datasets unsuitable for modern transformer-batt architectures, which rely on large amounts of data for effective learning. Additionally, the data for both datasets were collected from sources that use
3.2 Pre-Processing
The dataset was first cleaned by removing dupli- cate entries, URLs, and spam text. Afterwards, personally identifiable information was removed.
Pnttpss//Gaktarbbai.com

---

33548
49.4%
Symptom
Health Condition
Dieinee Emiiies
Medicine
Specialist
Age
17862
3.3%
Dosage
15000
7.1%
Medical Procedure
10000
8006
5585
14.6%
3000
5752
16.2%
5.5%6.0%
Symppoon
Specialist
Age
Dosage
Medical
Condition
(a) Distribution entities types
(b) Comparison distinct number entities per type
Atenge ofTookm Eneyy
2.45
2.37
2.09
1.9
Symptom
Specialist
Age
Condition
(c) Comparison the average number tokens per en- tity each entity type
Figure 1: Statistics for individual entities of Bangla Bangla-Healthhe HealthNER
Entity
Description
Symptom
Indicators of an underlying issue, such as pain (काा))) fever (G)4), etc.
Health Condition
Diseases like sinusitis (लरकगा)लल or health events like pregnancy (I9ाGाl))
Age
Numbers (numeric or textual) that specifically convey the age of a patient, e.g.. 5 months (9i6 |ाभ), years (2 तहœ), etc.
Medicine
Names of medicine like Omidon, Paracetamol ((اا)))ūا)))) etc.
Dosage
The amount of medication take, such tablet day, spoonfuls of syrup, etc.
Medical Procedure
Medical tests like Ultrasound (भा))ा) or procedures like surgery (भाक)ी)) Names of specific medical specialists like Medicine Specialist (लरलिल रिका)ा, Urologist (िडलाा(ाललि)) etc.
Specialist
Table 2: Entity descriptions for
Names were removed with the help a general Bengali NER model followed by manual inspec- tion, and email addresses and phone numbers were removed using regular expressions.
notation process, the details which are provided in Appendix A.2. The annotations follow the Muside-Outside-Bageee
(IOB) format with non- overlapping entities and consist of seven types of entities as described in Table 2. Five percent of the samples given to the annotators were taken from pre-annotated subset of the dataset and were used to calculate the Inter-Annotator Agreement (IAA). The guidelines, entity definitions, and pre- annotated data were all carefully prepared by the authors after extensively studying several works related to the domain (Li et al., 2016; Sazzed, 2022: Bodenreider, 2004). The quality of the fi-
3.3 Annotation
The dataset annotated using open-source annotation tool+ by team of 20 undergraduate students with least higher secondary level of formal education in biology. They were provided with extensive guidelines to follow during the an-
h/titps://github.. htwp:///ww.ccmmeeeeooo//e/eeeeaa

---

OJ. This distinction occurs because the name of the medicine, 'Montair', in the second case im- plies that it is a medicine, thus making the word 'medicine' redundant, unlike in the first case. Sim- ilarly, the phrase "तस का भहा लाकन ('ENT Doc- tor') which should be tagged as [B-Specialist, I- Specialist, I-Specialist, I-Specialist] but the phrase "तर शात काी रिमन Gाकन ('ENT Specialist Doctor') should be tagged [B-Specialist, I- Specialist, I-Specialist, I-Specialist, O]. Again, the word 'doctor' in the second case redundant since is implied by the words 'ENT Specialist'.
0.95
03935
0.M57678 0.926978
0.00
0.995510
0.85
0UR36722
01800193
0.80
0,748917
Non Eaity
Symptom Ilaath
Medicine
Dosige
Medical Proczdure
Specialist
Condiiin
Figure 2: Average IAA F1-Score (non-weighted) per
entity type
nal dataset was also ensured by following the work of Kabir et al. (2023). The average weighted F1-score for TAA was found to be 88.56%, and the average Cohen's Kappa score (Cohen, 1960) was found to be 67.19% (substantial agreement). An entity-wise breakdown of the average IAA F1- Score is provided in Fig. 2.
Benchmarking
To establish benchmark, the dataset was fine-tuned on three token classification models- BanglaBERT (Bhattacharjee et al., 2022), Ban- glishBERT (Bhattacharjee et al., 2022), and mBERT (Devlin et al., 2019). Aside from this, to analyze the performance of large language mod- els on Bengali medical NER, we also included the inference results of zero-shot ChatGPT (gpt-3.5- turbo, accessed June 2023) (Brown et al., 2020).
The annotation process revealed certain issues with the existing tagging formats used for NER tasks. For example, consider the phrase `oल ,6 पर,, which can be translated word-for-word into 'arm severe pain'. According to our guide- lines, the intensifier word '5ोz" should not be con- sidered part of the symptom so that the accurate tags would be [B-Symptom, O, I-SymptomJ. Un- fortunately, the IOB format does not allow sin- gle entity be split into parts in this man- ner. This issue does not occur English datasets since adjectives nearly always come before or af- ter nouns in the English language, which why the word-for-word translation is not grammatically correct. The grammatically correct version of this phrase, 'severe arm pain', would have the tags [O, B-Symptom, I-Symptom]. As consequence of this shortcoming of the tagging format, the anno- tators were frequently forced to deviate slightly from the guidelines and include the non-entity to- kens that appeared in between entity tokens. The issue was particularly pronounced for the 'Symp- tom' and 'Health Condition' entities since the in- formal nature of the data meant that patients fre- quently described issues in verbose manner.
For the benchmarking process, the dataset was divided into training, validation, and test splits us- ing the ratio 80 10 10. Further details about the experimental setup provided in Appendix A.1, and Table 3 provides the results of our ex- periments. The scores for the evaluation metries are calculated at the token level and are averaged across multiple runs except in the case of GPT 3.5. Both BanglaBERT and BanglishBERT outper- formed the mBERT model, which reinforces the understanding that models pre-trained spe- cific language achieve better performance on downstream tasks than multilingual ones. Ban- glishBERT achieved the highest scores, likely due to the presence of text that combines Bangla and English in the dataset.
The GPT 3.5 model was not fine-tuned dataset but was given prompts to assign labels each word in the test samples. Past literature study- ing the performance of ChatGPT on downstream tasks related to bio-medicine without fine-tuning found that domain-specific models achieve signif- icantly better results, even if expert prompting is used (Jahan et al.. 2023). Our results agree with this conclusion, all the fine-tuned models out- perform zero-shot ChatGPT by large margin.
It was also found that there were subtle vari- ations in very similar concepts, which should create challenging situations. For example, the phrase 'सरस शरा पशर. ("headache medicine") ) should be tagged as [B-Medicine, I-Medicine, I- Medicine] whereas the phrase "Øादдा" NaTe' ('Mon- tair medicine") should be tagged as [B-Medicine,
A thorough error analysis of the predictions the fine-tuned models reveals that the expected

---

Model
Accuracy (%)
Precision (%)
Recall (%)
F1-Score (%)
GPT 3.5 (zero-shot)
72.43
28.62
10.64
15.51
mBERT
87.98 + 0.25
49.46 + 1.15
61.90 + 0.47
54.98 + 0.82
BanglaBERT
87.90 = 0.01
48.00 + 0.38
63.66 = 0.29
54.74 0.34
BanglishBERT
88.42 = 0.14
50.20 + 0.62
63.65 = 0.95
56.13 + 0.75
Table 3: Benchmark results for Henghe-
challenging situations discussed in section 3.3 did indeed cause issues. The models seemed to have difficulty understanding subtle differences in sim- ilar concepts. For example, a large number of the responses to questions concluded with advice to visit specialist, such as नपरनकार A,9R,.f. f8, लिान लराा l' ('Please visit an E.N.T. specialíst.'), It was observed that samples that concluded slightly differently, such भाराका शागपতानत Fre रिका लाााारपा कशन I' ('Please contact the children's unit of BIRDEM Hospital') resulted in the models marking incorrect phrases, 'children's unit' this case, with the label 'Specialist'. Similar issues were also seen with the 'Age' class, where the models marked any mention of time in years, such 'O पहा wiल रा cots सिापयिाा I' ('My arm broke 3 years ago") with the label 'Age'. This most likely caused by the majority of such mentions in the dataset genuinely being the patient's age.
sentence structures, and regional dialects, are ob- served in various downstream tasks, meaning our
findings can be generalized broader contexts and applications. The dataset can also be lever- aged to achieve cross-domain adaptability in fields such as social media, customer reviews, and on- line forums, where users communicate using collo- quial and unstructured language. The proposed in- sights also have cross- poss-llinguisti transferability other Indo-Aryan languages with similar linguis- tic characteristics. This could prove beneficial in developing NER models for languages with lim- ited labelled data. The adaptability of NER as a feature can facilitate automated clinical decision- making systems as well.
Limitations
One limitation that caused notable difficulties dur- ing the annotation process was the shortcomings of the tagging format used, as discussed sec- tion 3.3. Despite our extensive guidelines, the un- structured nature of the Bengali language resulted in our annotators being forced to mark words that separated single multi-word entity part of the entity due to the limitations of the tagging format. This inevitably had a negative effect on the fine- tuning process. Working towards establishing a tagging format that addresses these limitations and fixing the annotations of this dataset accordingly is likely to lead to improved performance.
5 Conclusion
In this paper, we introduced a manually anno- tated Bengali Named Entity Recognition dataset of health-related texts. The paper highlights sev- cral key features of the dataset, including its abil- ity to capture the linguistic diversity and complex- ity of the Bengali language, as well as its emphasis on the informal nature of speech used by the aver- age patient in medical contexts. We evaluated the dataset Bengali and multilingual token classifi- cation models establish benchmark and further explored the performance of the large language model GPT 3.5 on the dataset. Given that the dataset introduces significant variety in language to the domain compared to existing data, archi- tectures trained on past data struggled to perform well. This opens up a new direction for future work, which can concentrate exploring meth- ods to improve this performance.
Ethics Statement
The privacy of the patients of the health platform was a top priority during the data collection pro- cess. As such, the data was de-identified through manual inspection to ensure the removal of all per- sonal information. All of the annotation work was undertaken either by the authors themselves by hired individuals. The annotators were provided with monetary compensation for their work, which above the minimum wage. The annotation pro- cess has also been de-identified to prevent any pri- vacy violations of the annotators.
The research and data presented in this paper have implications that extend beyond the domain NER in CHQs. The challenges addressed, such as dealing with informal language, complex

---

Acknowledgements
Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186.
The funding for this research was provided by the Islamic University of Technology (IUT) under the IUT Research Seed Grants (IUT RSG). We are grateful for their support.
Md. Zahidul Haque, Sakib Zaman, Jillur Rahman Saurav, Summit Haque, Md. Saiful Islam, and Mo- hammad Ruhul Amin. 2023. B-ner; novel bangla named entity recognition dataset with largest entities and its baseline evaluation. IEEE Access, 11:5194-- 45205.
References
Diego Mollá Aliod, Menno van Zaanen, and Daniel Smith. 2006. Named entity recognition for question answering- In Australasian Language Technology Association Workshop.
Tanvir Islam, Sakila Mahbin Zinat, Shamima Sukhi, Zakir Hossain Zamil, Aynur Nahar, and MF Mridha. 2022. An attention- based medical ner in the bengali language. In Proceedings of 2nd International Con- ference on Artificial Intelligence: Advances and Ap- plications: ICAIAA 2021, pages 131-140. Springer.
Abhik Bhattacharjee, Tahmid Hasan, Wasi Ahmad, Kazi Samin Mubasshir, Md Saiful Islam, Anindya Iqbal, M Sohel Rahman, and Rifat Shahriyar. 2022. Banglabert: Language model pretraining and bench- marks for low-resource language understanding evaluation in bangla. In Findings of the Association for Computational Linguisties: NAACI, 2022, pages 1318-1327.
Israt Jahan, Md Tahmid Rahman Laskar, Chun Peng. and Jimmy Huang. 2023. Evaluation of chatgpt biomedical tasks: A zero-shot comparison with fine- tine-tuned generative transformers. arXiv preprint brXiv:2306.04504
Mohsinul Kabir, Tasnim Ahmed, Md Bakhtiar Hasan, Md Tahmid Rahman Laskar, Tarun Kumar Joarder, Hasan Mahmud, and Kamrul Hasan. 2023. Deptweet: A typology for social media texts to de- tect depression severities. Computers in Human Be- havior, 139:107503.
Samit Bhattacharya, Monojit Choudhury, Sudeshna Sarkar, and Anupam Basu. 2005, Inflectional mor- phology synthesis for bengali noun, pronoun and verb systems. In Proc. of the National Conference on Computer Processing of Bangla (NCCPB 05), pages 34-33.
Loic Lannelongue, Jason Grealey, and Michael In- ouye. 2021. Green algorithms: quantifying the car- bon footprint of computation. Advanced science, 8(12):210777.
Olivier Bodenreider. 2004. The unified medical lan- guage system (umls): integrating biomedical termi- nology. Nucleic acids research, 32:D267-DDD -D270.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan. Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 1977-199..
Jiao Li, Yueping Sun, Robin J Johnson, Daniela Sci- aky. Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn Mattingly, Thomas C Wiegers, and Zhiyong Lu. 2016. Biocreative edr task corpus: resource for chemical disease relation extraction. Database, 2016
Elettra Carini, Leonardo Villani, Angelo Maria Pez- zullo, Andrea Gentili, Andrea Barbara, Walter Ric- ciardi, and Stefania Boccia. 2021. The impact of digital patient portals on health outcomes, system ef- ficiency, and patient attitudes: Updated systematic literature review. Journal of Medical Internet Re- search, 23.
Amit Mishra and Sanjay Kumar Jain. 2016. A survey question answering systems with classification. Journal of King Saud Universiny-Coppuer and In- formation Sciences, 28(3):345-33)
Vasuki Nadapana and Hima Bindu Kommanti. 2022. Investigating the role of named entity recognition in question answering models. IEEE 3rd Global Conference for Advancement in Technology (GCAT), pages 1-7.
Shammur Absar Chowdhury, Firoj Alam, and Naira Khan. 2018. Towards bangla named entity recog- nition. In 21st International Conference of Com- puter and Information Technology (ICCIT), pages 1- IEEE.
Salim Sazzed. 2022. Banglabiomed: A biomedical named-entity annotated corpus for bangla (bengali). In Proceedings of the 21st Workshop on Biomedical Language Processing, pages 323-329.
Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological mea- surement, 20(1):37-45 46.
Syed Mohammad Shahed. 1993. Bengali folk rhymes: An introduction. Asian folklore studies, pages 143- 160.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for
H.A.Z. Sameen Shahgir, Ramisa Alam, and Md. Zarif UI Alam. 2023. Banglaconer: Towards robust bangla complex named entity recognition. ArXiv. abs/2303.09306.

---

A Appendix
3. Symptoms are indicators of health condi- tion. A health condition specific discase or medical event. Example: शाय भसत डायार गशकनगर शह भाअ
A.1 Experiment Setup
The experiments done to establish our benchmark results were run on an Nvidía 3090 GPU with 24 GB of VRAM using the Trainer library avail- able through Hugging Face and Cuda Version 11.6. The dataset was split into training, validation, and test sets at the ratio 80 10 10 and trained for 25 epochs on each of the models. A batch size of
शह पो: रमरे
4. Injuries are considered health conditions.
Example:
WHE अाेक (SC
128 was used along with an AdamW optimizer, linear learning rate scheduler, cross-entropy loss, 40% dropout, and a weight decay of 3e-2. A learn- ing rate of le-4 was used with the first 20% of the training steps being used for warm-up. The use of Gradient Checkpointing additionally allowed us to maximum token length of 512.
Unnecessary information, such as the sever- ity of a symptom, should not be annotated. Example:
8[ 3 Tcd पपत Cीन X
8 अ उलt परक लpन /
6. Redundant information (such as the word डाकत following the word -ललगगा))) should be annotated.
In the course our experiments, we also at- tempted to calculate an estimate our carbon footprint. We did this by following the work of Lannelongue et al. (2021). It was found that each run of the experiments under our hardware setup generated approximately 331.17 grams, 222.68 grams, and 216.97 grams of CO2 for the mBERT, BanglaBERT, and BanglishBERT models respec- tively. is interesting to see that the models that achieved comparatively better results in our ex- periments, BanglaBERT and BanglishBERT, also have a smaller impact on CO2 emissions.
Example: вाय अररा VaR
ना., कान, शरा Gाका
बाे, कात, शाी लिोे शाका
7. It is important to pay attention to contextual information when identifying entities. For example, the term 'Corona' can be used describe both the COVID-19 disease and the COVID-19 19 pandemic in Bengali.
Example:
काबात bनालगली लािन लाया भारक X
A.2 Annotation Guidelines
. रि लमलनमत लाकाी /
In addition to the basic instruction of labelling each word in the samples with one of the seven entity labels, the annotators were provided with ex- tensive instructions how address large va- riety of different scenarios that were seen to arise. These instructions are provided below. The colour code used for the examples as follows: Symp- tom, Health Condition, Age, Medicine, Dosage, Medical Procedure, Specialist.
8. Multiple symptoms mentioned together should be annotated separately.
Example:
bad cold and fever X bad cold and fever /
Medicine names can include the amount chemical in each intake.
Example:
लााता So fिaT शात X
1. Entities should be annotated as specifically as possible. If an entity contains a sub-entity, the top-level entity should be annotated. Example:
लातत So fिUा शात /
10. Doctors frequently write the dosage in the for- mat '0+1+1'.
Example:
TT 8 पित पात माबागा X
लाता So लिया 0+0+1
श 8 पित शलन मााागरा /
11. Medical specialists typically have uniquely
2. A single entity should not be divided into two parts, even if this requires including some necessary information. Example:
identifiable names (e.g., कहााতरपालिि) but
are sometimes referred to by more generic names (c.g., गोरिि हानाय.
Example:
पाता मार डा R सित (ात पा Cरशl कराा X
परका कमलिररत पाकागर (रया
पााार मार शह R सि (C पाम GUU कराा


Reviews:
[{"review_id": "tFYwXDemFP", "reviewer": "Reviewer_LNMK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewers for their kind reviews. We hope to update the final version of our manuscript to address their expressed concerns and are including our responses herein.\n\n### Response to Reasons to Reject #1:\n\n**Insufficient Benchmark**: The number of models available that are capable of performing token classification on Bengali text is limited. There are only three language models that are widely used in the domain, BanglaBERT, BanglishBERT, and mBERT, all three of which have been included in our benchmark.\n\n**Code Reproduction**: We have provided details regarding the experimental setup and hyper-parameters in Appendix A1. Moreover, all the codes related to dataset curation, preprocessing, post-processing, baseline training, etc. will be publicly available upon the acceptance of the manuscript. We will put extra effort into improving the documentation of the code-base as much as possible to ensure reproduction.\n\n**Generalizability and broader contexts**: The research presented in this paper has implications that extend beyond its specific focus on Named Entity Recognition (NER) in Consumer Health Questions (CHQs). The challenges addressed by the curated dataset, such as dealing with informal language, complex sentence structures, and regional dialects, are observed in other downstream tasks for other languages as well. As a result, the findings of this research can be generalized to broader contexts and applications.\n\nThis dataset can be leveraged to achieve Cross-Domain Adaptability in the fields such as social media, customer reviews, and online forums, where users communicate using colloquial and unstructured language. Regarding Cross-Linguistic Transferability, the proposed insights are transferable to other languages with similar linguistic characteristics. This could prove beneficial in developing NER models for languages with limited labeled data. The adaptability of NER as a feature might facilitate automated Clinical Decision-making systems as well.\n\n### Response to Reasons to Reject #2:\n\n**Venue Suitability**: We thank the reviewer for the valuable suggestion. Also, we would like to bring to the reviewer's kind attention that the resources track of EMNLP is a diverse platform that makes room for a variety of tasks and has accepted similar kinds of resources on under-resourced languages such as Bengali in the past [1,2]. If accepted, our work has the potential to contribute significantly to the Clinical Health community and the exposure of such a venue like EMNLP would help our work reach its stakeholders.\n\n**References**:\n\n[1] Ekram, Syed Mohammed Sartaj, et al. \"BanglaRQA: A Benchmark Dataset for Under-resourced Bangla Language Reading Comprehension-based Question Answering with Diverse Question-Answer Types.\" Findings of the Association for Computational Linguistics: EMNLP 2022. 2022. (https://aclanthology.org/2022.findings-emnlp.186/)\n\n[2] Islam, Khondoker Ittehadul, et al. \"SentNoB: A dataset for analyzing sentiment on noisy Bangla texts.\" Findings of the Association for Computational Linguistics: EMNLP 2021. 2021. (https://aclanthology.org/2021.findings-emnlp.278/) \n\n### Response to Reasons to Reject #3:\n\nWe thank the reviewer for this valuable insight. For augmenting a language dataset, we can follow approaches such as context-based or rule-based augmentation. Since this is an NER task, context-based augmentation requires re-annotating the dataset which is not feasible in this time frame. However, rule-based augmentation that involves substituting tokens with synonyms could indeed improve the quality of the dataset and is something we will be looking into.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The constructive work of providing experimental resources for the Medical NER task in the low-resource language of Bengali is good, but the workload and contributions are not sufficient.", "reasons_to_reject": "While this research work appears to be primarily resource-oriented, the authors have made commendable contributions by undertaking substantial annotation tasks to enrich the language resources. However, several issues remain:\n\nInsufficient Benchmark Results: The inclusion of only GPT-3.5 and BERT benchmark results is inadequate. The authors should provide more solid comparative experiment details. There is a lack of information in the code section, making reproduction challenging. Please include more details to validate the generalizability of the research results or findings in broader contexts or applications.\n\nData Set Detail and Experimentation: More information about the dataset and broader experimental results on large language models and downstream task models are required. Also, consider augmenting this language dataset with samples from other language resources. This could help enhance the completeness of both the dataset and the corresponding research report.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper primarily addresses the problem of insufficient and inadequate datasets for Named Entity Recognition (NER) in the Bengali language, particularly in the context of Consumer Health Questions (CHQs). This limitation hinders the development of effective decision-making systems for medical applications intended to serve Bengali-speaking populations. The problem is complex due to the inherent flexibility and diversity in the Bengali language, along with its widespread informal usage and significant regional dialectal variations.\n\nThe main contributions of this paper are:\n\nNew Data Resource: The paper introduces 'Bangla-HealthNER,' a manually annotated Bengali medical NER dataset. With 31,783 samples, it is the largest available Bengali medical NER dataset. This dataset is gathered from a public online health platform in Bangladesh, ensuring a broad representation of day-to-day language usage in various linguistic styles and regional dialects. This contribution is particularly significant as it helps fill the gap for low-resource languages.\n\nReal-world Relevance: This dataset, given its source, incorporates the informal language commonly used by patients, adding a layer of realism and practical applicability that could prove vital to the development of medical decision-making systems.\n\nEvaluation and Benchmarking: The dataset has been evaluated on state-of-the-art Bengali and multilingual token classification models to establish a benchmark. BanglishBERT achieved the highest performance with an F1-score of 56.63%, highlighting the inherent difficulty and complexity of the task.\n\nPublic Availability: The authors intend to make the 'Bangla-HealthNER' dataset publicly available upon acceptance of the manuscript, fostering future research and development in this area.\n\nOverall, this paper contributes to the field of NLP in a low-resource language context, by providing a dataset and benchmarks for NER in Bengali medical language processing.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hL7Z7wVJhr", "reviewer": "Reviewer_V5Uk", "summary": "", "questions": "Was inter-annotator agreement computed per entity type? Based on the definitions, it can be expected that agreement would be higher for e.g. \"age\" or \"medication\" vs. \"symptom\" or \"health condition\".", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate that the reviewer took the time to go through our paper thoroughly. We hope to address all of the concerns they have expressed. \n\n**Response to Reasons to Reject #1**: A health condition includes both diseases and medical events. A symptom is an indicator of a health condition. For example, if someone had a bone fracture, the fracture itself would be the health condition while any pain they experience would be the symptom.\n\nPrevious work with CHQs under the English language has made use of UMLS specifically [1], and we did examine this work in the initial stages of our research. Unfortunately, the complete lack of medical terminology used in our data along with the informal nature of the language used in the samples meant that we could not determine mappings between the official UMLS terminology and our data to a useful extent.\n\nWe did, however, study the terminology quite thoroughly before establishing our entity definitions. We also studied the guidelines of several past medical NER datasets, as well as the ones on CHQ specifically, and used our findings as the basis for our entity definitions.\n\n**Reference**:\n \n[1] Shweta Yadav, Deepak Gupta, and Dina Demner-Fushman. 2022b. Chq-summ: A dataset for consumer healthcare question summarization. arXiv preprint arXiv:2206.06581.\n\n**Annotation Tool**: To meet the different requirements of our data curation process we developed our own customized annotation tool which had a user-friendly UI and support across different platforms. The issue of discontinuous entities does not arise due to the custom-built annotation tool, since there are alternatives that allow for this as the reviewer has pointed out. The decision to not use discontinuous entities was a conscious one. Studying past research, we found that by far the most commonly used format for NER annotations is the IOB format. Since the domain of Bengali medical NER is still in its infancy, we chose to follow this format so as to maximize the usability of our work.\n\n**Pre-annotation policy**: We used pre-annotated samples to calculate the Inter-annotator agreement.  The pre-annotations were produced initially by the authors while establishing the entities to be used. We studied existing datasets [1] and their guidelines extensively when creating these annotations to ensure they are of high quality. For evaluating the final annotation quality, we followed the methodology provided by Kabir et al. (2023).\n\n**Reference**:\n\n[1] Salim Sazzed. 2022. Banglabiomed: A biomedical named-entity annotated corpus for bangla (bengali). In Proceedings of the 21st Workshop on Biomedical Language Processing, pages 323–329\n\n[2] Kabir, Mohsinul, et al. \"DEPTWEET: A typology for social media texts to detect depression severities.\" Computers in Human Behavior 139 (2023): 107503.\n\n**Regarding lines 53-57**: Thank you for this critical observation from the reviewer. We acknowledge that we misspoke here by saying that ‘it is yet to be addressed’.\nWe will change the text with relevant references:\n\nCurrent Statement | Statement for Revised Manuscript\n---|---\nBeing able to extract the correct medical knowledge required to answer the question from the vague descriptions containing unstandardized medical terms used by the general public is a challenging task *that is yet to be addressed* | Being able to extract the correct medical knowledge required to answer the question from the vague descriptions containing unstandardized medical terms used by the general public is a challenging task.\n\n\n**Regarding results reported in Table 3**: The results shown in the paper were not averaged over several runs. We acknowledge that running for multiple trials will provide more statistical evidence on the overall performance of the models, so we have extended our experiments for multiple trials. The following table  shows the results for each of the three language models, mBERT, BanglaBERT and BanglishBERT, averaged over multiple runs. We will add this result in the Appendix section of the revised manuscript.\n\nModel | Accuracy (%) | Precision (%) | Recall (%) | F1-Score (%)\n---|---|---|---|---\nmBERT | 87.98 +- 0.25 | 49.46 +- 1.15 | 61.90 +- 0.47 | 54.98 +- 0.82\nBanglaBERT | 87.90 +- 0.01 | 48.00 +- 0.38 | 63.66 +- 0.29 | 54.74 +- 0.34\nBanglishBERT | 88.42 +- 0.14 | 50.20 +- 0.62 | 63.65 +- 0.95 | 56.13 +- 0.75\n\nFor the BanglishBERT model, which has the best F1-Score (56.13 +- 0.75%), the result remains similar to that in our original manuscript (56.63%). A similar pattern is observed for the other models as well. From this, we can conclude that, although averaging over multiple runs causes the results to vary slightly, it is not to an extent that should cause concern with the original results.\n\n**Regarding the Data-use agreement**: We have made every attempt to ensure that the data collection process followed existing data privacy laws. The scraping of publicly available data from websites is allowed under the data privacy laws of Bangladesh, and the website itself does not forbid any such collection or republishing under their terms.\n\nFor sharing the data, we would prefer to allow downloading the data via GitHub under the [CC BY-NC-SA 4.0 license](https://creativecommons.org/licenses/by-nc-sa/4.0/), as this will be the least restrictive for research purposes. However, if the reviewers feel that this could potentially create issues for us in the future, we can require an agreement before access is granted.\n\n**Regarding lines 326-332**: The reviewer has correctly pointed out that our process is more in line with de-identification rather than anonymization. We can rephrase lines 326 - 332 so as to not imply a formal guarantee of anonymization, since we do not provide proof of such.\n\n**Regarding CO2 impact**: We appreciate the suggestion and will include the results of the CO2 impact estimates as an appendix in our paper.\n\n**Regarding the Inter-Annotator Agreement**: We agree that reporting the inter-annotator agreement per entity type would provide further insights, and we also expect the results to be higher for certain classes than for others. We appreciate that this shortcoming was pointed out and will include the missing information in our paper. However, it should be noted that both the F1-score and Cohen's Kappa take class imbalance into account.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The manuscripts presents a new resource for medical named entity recognition in Bengali; the annotated corpus as well as annotation guidelines will be shared with the community. \n- The dataset is used to evaluate several masked language models (multilingual as well as Bengali)\n- An error analysis of the evaluation is reported\n\n## edited to add: ##\nThe authors' responses to reviewers comments help clarify a number of points and suggest that a revised version as outlined by the response would be improved over the initial submission and likely of interest to the community.", "reasons_to_reject": "- The annotation process needs to be described in further details. For example: \n\t* The definitions of the entities in Table 2 seem rather vague. In particular, what is the difference between \"symptom\" and \"health condition\"? Was anchoring to a knowedge source such as the UMLS considered? Were other annotated corpus and annotation guidelines considered? A number of annotation schemes for biomedical text are compared and discussed in Deléger et al. 2017; this can be used as a pointer to existing annotated corpora in the biomedical domain and to provide some context to the work presented in terms of entities selected and definitions used. \n\t* was an annotation tool used? Which one? How was it selected? I recommend refering to Neves and Seva 2019 for an in-depth review of annotation tools; l.201-204 seems to describe the issue of discontinuous entity, which as been addressed in English and other languages. Some annotation tools such as BRAT do support the production of discontinuous entities - as well as nested entities. \n\t* l. 186 suggests that pre-annotations were supplied to the annotators. How were the pre-annotations produced? Was the pre-annotation quality evaluated, e.g. by comparing to final annotations? \n- lines 53-57 seem to describe the general field of NLP for social media and user generated content which has been extensively studied in the past decades for the biomedical field. Example of relevant work can be found in Conway et al. 2019 or Gonzalez et al. Much of this work has addressed English, but some other languages too.  \n- the differences in results reported for the different models in Table 3 seem very close. Are they significant? Are the results averaged over several runs? Please see Reimer and Gurevitch 2017 on why this might be needed. \n- The manuscript does not report information on the data use agreement / user consent that was obtained during the data collection process. Similarly, licence / data use agreement for the corpus that will be shared is missing. \n- The process described lines 326 - 332 seems consistent with de-identification (removal of directly identifying information) rather than anonymization (formal guarantee that the link between data and patient cannot be recovered by any means). \n- The report of compute use in appendix A.1 could be complemented by CO2 impact estimates, which can be obtained from e.g. http://calculator.green-algorithms.org/  without re-running experiments.", "questions_to_authors": "Was inter-annotator agreement computed per entity type? Based on the definitions, it can be expected that agreement would be higher for e.g. \"age\" or \"medication\" vs. \"symptom\" or \"health condition\".", "paper_topic_and_main_contributions": "The manuscript reports on the development of a dataset for the evaluation of named entity recognition for the biomedical field in Bengali. The study corpus is obtained from an online health platform. Seven entity types are covered. Annotations were produced by a team of 20 annotators with substantial agreement. The annotated corpus is then used in an evaluation of masked language models (multilingual as well as Bengali).", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Conway M, Hu M, Chapman WW. Recent Advances in Using Natural Language Processing to Address Public Health Research Questions Using Social Media and Consumer Generated Data. Yearb Med Inform. 2019 Aug;28(1):208-217. doi: 10.1055/s-0039-1677918. Epub 2019 Aug 16. PMID: 31419834; PMCID: PMC6697505.\n\nDeléger L, Campillos L, Ligozat AL, Névéol A. Design of an extensive information representation scheme for clinical narratives. J Biomed Semantics. 2017 Sep 11;8(1):37.\n\nNeves M, Ševa J. An extensive review of tools for manual annotation of documents. Brief Bioinform. 2021 Jan 18;22(1):146-163.\n\nNils Reimers and Iryna Gurevych. 2017. Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 338–348, Copenhagen, Denmark. Association for Computational Linguistics.\n\nalso G Gonzalez et al. : Social Media Mining for Health (#SMM4H) shared tasks since 2016\n\n", "Missing_References": "Conway M, Hu M, Chapman WW. Recent Advances in Using Natural Language Processing to Address Public Health Research Questions Using Social Media and Consumer Generated Data. Yearb Med Inform. 2019 Aug;28(1):208-217. doi: 10.1055/s-0039-1677918. Epub 2019 Aug 16. PMID: 31419834; PMCID: PMC6697505.\n\nDeléger L, Campillos L, Ligozat AL, Névéol A. Design of an extensive information representation scheme for clinical narratives. J Biomed Semantics. 2017 Sep 11;8(1):37.\n\nNeves M, Ševa J. An extensive review of tools for manual annotation of documents. Brief Bioinform. 2021 Jan 18;22(1):146-163.\n\nNils Reimers and Iryna Gurevych. 2017. Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 338–348, Copenhagen, Denmark. Association for Computational Linguistics.\n\nalso G Gonzalez et al. : Social Media Mining for Health (#SMM4H) shared tasks since 2016"}}, {"review_id": "eeJSwqfNRq", "reviewer": "Reviewer_NwnB", "summary": "", "questions": "Q1: Can you give more details how the texts were selected from platform, e.g. random sampling or by using certain queries? (lines 150-155)\n\nQ2: Why did you opt to divide the answers and the questions into separate samples (lines 154,155)? Can the connection still be reconstructed from the annotated dataset?  For better usability, it would be beneficial if the connection could be made.\n\nQ3: Can you confirm that you have taken all legally necessary measures to anonymize the texts? In some countries, in addition to names, telephone numbers and email addresses, other secondary information (such as locations, age, etc.), which may be used to indirectly identify individuals, must be anonymized, too. \n\nQ4: How are the results metrics of the benchmark computed – on token or on mention level? How is accuracy defined here? I guess accuracy on token level, right?  \n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our gratitude for the invaluable feedback you provided regarding our manuscript. We hope to address your concerns comprehensively in the subsequent text.\n\n**Code Reproduction**: We have provided details regarding the experimental setup and hyperparameters in Appendix A1. Moreover, all the codes related to dataset curation, preprocessing, post processing, baseline training, evaluation criteria, etc., will be made publicly available upon the acceptance of the manuscript. We will put extra effort into improving the codebase documentation as much as possible to ensure reproduction.\n\n**Anonymity measure**: As described in lines 168-173, the anonymization process involved the automated removal of personally identifiable information and was followed by a manual inspection by the annotators during the annotation process. The manual nature of the annotation process means that every sample was viewed by at least one human annotator at some point in time. We are confident that any personally identifiable information has definitely been caught in this process.\n\n**Text selection process from the platform**: The source of the data was from a publicly available Digital Health platform, and the texts were collected via random sampling. Specific queries were not used since our work concentrates on general healthcare and not specific subcategories.\n\n**Regarding treating the question and answer samples separately**: We opted to separate the questions and answers since our goal was to extract the NERs from a given statement regardless of it being a question/answer sample. Hence the connection between the Q-A was not preserved.\n\nHowever, since the reviewer suggests that retaining the connection would be beneficial, we can also release a version of the annotated dataset that retains the connection. Unfortunately, our benchmark results would not be applicable to that version. \n\n**Legal issues regarding Anonymity measure**: We have taken care to remove any and all information that is considered personally identifiable under the data privacy laws of Bangladesh.\n\n**Result metrics**: We have computed metrics for accuracy, precision, recall, and F1-score at the token level. The score reported in the abstract is for the F1-Score. We will revise the manuscript and remove any ambiguity regarding this issue. \n\n**About extending Table 1**: We appreciate the suggestion and will extend Table 1 to include this information. A portion of the information that will be included is provided below:\n\n**No. of Distinct Entities per Entity Type**\nEntity Type | Count\n--- | ---\nSymptom | 33,548\nHealth Condition | 8,171\nMedicine | 17,862\nSpecialist | 7,024\nAge | 5,585\nDosage | 8,006\nMedical Procedure | 3,752\n\n**Average Lengths (No. of Tokens) per Entity Type**\n\nEntity Type | Average No. of Tokens\n--- | ---\nSymptom | 3.96\nHealth Condition | 2.33\nMedicine | 2.43\nSpecialist | 2.09\nAge | 1.90\nDosage | 4.91\nMedical Procedure | 2.37\n\n**Regarding tagging format**: We also briefly considered annotating the offsets, but as the reviewer has pointed out, our own examination of past work also suggested that this is not a commonly chosen approach, which is why we ultimately chose to not pursue it for now. In our humble opinion, the rarity of this approach combined with its apparent importance in languages such as Bengali is indicative of the need for further research in this area.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-\tThe work provides a resource for non-English language texts in the biomedical field. Resources in this field are very scarce and the dataset thus represents a valuable contribution for the research community.\n-\tThe authors give a convincing motivation for the work and the article is essentially written in a comprehensible way (even if details are missing - see reasons to reject).\n", "reasons_to_reject": "-\tThe paper misses to describe essential properties of the created dataset (e.g. number of (distinct) entities per type etc. – see detailed comment below), which complicates the assessment of the resource created\n-\tDetails of the benchmark and the evaluation protocol are missing, which limits the reproducibility.\n-\tI am not entirely sure whether all necessary measures have been taken to ensure the anonymity of persons / authors of the texts collected.\n", "questions_to_authors": "Q1: Can you give more details how the texts were selected from platform, e.g. random sampling or by using certain queries? (lines 150-155)\n\nQ2: Why did you opt to divide the answers and the questions into separate samples (lines 154,155)? Can the connection still be reconstructed from the annotated dataset?  For better usability, it would be beneficial if the connection could be made.\n\nQ3: Can you confirm that you have taken all legally necessary measures to anonymize the texts? In some countries, in addition to names, telephone numbers and email addresses, other secondary information (such as locations, age, etc.), which may be used to indirectly identify individuals, must be anonymized, too. \n\nQ4: How are the results metrics of the benchmark computed – on token or on mention level? How is accuracy defined here? I guess accuracy on token level, right?  \n", "paper_topic_and_main_contributions": "The article describes the creation and annotation of a Bengali dataset consisting of biomedical texts from an online consumer health platform. The authors describe the annotation procedure and conduct initial experiments with transformer-based BERT models and ChatGPT.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "-\tTable 1 should be extended to include more detailed information about the created dataset, i.e number of (distinct) entities per entity type, average number of entities per text type (queston, answer), metrics concerning the lengths of the entities (e.g. average/median). This information help to get a better insight into the created dataset.\n-  A small note: The shortcomings of the tagging format described in 3.3 could be circumvented by annotating the offsets in the text instead of IOB labels (and allowing an entity to span several offsets). However, this option is very rarely chosen and the number of approaches that support prediction of such non-consecutive entities seems even smaller to me. "}}]

Author rebuttal:

Meta-review: The main contribution of this paper consists in the construction of a new manually annotated dataset and a benchmark for NER in Bengali for medical language processing. The experimental evaluation, though limited, appears to be sufficient for assessing quality and justifying the effort.  Reviewers and authors engaged in a constructive discussion, and the authors' responses clarified numerous points and show willingness to incorporate reviewers' suggestions into the next version of the paper. Reviewers finally converged on Soundness and two of found the paper exciting. Overall, this short paper can indeed be of great interest to the EMNLP community. 

**Pros** 

- coverage of a non-English, less-resourced language (Bengali); 

- coverage of a significant domain, biomedicine; 

- coverage of ordinary language usage, various linguistic styles, and regional dialects. 

- creation of the largest dataset available for Bengali, with and evaluation against SOTA for Bengali; 

- findings appear to be generalizable to other dialogical, informal contexts; 

- the paper is easy to follow and understand. 

 

**Cons** 

- as this paper focuses on a data resource, it lacks essential details regarding the annotation process and dataset composition properties, as pointed out by Reviewers V5Uk and NwnB. Nevertheless, the authors have shown awareness of these missing details, which will be included in the revised version; 

- The significance of the performance differences among the various models is not self-evident and would need a more thorough investigation. Authors however performed an extra test and provided the results in the discussion, which clarifies the point and should be included in the paper or appendix; 

- some background references need to be acknowledged (see reviewers' suggestions);
Decision: Accept-Findings