BanLemma: A Word Formation Dependent Rule and Dictionary Based Bangla Lemmatizer
Authors: Sadia Afrin
Md. Shahad Mahmud Chowdhury
Md. Ekramul Islam
Faisal Ahamed Khan
Labib Imam Chowdhury
Md. Motahar Mahtab
Nazifa Nuha Chowdhury
Massud Forkan
Neelima Kundu
Hakim Arif
Mohammad Mamun Or Rashid
Mohammad Ruhul Amin
Nabeel Mohammed
Venue: EMNLP 2023 Findings (2023)
Keywords: Computational Linguistics
Morphology
Bangla NLP
Lemmatization
TL;DR: This study propose some linguistical rules for lemmatization and utilize a dictionary along with the rules to design a lemmatizer specifically for Bangla.

Abstract: Lemmatization holds significance in both natural language processing (NLP) and linguistics, as it effectively decreases data density and aids in comprehending contextual meaning. However, due to the highly inflected nature and morphological richness, lemmatization in Bangla text poses a complex challenge. In this study, we propose linguistic rules for lemmatization and utilize a dictionary along with the rules to design a lemmatizer specifically for Bangla. Our system aims to lemmatize words based on their parts of speech class within a given sentence. Unlike previous rule-based approaches, we analyzed the suffix marker occurrence according to the morpho-syntactic values and then utilized sequences of suffix markers instead of entire suffixes. To develop our rules, we analyze a large corpus of Bangla text from various domains, sources, and time periods to observe the word formation of inflected words. The lemmatizer achieves an accuracy of 96.36% when tested against a manually annotated test dataset by trained linguists and demonstrates competitive performance on three previously published Bangla lemmatization datasets. We are making the code and datasets publicly available at https://github.com/eblict-gigatech/BanLemma in order to contribute to the further advancement of Bangla NLP.

BanLemma: A Word Formation Dependent Rule and Dictionary Based
Bangla Lemmatizer
Sadia Afrin', Md. Shahad Mahmud Chowdhury', Md. Ekramul Islam', Faisal Ahamed Khan', Labib Imam Chowdhury', MD. Motahar Mahtab', Nazifa Nuha Chowdhury', Massud Forkan', Neelima Kundu', Hakim Arif, Mohammad Mamun Or Rashid', Mohammad Ruhul Amin', Nabeel Mohammed' GGig Tech Limited, Dhaka, Bangladesh, UUniversity of Dhaka, Bangladesh BBangladesh Computer Council, Dhaka, Bangladesh, "Fordham University, New York, USA, North South University, Dhaka, Bangladesh
Abstract
(Carpuat, 2013), etc. Lemmatization is a partic- ularly challenging task in highly inflectional lan- guages such as Bangla (Bhattacharya et al., 2005), due to the large number of inflectional and deriva- tional suffixes that can be added to words. Gener- ally, lemmatization reduces the inflectional form of a word to its dictionary form.
Lemmatization holds significance in both nat- ural language processing (NLP) and linguis- ties, as effectively decreases data density and aids in comprehending contextual mean- ing. However, due to the highly inflected nature and morphological richness, lemmati- zation in Bangla text poses a complex chal- lenge. In this study, we propose linguistic rules for lemmatization and utilize dictio- nary along with the rules to design a lemma- tizer specifically for Bangla. Our system aims lemmatize words based their parts of speech class within given sentence. Unlike previous rule-hased approaches, we analyzed the suffix marker occurrence according the morpho-syntactic values and then utilized se- quences of suffix markers instead of entire suf- fixes. To develop rules, we analyze large corpus of Bangla text from various do- mains, sources, and time periods observe the word formation of inflected words. The lemmatizer achieves an accuracy 96.36% when tested against manually annotated test dataset by trained linguists and demon- strates competitive performance on three previously published Bangla lemmatization datasets. We are making the code and datasets publicly available at https://gi thgb. com/ eblict- grigatech/BBnneen. contribute to the further advancement of Bangla NLP.
Lemmatization in Bangla has several challenges due to various linguistic factors. Firstly, the lan- guage exhibits a wide range of morphological di- versity, making it difficult for system to cover all its aspects (Islam al., 2022). Secondly, Bangla has approximately 50000 roots, each capa- ble of generating a large number of inflected words based on several factors such as tense, gender, and number (Chakrabarty et al., 2016). Thirdly, Bangla words can have multiple meanings, known as polysemy, depending on their Part-of-Speech (PoS), surrounding words, context, and other fac- tors (Chakrabarty and Garain, 2016). Additionally, the development of Temmatization systems for this complex morphological language is hindered by the lack of available resources.
Prior rescarch attempts have employed different methodologies (e.g., learningbbased, rule-based, and hybrid approaches) (Pal al., 2015; Das al., 2017; Islam al., 2022; Chakrabarty and Garain, 2016). Despite the fact that some studies have shown satisfactory performance in specific scenar- ios, there remains pressing need for a robust lem- matizer tailored to the Bangla language.
in order to
1 Introduction
Lemmatization is a crucial task in Natural Lan- guage Processing (NLP), where the goal is to obtain the base form of a word, known as the lemma. It has widespread applications in sev- eral NLP tasks, such as information retrieval (Bal- akrishman and Eloyd-Yemoh,, 2014), text classifi- cation (Toman et al., 2006), machine translation
In this study, we have taken a rule and dictionary-based approach to tackle the lemmati- zation problem in Bangla. Unlike other rule-based studies for lemmatization in Bangla, we have de- rived the stripping methods based on the suffix marker sequences considering the PoS class of word. A suffix काानगााा (dergulote) from word सिाडाा (jifudergulote)
combination of
*Corresponding author wwwaallseee@@ggggecccc....o
(कá)) (der), काा (gulo), and c (tc) markers. We strip the last to the first marker sequentially to ob-
'The repository contains the codes, analysis datasct, list of markers, test datascts, and a sample dictionary.

---

kigyabriced orxa (urefeerr)
Cross Dataset Performance
100
togger
Noun
Marvers
95.75%
93.58%
94.34%
Simp timphosts
(manudh mmnnchhuuakee)
91.14%
Corresponding Dataset's s Lemmatizer
Sapeectie (nanuthgutskn)
Stip Case
Marter CP (No)
"ETESST (manushgunn
85
Lemmatizer
81.95%
80
Noun lemmotzzaiion
BenLem
BaNeL
Chakrabarty
BonLemma
al.
(a) Illustration of BanLemma lemmatizer
(b) Cross dataset evaluation
Figure I: Figure a shows the illustration of our proposed Banl eemaa lemmatizer. For simplicity, demonstrate Noun lemmatization only. For noun, emphasis, case, and plural markers are stripped in succession as specified in Section 3.1. Figure la shows its effectiveness on cross-dataset settings. On Chakrabarty and Garain (2016), we outperform their lemmatizer by significant margin and achieve competitive results on the whole dataset of Islam et al. (2022), On Chakrabarty al. (2017), we outperform their model on the corrected PoS and Lemma version of their evaluation set, elaborated in Section 5.3.
tain the lemma Pae (Tifu; child). To derive the marker sequences, we analyzed the word forma- tions using a large Bangla text corpus. By em- bracing the marker sequence striping approach, we are able to effectively address wide range of suf- fixes in the Bangla language, where other studies exhibit substantial limitations. Figure shows an illustration of our lemmatization process and its ef- fectiveness in cross-dataset settings where our lem- matizer achieves higher accuracy than other lem- matizers when tested on their respective datasets (Chakrabarty and Garain, 2016; Islam et al., 2022; Chakrabarty et al., 2017). The key contributions of this study are follows:
22675 words through a systematic approach to manually analyze the inflected words.
To assess the efficacy of BanLemma, we have employed both intra-datasct and cross-dataset evaluation. This evaluation framework en- ables us to measure the robustness of our pro- posed system across multiple datasets.
Utilizing human annotated PoS tag, we have achieved 96.36% accuracy on intra-dataset testing. Moreover, in cross-dataset test- ing. BanLemma surpasses recently published methodologies, exhibiting substantial perfor- mance improvements ranging from 1% 11% (see Figure 1b), which implies our pro- posed BanLemma' robustness.
We introduce BanLemma, a lemmatization system specifically designed for Bangla. By leveraging precisely crafted linguistical framework, our system demonstrates superior performance compared to existing statc-of- the-art Bangla lemmatization methods.
Related Work
Preliminary works on lemmatization mainly con- sisted of rule-based and statistical approaches. Pal et al. (2015) created Bangla lemmatizer for nouns where they removed non-inflected nouns using the Bangla Academy non-inflected word list (Choud- hury, 2008) and removed the suffixes via the longest match suffix stripping algorithm. Das et al. (2017) created a lemmatizer for Bangla verbs ac- cording to tense and person using Paninian gram- mar described in Ashtadhyayi".. Kowsher et al.
We present set of linguistical rules interpret- ing the process by which inflected words in the Bangla language are derived from their re- spective base words lemmas.
The linguistic rules are derived from rigorous analysis conducted an extensive Bangla text corpus of 90.68 90.65M unique sentences. encompasses vast collection of 0.5B words, where 17. words are distinct. We sampled
hhttps:/asshaadlaay..com

---

Sentence निमिि कड [ि
Word
PoS
Lemma
Raw Corpus
PoS Cluster Corpus
(niomito kor
Parts of Speech (PoS)
noun
verb
32.47%
35.67%
50.58%
dao;
Pay your
(kor; taxes)
Noun
(kor;
18.20%
adjective
46.42%
taxes
regu-
tax)
26.21%
larly.)
pronoun
4,37%
15.35%
adverb
4,88%
शा কলাা क कक
(fa bolec"i ja
कर
क
conjunclion
1.15%
koro; Do as say.)
(koro; do)
Verb
(kora; to do)
posiposition 2.68%
8.65%
interjection
0.08%
Table I: Meaning difference of word based on its PoS class. When used as noun, the word "कæ" means "tax", while used as a verb, means "to do".
Figure 2: Distribution of all words in the raw text cor- pus and the analysis dataset by their PoS classes, cluding punctuation and symbols, Both corpora have a similar distribution with an abundance of nouns, verbs, and adjectives and small number of interjections.
(2019) used two novel techniques jointly: Dic- tionary Based Search by Removing Affix (DB- SRA) and Trie (Cormen et al., 2009) to lemmatize Bangla words. Chakrabarty and Garain (2016) pro- posed novel Bangla lemmatization algorithm us- ing word-specific contextual information like part of speech and word sense.
ing the word class or semantic meaning (Lieber, 2021). They do not involve adding prefixes and altering the word's meaning. According to Karwa- towski and Pietron (2022), for lemmatization it is crucial to determine the word's intended PoS accu- rately and its meaning within sentence, consid- cring the broader context. In this study, we have adopted a word formation-dependent rule-based approach, considering the following factors: i) The lemmatizer will operate on the inflected forms only and leave the derivational forms as they are. ii) The rules depend on the words' PoS class to use the contextual information.
Contextual lemmatizers lemmatize word based on the surrounding context using deep neu- ral networks. Chakrabarty al. (2017) employed a two-stage bidirectional gated recurrent neural net- work to predict lemmas without using additional features. Release of the Universal Dependencies (UD) dataset (de Marneffe et al., 20vee et al., 2017) and Sigmorphon 2019 shared task formed the basis of encoder-decoder architectures solve the lemmatization task as a suring-transductionn task (Qi et al., 2018; Kanerva et al., 2018). For the Bangla language, Saunack et al. (2021) employed a similar two-step attention network that took mor- phological tags and inflected words as input. Is- lam et al. (2022) used PoS tags of each word as additional features the encoder-decoder network achieving 95.75% accuracy on validation dataset.
3.1 Development of Lemmatization Rules
To analyze the behavior of inflected words, we uti- lized raw text corpus of 90.65 million sentences, totaling about 0.58 billion words, where approxi- mately 6. million words are unique. The corpus was crawled from l 12 sources, covering ten differ- ent domains across various time periods. Figure 3 and Figure 4 in Appendix A.1 provide visual representations of the dataset's distribution across domains and time respectively.
Earlier rule-based approaches did not consider the composition of suffixes and removed them based on the highest length or trie-like data struc- tures. In contrast, provide specific rules on how sequence of markers forms suffix based on word's PoS tag and show its efficacy on cross- dataset setup.
To obtain the PoS tags for each word in the dataset, we employed the automatic PoS tagger from the BNLP toolkit (Sarker, 2021). We pro- jected each narrow PoS class to its corresponding basic PoS class: noun, pronoun, adjective, verb, adverb, conjunction, interjection, and postposition (Islam et al., 2014). For example, NC (common noun), NP (proper noun), and NV (verbal noun) were mapped the class "noun". This allowed to categorize the words based on their PoS classes.
3 Methodology
In Bangla, the meaning of word is greatly influ- enced by its PoS class within given context of sentence (see Table 1). Inflections are morphemes that convey grammatical features without chang-
After categorizing the words, we selected 19591 words as the analysis dataset which was used to

---

analyze the inflection patterns of Bangla words. The analysis dataset preparation procedure clab- orated in Appendix A.2. Word distribution of the raw text corpus and analysis dataset per PoS class is depicted in Figure 2. As the majority of the words in the analysis dataset were in collo- quial form, there was insufficient data available to study words in classical forms, such करकननकश (tahadigar; their), सिाालिियस (gilac"ilen; went), etc. To address this limitation, we collected classical texts from specifically selected sources'. Utiliz- ing 500 sentences comprising 5155 total words, where 3084 words were unique, we manually cre- ated clusters for the corresponding PoS classes us- ing these classical texts. We did not employ the automatic PoS tagger here as it was trained only on colloquial text (Bali et al., 2010). Adding clas- sical texts allowed to include a wider range of words, with a total of 22675 words.
Case markers in the Bangla suffix system de- termine the noun's role in the sentence, indicat- ing subject, object, possessor, or locative position. From seven Bangla cases, four case markers are used as noun suffixes e.g., nominative, objective, genitive, and locative (Mahmud et al., 2014). De- terminer markers in Bangla noun suffixes provide specificity and indicate singularity, while plural markers indicate multiple entities or instances of noun phrase. Some plural markers are specifi- cally used with animate nouns, such as ll (gon), शम (brindo), गहगी (mondoli), शन (kul), etc. while others are used with inanimate objects, like शाকन (abli), @D (gucc"o), शाम (gram), && (co"), etc. (Is- lam and Sarkar, 2017). Though these suffixes are found in traditional grammar and literature, their frequency of usage is quite low. Emphasis markers र (i) and s (o) are employed emphasize nouns, Table 9 in Appendix A.3 lists the markers used in nouns.
The morphological synthesis of different PoS is highly effective in determining whether inflections are applied and helps identify the lemma. The in- vestigations of the analysis dataset revealed inter- esting sequential patterns of inflected words from different PoS classes. Nouns and pronouns were found to have four inflectional suffixes, including case markers (Moravesik, 2008), plural markers, determiners, and emphasis markers. Verb inflec- tions, on the other hand, depend on factors like tense, person, and number. Adjectives, in com- parison, have only two suffixes उन (taro) and कम (tomo) which indicate the comparative and superla- tive degrees respectively (Das et al., 2020). Lastly, only emphatic inflections are found in adverb and postposition word classes. should be noted that while other PoS classes exhibit distinct patterns of inflection, conjunctions, and interjections function without undergoing any inflection.
In Bengali, the word भासनकलााि (manufgu- lokeo) is formed by combining the base word भारर (manul; human) + काना (gulo) + a (ke) + s (o) where lemma is भान (manuf; human) with शान (gulo) plural, O (ke) case, and is (o) emphatic markers. This inflected form expresses the mean- ing of "even the humans" and conveys plurality, objective case, and emphasis in a single word. The order of these suffixes is crucial because altering the sequence, such as मााककरकााा (manufokeg- ulo), would result in a nonsensical word. The key considerations in noun morphology are selecting the appropriate suffixes, figuring out how to ar- range them, and understanding the changes that take place at the border during affixation (Bhat- tacharya et al., 2005).
From analysis, we find that the emphasis marker always takes the end position of a noun and does not occur in the middle of the suffix sequence, where the other markers can be combined in dif- ferent ways. Table 10 in Appendix A.3 lists some examples of how nouns are inflected by taking dif- ferent markers in specific sequence in the marker combinations, represented by Equation 1.
3.1.1 Inflections of Nouns
Nouns in Bangla comprise both NP and NC, which contribute significant portion to the vocabulary of Bengali phrases. Nominal inflections are ob- served at four levels of nouns, including inani- mate, animate, human, and clite (Faridee and Ty- ers, 2009). These inflections are added to nouns to signify grammatical roles and incorporate morpho- logical features.
W -+ LPL+ + CM) II (CM + PM)
+DM + CM + EM (1)
Here, W, L, PM, CM, DM, and EM denote the original word, the corresponding lemma, plural marker, case marker, determiner marker, and em- phasis marker, respectively. We would use these
UUsed the following sources to extract classical text: httpee///bnnnnnraaaaaa annt..riinnn....aaaaaaamaaaaaa.l org,
attps://kobita. banglakosh.com com, https://rabinde caccanndra-rcchane rachanabsli. mltr.orr .org

---

notations in the following equations also. Accord- ing to the equation, an inflected form of word can consist of lemma and up to four possible suf- fixes. The first suffix can be either a plural marker or a case marker, and they can alternate positions. However, it is not possible for one type of marker to occur twice consecutively. It is also possible to omit any of these suffixes from the sequence.
Word
Suffix
Root
Lemma
NIT (jucc"s)
10e (cc"i)
49 (ja)
40es0 (sag
RI2e (jabo)
(bo)
4 (m)
*0 (jag"
ाeलाीिका (gFec"ilam)
अरशिना (och)
M (gi)
काडेा (aaga)
Table 2: The two-pass approach to lemmatize verbs in Bengali. Firstly, a suffix such as शि (c"ilam; was) is removed from a word, for instance लभारयााार (k"el- chilam; was playing), to retrieve its root form (яसा (k"at) to play). Then, the root matched to lemma in root- lemma mapping to obtain the lemma form, such as (य (k"ela; play).
3.1.2 Inflections of Verbs
Bangla extensively utilizes verbs, which are action words comprising a verb root and an inflectional ending. The inflcctional ending varies based on the tense (present, past, future), person (first, second, third), and honor (intimate, familiar, formal) (Mah- mud et al., 2014). Traditional Bangla grammar di- vides the tense categories into ten different forms. Verbs in Bangla can be written in classical/itterrry form or colloquial form. Table 11 of Appendix A.4 showcases the different forms of verbal inflec- tion Bangla.
3.1.3 Inflections of Pronouns
Bangla pronouns represent specific nouns and ex- hibit similar inflectional patterns to noun classes. The language has nine types of pronouns, catego- rized into first, second, and third person based on personal distinctions (Dash, 2000). Appendix A.5 lists the singular, plural, and possessive forms of Bangla personal pronouns, offering comprehen- sive understanding of their usage in the language.
Many Bangla personal pronouns have inherent suffixes that are integral to the words, and stripping these suffixes can result in meaningless strings. For example, pronouns like शाता (amar; my), लिाा (tomar; your), कााालक (amader; our), сिसाो (tomader; yours) contain case markers * (r>) and ! (der) as inherent parts, which can fur- ther be inflected with other markers like the plu- ral marker, determiner, and emphasis marker. Ad- ditionally, other pronouns are inflected with four nominal suffixes, including the plural marker, case marker, determiner, and emphasis markers. For in- stance, the pronoun लिााललनाटरर (tomaderkej) is inflected with the case marker + (ke) and the em- phasis marker र (i). However, the marker कमड (der) is considered part of the pronoun itself, and our lemmatizer does not strip it, resulting in the lemma being किमापीर (tomader; yours), even though लमर (der) can function as a case marker. Pronoun lem- mas can be inflected using the marker sequence shown in Equation 2.
In Bangla, verb suffiees do not break down into markers like other parts of speech. Removing the suffixes from verb does not yield the lemma but rather the root form of the verb. For instance, शालिक (jace"i; going), शााली (jabo; will go), and लिंअारिाााा (giec";lam; went) are different forms of the verb शाोश (aao"a; to go), which is the dictionary word (Das et al., 2020; Dash, 2000).
After stripping the suffixes from the inflected verbs शालिर (jace"i; going) and यााना (jabo; will go) we would get the suffixes ToR (अ] ((œ) (ce*i(ja + ce'i)) and ब (शा aT) (bo(ja + bo)) respec- tively. After stripping the suffixes, get the root भा (ta; go). This lemmatizer will map the root सा (ja; go) the lemma शाशशा aj to go). How- ever, in Bangla verbs, some exceptions are found such as लिानररशााा (gieec"iaam; went), पा (efcc"i), etc. After stripping the verbal inflections we get the suffixes लपशिना (f+t लासिना)) (eeeiiammgg + Pec"ilam)) and पामरि (ब7 + a.ee (efec"i (ef + ec"i )) and the root fिt (gi) and ब (ef) which does not match with the actual verb roots श (ja) and पान (af). In these cases, the lemmatizer will directly map the verb to the lemma using a root-form to verb-lemma mapping which is shown in Table 2. Briefly, a two-step process is followed accu- rately lemmatize verbs. Firstly, the suffixes are re- moved from the verb extract its root form. Then, a root-to-lemma mapping is applicd to determine the final lemma form of the verb.
W = I. PM DM + CM + F.M (2)
3.1.4 Inflections of Adjectives, Adverbs and Postpositions
Bangla adjectives serve as modifiers for nouns pressing their features and can also modify ad- verbs. Suffix markers associated with adjectives indicate comparative and superlative degrees (Das et al., 2020). There are only two degree mark- ers, के (taro) as comparative and कर (tamo) as

---

superlative, which inflect adjectives to indicate degree. The lemmatizer strips the degree marker from an adjective and results the corresponding positive adjective. For example: अरडन (brihottor; largest) is lemmatized as g (brihot; large) and Tho (K"uorooomo; smallest) is lemmatized as T (k"udro; small). Adjectives can also take the form of numerics when quantifying nouns. For exam- ple, पلकß (ekti; a) is an adjective inflected with nominal suffix. In such cases, the lemmatizer will not strip the suffix, resulting in the lemma being the same as the inflected form. Moreover, because of syntactic structure, nouns can function as adjec- tives e.g सााा पिंन भालिााा (ager diner sritigulo; The memories of the previous day). Here, शालन (ager; previous) will be unchanged as the lemma for being an adjective in this sentence. Addition- ally, adjectives can be inflected with emphatic markers. Equation 3, where DgM represents the degree marker, indicates the sequence of markers that can inflect an adjective.
Algorithm 1 Marker stripping method
Require: A word (W), Marker list (M), and Dic-
tionary words (D.:)
Ensure: The marker list sorted according
length in descending order.
function strip marker(W, M, Dw)
I +- len len(W) Lmaz t 0
Mmarr + string()
for all m E M do if W endswith then W len(m)]
if w Dw then return D [w]
else if len(m) Lmar then
Lmax &- len(m)
Mmaz +
end if
end if
end for
if Lmaz 0 then W W(O.. L Imaa]]
W - DgM EM
end if return W end function
Adverbs modify verbs to indicate how an action takes place. Postpositions, on the other hand, serve a functional role establishing syntactic connec- tions between syntactic units (Bagchi, 2007). Post- positions can also undergo inflection by emphasis markers only. Adverbs and postpositions are in- flected according to W = L. + E.M sequence. Our analysis revealed that words belonging to conjunction and interjection PoS classes do not get inflected in the Bangla language.
3.2.1 Development of BanLemma Dictionary
The dictionary used in the lemmatization process includes inflected words and their corresponding lemmas. For instance, (करललाकाा (onsidarke; to the partner), करभोगा (onfidar; partner)) rep- resents the mapping of an inflected word to its lemma. However, for base words, the key and value in the mapping are the same, as in (अकо) (kæton; flag), (का (kætoo; flag)). The dictio- nary is organized into 6 PoS clusters (c.g., nouns, pronouns, verbs, adjectives, adverbs, and postpo- sitions) containing total of around 71.5k word- lemma pairs. To prepare the dictionary, we used sources including Accessible (2023); Chowdhury (2012). The dictionary format and organization are shown Figure Appendix A.7.
3.2 BanLemma
BanLeemma consists of two main components: PoS-dependent nules and a dictionary. When given input sentence, BanLemma employs auto- matic PoS tagger, suffix list, and a dictionary. The PoS tagger assigns tags to each word in the sen- tence, resulting in list of word-PoS tag pairs. Sub- sequently, BanLemma iterates over each element of the list and applies the relevant lemmatization rule based on the PoS tag. In the case of noun, BanLemma utilizes a method based on Equation 1 to determine the lemma. In contrast, the method utilizes the dictionary and applies sequential suf- fix stripping determine the lemma described in Algorithm We discuss more detailed and implementation-oriienees pscudo codes in the Ap- pendix A.6.
4 Experimental Setup
We evaluate BanLemma's performance using dif- ferent PoS taggers: human-annotated tags, BNLP toolkit (Sarker, 2021), and ISI$ using the Stan- ford Postagger implementation. Additionally,
www. isical. www.diical.ac.nnnuttpllllll wit.n-utpalreesouceee.ppp 'plp. 'nlp.stanford stanford. edu/ http///ftaare/tgge....u

---

conduct cross-dataset evaluation and compare our methodology with existing Bangla lemmatizers, in- cluding BenLem (Chakrabarty and Garain, 2016), BaNeL (Islam et al., 2022), and Chakrabarty et al. (2017).
Accuracy (%)
PoS
All
CSCL
NOAD
NOD
Noun
95.20
94.89
94.60
90.79
Pronoun
94.28
93.59
94.12
87.50
Verb
95.12
96.58
84.11
78.26
Adverb
96.88
96.15
96.67
98.28
Adjective
96.93
98.11
98.40
97.47
4.1 Test Dataset Preparation
Postposition
100.00
100.00
We created test dataset using the text corpus described Section 3.1. Instead of random se- lection, we employed a systematic approach to choose 1049 sentences while maintaining the same domain distribution. The detailed procedure for preparing the test dataset is discussed in Appendix A.8. This dataset had 26% words overlapping with the analysis dataset, enabling reliable eval- uation of our proposed rules. To ensure accuracy, we manually annotated the PoS tags and lemmas of the test dataset. We also prepare separate test dataset containing only classical texts that contain 70 sentences totaling 607 words.
Others
100.00
100,00
100.00
100.00
Overall
96.36
96.48
96.41
96.32
Table 3: PoS wise and overall performance of the lem- matizer using the human-annotated test dataset. All, CSCL, NOAD,, and NOD represents the dataset of all text, classical text, No Overlap with Analysis Dataset and No Overlap with Dictionary respectively. Others indicates all other PoS tags except the aforementioned PoS tags. For the "others" class, the word itself is con- sidered its lemma and there was non-overlapping Postposition for NOAD and NOD datasets.
cussed in Appendix A.9.
5 Results & Analysis
5.2
Dependency on Automatic PoS Tagger
5.1 Performance of BanLemma
Human annotated
BNLP
ISI
Table 3 summarizes the lemmatizer's performance for PoS categories. At first, we evaluate it using the whole test dataset and report the result in All column, where we achieve 96.36% overall accu- racy. To measure the performance on the classical texts only, separate the classical sentences and report the performance the CSCL column that demonstrates 96.48% overall accuracy. After that, we tried to evaluate the performance of words we did not include during the manual analysis of in- flected words, i.e., non-overlapping with the anal- ysis dataset. Column NOAD shows we achieved an accuracy of 96.41% in this attempt. Finally, we measured the performance of the words where not included
Metric
PoS
PoS
PoS
tagger
tagger
Accuracy (%)
96.67
89.32
84.77
Table 4: Lemmatizer's performance based on how PoS tags were obtained.
Table summarizes the impact of using an au- tomatic PoS tagger to get the tags of each word. It indicates that the lemmatizer tends to show signifi- cantly reduced performance when used with an au- tomatic PoS tagger. It also highlights that the rules are well capable of lemmatizing Bangla text more accurately given the correct PoS. Table 5 provides examples where the lemmatizer fails to generate accurate lemmas due incorrect PoS information. These examples highlight the dependency of the lemmatizer on the accuracy of the automatic PoS tagger. demonstrates that the lemmatizer capa- ble of producing the correct lemma when provided with accurate manually annotated PoS tags.
neither the word nor the lemma in the dictionary. We report the accuracy be 96.32% for this experiment in the NOD column. The table shows that the lemmatizer achieves a perfect accuracy of 100% for postpositions, which can be attributed to the limited number of lem- mas and inflections in this category, most of which are included in the word-lemma mapping dictio- nary. The "others" category also achieves 100% accuracy, as the lemmatization process considers the word itself as the lemma for any category not explicitly targeted for lemmatization. We further evaluated the performance of BanLemma in terms of precision, recall, and Fl score, which dis-
5.3 Cross Dataset Evaluation
Table presents the results of the cross-dataset evaluation. Our lemmatizer outperforms BenLem, achieving an 11.63% improvement in accuracy on their provided test datasct. BaNeL did not provide any separate test dataset. So, we evalu-

---

Automatically Predicted
Lemma with
Manually Annotated
Lemma with
Target lemma
Word
predicted
annotated
PoS tag
PoS tag
PoS tag
PoS tag
भरोे (fobaj)
भररद (Sobaj)
adjective
क (foba)
pronoun
भपर (Sobai)
भानपाी (b"albafi)
डारााा (b'albafa)
adjective
काকाা (b"albaji)
verb
कान (b"albafa)
शागा (hasan)
शगम (hasan)
verb
शशभ (hafa)
noun
शाभा (hasan)
Table 5: The table showcases instances where our lemmatizer produces incorrect lemmas when an automatic PoS tagger provides inaccurate tags but accurately lemmatizes word if the PoS tag is correct.
ated the lemmatizer on the entire BaNeL. dataset. Though they reported the performance on a test split, our lemmatizer demonstrates competitive performance achieving 94.80% accuracy on the whole dataset, which is only 0.95% less than their reported accuracy. On the other hand, our system exhibits lower performance on the test dataset pro- by Chakrabarty et al. (2017), achieving an ac- curacy of 80.0%%, which is 11.06% lower than the reported performance. Several factors contribute to this performance gap, including the reliance on automatic PoS tagger, which introduced inher- ent errors. Further investigation reveals significant inconsistency between the dataset used in their study and our considerations during the develop- ment of the lemmatizer. These inconsistencies are discussed in detail Appendix A.10.
C-PoS
Dataset Acc.
A-PoS
C-Lem.
Chakrabarty
79.97
87.09
94.34
et al. BaNeL 96.36
98.99
Table 7: Performance of our lemmatizer on the sam- pled cross dataset measured accuracy. The second column, Acc. indicates accuracy the unmodified datasets. C-PoS and A-PoS indicates manually and au- tomatically annotated PoS tags. C-Lem. indicates man- ually corrected lemmas.
performed three evaluation steps. Firstly, we as- sessed the performance of our lemmatizer modified data within the portion where manual ef- forts were applied. The accuracy column (Acc.) of the table presents the accuracy on this attempt to be 79.97%. Secondly, we examined the performance using an automatic PoS tagger while correcting the lemmas. The automatic PoS tags (4-PoS) and orig- inal lemma (O-Lem) column report this accuracy to be 87.09%. F'iaally, measured the overall per- formance using manually annotated PoS tags and corrected lemmas. The correct PoS (C-PoS) and corrected lemma (C-Lem) column of the table il- lustrate the outcomes of the final experiment be 94.3 94.34%.
Test
Ch
Study
Ace
dataset
Acc
BenLem
81.95
BenLem
Ours
93.58
BaNeL
95.75
BaNeL
Ours
94.80
Chakrabarty
Chakrabarty
91.14
et al.
et al.
Ours
80.08
Table 6: Lemmatization results on cross-dataset eval- uation. * We report the performance the entire dataset of BaNeL while they reported the metric on test split from the entire dataset (Ace= Accuracy and Ch Acc=Character Accuracy).
Since the BaNeL dataset already provides manu- ally annotated PoS tags, we focused solely eval- uating the performance of our lemmatizer the corrected lemmas. The fourth column of the table presents the performance in this scenario. In both cases, our lemmatizer demonstrated improved per- formance compared to the initial evaluations.
To evaluate the performance of our proposed system on the dataset provided by Chakrabarty et al. (2017), we manually annotated and corrected PoS tags and lemmas. We focused on 52 sentences comprising total of 695 words. Additionally, we reviewed and corrected 2000 lemmas from the BaNeL dataset. The results of evaluation summarized in Table For the Chakrabarty et al. (2017) dataset, we
Conclusion and Future Works
This study introduces BanLemma, Bangla lemmatization system aimed enriching Bangla language resources. BanLemma is composed of linguistically derived rules, obtained through rig- orous analysis of large Bangla text corpus. To

---

overcome the challenges associated with limited resources in Bangla lemmatization, we also pro- vide a comprehensive collection of morphological markers and rules. To demonstrate the effective- ness of BanLemma, we have conducted evalua- tions using human-annotated test dataset, anno- tated by trained linguists and some recently pub- lished Bangla datasets. Our proposed BanLemma achieved accuracy of 96.36% on human- annotated test set. Furthermore, in cross-dataset evaluation, BanLemma exhibited significant per- formance improvements ranging from 1% to 11%.
action of hanging or the state of something be- ing hung.
Automatic PoS dependency: The lemma- tizer heavily relies PoS information, which introduces errors when an automatic PoS tag- ger used the workflow.
Sentence
Word
Lemma
HiGß उरमाका his (doriti fuulle dag;
(fulile)
( (fulano)
Hang the rope) n उলाक
The results our study shed light on the formation of inflected words, offering a solution to address the limitations of previous lemmatization methods. These findings contribute to the advancement of research in this field and pave the way for further investigations in the domain of Bangla lemmatiza- tion.
कना (f'lc"))
उ
(doriti fuuc"e;
( (uula)
The rope is hanging)
Table 8: Difference of lemma of the same verb उ (f'ula) based on the context of the sentences. lemma is शागालाी (f"ulano) when it expresses action of hanging something. and the lemma is उगम (f'ula) when expresses that something is hanging.
Limitations
During our analysis, we found some limitations of BanLemma. We discuss these in the following points:
References
Accessible. 2023. Accessible dictionary. https:// accessiblediction gov. bd/. (Accessed on 04/24/2023).
Out of dictionary words: we identified a no- table pattern in the lemmatizer's 's behavior re- garding words that are not present in the dic- tionary but already are lemmas and end with suffix substring. In this case, the lemmatizer erroneously strips the suffix from the words. For instance, the word शगरम (nunnojomo; minimum) itself is a lemma, yet the lemma- tizer strips the ending substring उम (tomo), re- sulting in the lemma बाा (nunno), which is in- correct. We also noticed that this limitation is particularly prominent with proper nouns. It also emphasizes the significance the dictio- nary's richness in the lemmatization process. Words that are not present in the dictionary but end with suffix substrings will be inaccu- rately lemmatized.
Anwesa Bagchi. 2007. Postpositions in bangla. Lan- guage in India, 7(11).
Vimala Balakrishnan and Ethel Lloyd-Yemoh. 2014. Stemming and lemmatization: A comparison of re- trieval performances.
Kalika Bali, Choudhury Monojit, and Priyanka Biswas. 2010. Indian language part-of-speech tagset: Bengali linguistic data consortium. https: //catalog. 1dc. upemn edu/LDC2010T16. (Ac- cessed on 06/ 06/1532023).
Samit Bhattacharya, Monojit Choudhury, Sudeshna Sarkar, and Anupam Basu. 2005. Inflectional mor- phology synthesis for bengali noun, pronoun and verb systems. Proc. of the National Conference on Computer Processing of Bangla (NCCPB 05), pages 34 43.
Marine Carpuat. 2013. Nre: A machine translation ap- proach cross-lingual word sense disambiguation (semeval-2013 task 10). In Second Joint Conference Lexical and Computational Semantics ( SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEvall 2013),
Ambiguous semantic meaning: We ob- served that the lemmatizer struggles to com- prehend the semantic meaning of certain words, resulting in incorrect lemmatization. For example, Table illustrates case where the lemma varies depending on the context al- though having the same PoS class. The lem- mas differ based on whether they express the
pages 188 192.
Abhisek Chakrabarty, Akshay Chaturvedi, and Utpal Garain. 2016. neural lemmatizer for bengali. Proceedings of the Tenth International Conference

---

on Language Resources and Evaluation (LREC'16), pages 2558 2561.
Rafiqul Islam and Pabitra Sarkar. 2017.
Bangla
Academy Pramita Bangla Bhashar Byakaran. Bangla Academy.
Abhisck Chakrabarty and Utpal Garain. 2016. Benlem (a bengali lemmatizer) and its role in wsd. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 15(3).
Rafiqul Islam, Pabitra Sarkar, and Mahbubul Haque. 2014. Bangla Academy Pramita Bangla Bhaboharik Byakaran. Bangla Academy.
Abhisek Chakrabarty, Onkar Arun Pandit, and Utpal Garain. 2017. Context sensitive lemmatization us- ing twO successive bidirectional gated recurrent net- works. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 1481 1491, Vancouver, Canada, Association for Computational Linguistics.
Jenna Kanerva, Filip Ginter, Niko Mickka, Akseli Leino, and Tapio Salakoski. 2018. Turku neu- parser pipeline: An end-to-end system for the CoNLL 2018 shared task. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependenciex, pages 133-142, Brussels, Belgium. Association for Com- putational Linguistics.
Choudhury. 2008. Bangla academy bangla banan ab- hidhan. Pmo.gov.bd.
Michal Karwatowski and Marcin Pietron. 2022. Con- text based lemmatizer for polish language.
Jamil Chowdhury. 2012. Bangla akademy ba- nan abhidhan. https://nltz ittewbggv.c. in/ download/Bange word-list.... doc. (Accessed on 04/24/2023).
Md Kowsher, Imran Hossen, Skshohorab Ahmed, and Seth Darren. 2019. Bengali information retrieval system (birs). Linguisties, 1:1-12.
Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, and Clifford Stein. 2009, Introduction to Al- gorithms, Third Edition, 3rd edition. The MIT Press.
Rochelle Lieber. 2021. Introducing morphology. Cam- bridge University Press.
Md Redowan Mahmud, Mahbuba A Afrin, Md Abdur Razzaque, Ellis Miller, and Joel Iwashige. 2014. A rule based bengali stemmer. In 2014 International Conference on Advances in Computing, Communica- tions and Informatics (ICACCI), pages 2750-2756. IEEE.
Arijit Das, Tapas Halder, and Diganta Saha. 2017. Automatic extraction of bengali root verbs using paninian grammar. In 2017 2nd IEEE International Conference Recent Trends in Electronics, In- formation & Communication Technology (RTEICT), pages 953 956. IEEE.
Edith A Moravesik. 2008 The distribution of case.
Souvick Das. Rajat Pandit, and Sudip Kumar Naskar. 2020. A rule based lightweight bengali stemmer. In Proceedings of the 17th International Conference on Natural Language Processing (ICON). pages 400- 408.
Joakim Nivre, Daniel Zeman, Filip Ginter, and Francis Tyers. 2017. Universal Dependencies. In Proceed- ings of the 15th Conference of the European Chap- ter of the Association for Computational Linguistics: Tutorial Abstracts, Valencia, Spain. Association for Computational Linguistics.
Niladri Sekhar Dash. 2000. Bangla pronouns-a pus based study. Literary and linguistic computing. 15(4):433 444.
Alok Ranjan Pal, Niladri Sekhar Dash, and Diganta Saha. 2015. An innovative lemmatization technique for bangla nouns by using longest suffix stripping methodology in decreasing order. In 2015 Interna- tional Conference on Computing and Network Com- munications (CoCoNet), pages 675 678.
Marie-Catherine de Marneffe, Timothy Dozat, Na- talia Silveira, Katri Haverinen, Filip Ginter, Joakim Nivre, and Christopher D. Manning. 2014. Uni- versal Stanford dependencies: A cross-linguistic ty- pology. In Proceedings of the Ninth International Conference Language Resources and Evaluation (LRRFC'14), pages 4585-4592, Reykjavik, Iceland. European Language Resources Association (ELRA).
Peng Qi, Timothy Dozat, Yuhao Zhang, and Christo- pher D. Manning, 2018. Universal Dependency pars- ing from scratch. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 160-170, Brus- sels, Belgium. Association for Computational Lin- guistics.
Abu Zaher Md Faridee and Francis Tyers. 2009. De- velopment of a morphological analyser for bengali. In Proceedings of the First International Workshop on Free/Open-Souu Source Rule-Based Machine Transla- tion.
Sagor Sarker. 2021. Bnlp: Natural language process- ing toolkit for bengali language. arXiv preprint arXiv:2102.00405
Md Ashraful Islam, Md Towhiduzzaman, Md Tauhidul Islam Bhuiyan, Abdullah Al Maruf, and Jesan Ahammed Ovi. 2022. Banel: an encoder- decoder based bangla neural lemmatizer. SN
Kumar Saunack, Kumar Saurav, and Pushpak Bhat- tacharyya. 2021. How low is too low? monolin- gual take on lemmatisation in Indian languages. Proceedings of the 2021 Conference of the North
Applied Sciences, 135)

---

American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 4088 4094, Online. Association for Computa- tional Linguisties.
Word Count in millions (MM)
1000
542.7M
100
10
1.88M
Michal Toman, Roman Tesar, and Karel Jezek. 2006.
0.41M
0.48M
Influence of word normalization on text classifica- tion. Proceedings of InSciT, 4:354-350
0.1
0.25M
0.36M
0.14M
A Appendix
0.001
A.1 Raw Corpus Distribution
1850-1877
1875
1939-1925
1928-1959 1950
1950 1975
1975 2008
2000 present
Figure 3 and Figure 4 provide representations of the raw corpus's distribution across domains and time respectively.
Figure 4: The time distribution of the text corpus is shown in the figure, where the horizontal axis is dis- played on a logarithmic scale. significant portion of the data spans from 2000 to the present, while rela- tively small amount of data exists for the periods before 2000. Furthermore, approximately 7.79% of all words (46173307 words) do not have specific time frame as- sociated with them.
Acedemic book, Administrative seus, 2.8
Bangla news, 263M
Business news, 0.77M
Circuler, 0.57M
Encyclopedia, 170M
Essay, 0.17M
Literary book, 44.7M
Literary book, 44.7M
content, 1.3M
Bangla news, 263M
3.3M
Religious
Reportar news, 8.57M
मककंल (forkarke), and so on, which share the common initial substring भहकाश (farkar; govern-
Sports news, Kolkata naws, Lifestyle mogazine, 8.00M
Encyclopedia, 174M
Internationa nows, 3.51M
B Local news, 4.3M
ment), were grouped together. For the analysis, we then selected clusters that contain a minimum threshold of words. Initially, we set a minimum threshold of 10, but this resulted in an overabun- dance of nouns, verbs, and adjectives while filter- ing out clusters from pronouns, postpositions, con- junctions, and interjections due to limited words in those groups To address this, we individually determined the minimum threshold for each PoS class. The thresholds were set as follows: 14 for nouns, 7 for adjectives, 6 for verbs, 2 for pronouns and adverbs, and for postpositions, conjunctions, and interjections. These thresholds were deter- mined through combination of tuning and man- ual examination of the selected clusters. To neu- tralize the error of the automatic PoS tagger, we manually curated the words while removing any word if necessary. Finally, we came up with 19591 selected words for the analysis dataset. To study the classical texts rigorously, we additionally use some classical text sources and select some words. Subsequently, we selected total of 22675 words for further analysis.
Social media, 2.2M
commmecce, 0.84M
Manual, 1.3M
Figure 3: Distribution of word count of raw dataset corpus categorized by their respective domain. The amount of word count is denoted by 'M' (millions). The majority of the data originates from the "Bangla News" and "Encyclopedia" domains, while the "[sssa" data represents the smallest portion. However, there are 66757 words, accounting for 0.01% of all words, for which the domain could not be determined.
A.2 Analysis Dataset Preparation
To obtain the PoS tags for each word in the dataset, we used the PoS tagger from the BNLP toolkit, The tagger was trained on the Indian Language Pare-of-Speech Tagset: Bengali LDC2010T16 (Bali al., 2010) dataset, which comprises 30 narrow PoS classes. After projecting the narrow PoS classes, we grouped the words based on their PoS class for further analysis which formed the basis of our investigation into the behavior of in- flected words in Bangla. In order to conduct a de- tailed analysis of the words, employed sys- tematic approach to create a representative dataset. Initially, we clustered the words based on their longest common initial substring within each PoS group. For example, words like भडकक (forkar; government), भहकोर (forkari), भ (forkaro),
A.3 Markers and Noun Formation
A.4 Verb Suffixes
Table presents all suffixes that inflect the a verb root.

---

Type
Markers
Cupa SER
Plural
पानन (aboli), शन (kul), 99 (gon), as (gucc"o), क (gula), आी (guli), लोा (gulo), cम (der), बm (gram), be (col), डा (tal), बम (tro"), सहा (dol), ^ाम (dam), [िश (dig), (िी (digor), सअ (d)), निकक (nikor), निक (nica"), भा (pal), 2 (punjo), शभी (borgo), शम (brindo), अ (brojo), अेл (mandol), अहग (mandoli), शशश (mahol), भा (mala), पя (fut"), (ra), बोहि (raji), अशरि (rafi), Cفf (sreni), भां (fomuho), नर (foho), Casl (era), 6::T5 (occo")
Tugper
Noun Grop
Pronoun
Veb
Adaciive Group
Ponouut
Porpesation
Group
TEWERR TTEEE
Shart SHRVCT (EarTN COpUrSSTT
Fll LEBRE Thrnobeld
ope IEHETO ELPERTITT
VIRTE YEWE.
Case
काड (kar), काेत (kare), 6 (ke), ( (ker), co (te), म (ro), ब (re), (e), coCo (et), CaRt (er), ()), (e)
Determiner
भान (k (aa),,, शान (k"ani), दी (ta), 16 (ti), श (tuku), मशन (tukun),
(6 (te)
Figure 5: The analysis dataset preparation process.
Emphasis
2 (i), (o)
Table 9:
markers in used nouns which con- tains 37 plural markers, 12 case markers, 7 determiners, and emphasis markers, totaling 58 markers.
A.5
Personal Pronouns
Table 12 lists the personal pronouns in singular, plural, and possessive forms.
A.6
Lemmatization Algorithm and Methods
is found in the dictionary, the lemma is immedi- ately returned. However, the remaining prefix is not found in the dictionary, the algorithm contin- ues matching to determine if stripping a shorter suf- fix marker would result in a dictionary entry. This design choice allows for handling cases where mul- tiple markers are present. For example, when strip- ping case markers from the inflected word (िानर (of the boy), the algorithm would first match the marker के (s=R). Stripping this marker would result in the word (क9ी which is not the correct lemma. By continuing matching process, the algorithm would then match the marker D, result- ing in the correct lemma (िा (boy). However, if any shorter marker is not found, it would strip the longest marker at the end.
Algorithm 2 summarizes the overall lemmatiza- tion algorithm.
Algorithm provides a summary of the lemma- tization method for noun words. The stripping pro- cess begins with the last marker, which in the case of nouns is the emphasis marker. Equation 1 il- lustrates that the sequence of the last three mark- ers is fixed. However, if the first marker is plural marker, then the second marker will be case marker, or vice versa. After stripping the last three types of markers, the algorithm determines the next last marker and selects a second appropri- ate sequence for stripping the remaining markers.
The marker stripping method is described in de- tail in Algorithm 1. This algorithm is responsi- ble for identifying and removing markers at the end of word. It begins by checking if word ends with a marker. If match found, it deter- mines whether stop the matching process based on whether the remaining prefix of the word is present in the dictionary. If the remaining prefix
The lemmatization methods for other PoS classes can be achieved by modifying Algorithm 3. The details of these modifications are discussed in Algorithm to Algorithm 7. Algorithm presents the algorithms to lemmatize the words from the corresponding PoS class.

---

Word (noun)
Lemma
Piural
Case
Plural
Determiner
Case
Emphasis
(jongomi)
6ा99 (jangon)
(i)
ालकककक (jikk"akke)
ाकक (jikk"sb ak)
c (ke)
माननरारर (manufkej)
रा (manuf)
R (ke)
x (i)
aidloo (me'etike)
CACc (me'eti)
1G (ti)
ç (ke)
M (gac"tafeg)
Me (gac")
DJ (ta)
(8 (te)
6 (o)
Mioces0iee (jffudertateg)
Pe (fifu)
Cee (der)
bf (ta)
6 (je)
s (o)
A (ma'ederkeo))
(ma)
CR (Fe)
(Re (der)
G (ke)
s (o)
xIC (maledertafeg)
(ma)
CR (?e)
( (der)
51 (ta)
( (te)
< (o)
अररcाst (b"uPera)
भोर (b"aj)
CR (ee)
x (ra)
राराका (balokgulo) <āeलicce (bojgulitei)
राक (balok)
काता (gulo)
< (boi)
eभि (guli)
co (te)
Table 10: Examples of Bangla words which are formed with different sequences of noun suffixes to make mean- ingful words.
Future
Future
Person & Forms
Present (Simple)
Present (Cont.)
Present
Past
Past
Past
Past (Habitual)
Future (Simple)
(Cont.)
(Compl.)
(Compl.)
(Simple)
(Cont.)
(Compl.)
VNF
VNF
1x
<it
1st
Co.
(c"l)
(ec"I)
(lam)
(c'ilam)
(ce'ilam)
(tam)
(bo)
(c)
Person
24
spolt
axifa
#cofbent
R05
Cl.
(Itcc"))
(I'c*I)
(ilam)
(itcc"llam)
(I'ac"Ilam)
(item)
(ibo)
(ite)
MM
In.
Co.
(kor)
(c'l)
(ec"l)
(chll))
(sc'ilo)
(ti)
(c)
2nd
figer
Co.
Person
Fm.
(0)
(c'o)
(oc'o)
(chilc)
(ce'ilo)
(ta)
(bc/ba)
(c)
CL.
(itce"o)
(Fac²o)
(ile)
(itce"tlo)
(Pacc"loo
(itc)
(ibc)
(ite)
4(de))
Co.
(c'en)
(ec"en)
(len)
(chilen)
(ec'ilen)
(ten)
(ben)
(c)
20m
Extifecer
Rcom
208
CL.
(itec"en)
(Pac"en)
(ilen)
(Fachilen)
(i'ac"llem
(iten)
(jben)
(ite)
cet
Co.
In.
(c)
(c'e)
(ec"c)
(lo)
(chilo)
(ec"ilo)
(bc)
(c)
2x10e Rom
Rcoteer
कxtifee
Rcol
RG4
R08
2311
3rd
Cl.
(itcc"cn)
(lac*c)
(ilo)
(itce"llo)
(l'c""ll)
((bc)
(ite)
Person
[ipe
difepnt
cet
Co.
Fm.
(e)
(c'e)
(ec'e)
(lo)
(c'ilo)
(cc'ilo)
(be)
(e)
Epopt
R300e
Eceiter
Ecol
CL.
(itce"c)
(Fac'c)
(ilo)
(Itce"tlo)
(P'c""lo)
(ito)
(ibc)
(ite)
ddfl
<(de
<iloom
Co.
(ec'en)
(len)
(chilen)
(ec'ilen)
(ten)
(ben)
(c)
20mA
Scolkpee
Scom
308
CL.
(itec"en)
(Pacten)
(ilen)
(itec"ilen)
(iyac"ilen)
(iten)
(jben)
(ite)
Table 11: List of all suffixes that inflect the root स (kor) of the verb कशा (kora) depending on the tense, person, and honor. This covers the colloquial form (Co.) and the extended verb forms of the Bangla classical style (CI.). The table also includes suffixes for intimate (In.), Familiar (Fm.), and Formal (Fr.) endings. In addition, the table distinguishes between the Continuative (Cont.) and Completive (Compl.) aspects of the tense.
Pussessive
Possessive
Style
Singular
singulor
plaral
Colloquial
.nf (ami). पागाटर (smuke)
(amarr), अामारर (amake)
पामरर (amra)
(लNा (ummsterr
First
Lassical
4[A [ama)]
श (umi)) अर (jui)
Co:48 (jomur), CeTE (jor).
сेाररा (somra), cотв (jora),
ललानररर! (somafer), сиTOसe (jode)).
Colloquial
Second
(apri)
अासरक (apnar), canipe (jomake)
पालााा (agnara)
पाता0 (apnader)
Classical
(jomi))
6 (she), TaR (tmi), 4 (e). 8 (o), BR (uni)
8. (tar), 48 (er).
3|GR (jader), 4996 (eder),
Colloquial
(parn), (era), een (ora)
Third
5i (or), अनस (unar)
6D9 (odor)
6ा (iahur), 36x (thar),
डाकर (jabara), F6xt (thurs),
अा0|0त (jahader), Cojsr (thader),
Classical
टात (uhar)
SU3T (uhura)
SUाe (uhader)
Table 12: List personal pronouns in singular, plural, and possessive form where suffixes are included with the base form of the word.

---

Algorithm 2 The Lemmatization algorithm
Algorithm 3 Noun lemmatization method
Require: A sentence (2)), PoS Tagger (p_ tagger), Suffix and Marker list (S), and Dictionary (D) Ensure: The PoS tagger (p tagger) returns list tfpll where the first element the word and the second element the PoS tag. Suffix lists are clustered into PoS classes and sorted accord- ing to length in descending order.
Require: noun word (W), Clustered marker list
(S), and Dictionary (D)
Ensure: The word is noun. Suffix lists are clus- tered into markers in a hash map where the key is the marker name and the value is a list of mark- ers.
function noun lemma(W, S, D) D < D[nouns]
procedure lemmatize(T, tagger, S, D)
lemmas + lisl()
if TWE E Dw then return D= end if
words with tags p_ tagger(T) (T)
for all (W,tag) words with tags do
tag- noun then
StripSeq + [em, cm, dm] for all m E StripSeq do + -sripp marker(W, Slm], Dw) if W E D' then return Dw [W] end if end for SecondStripSee, + list() for all E S[pm)] do if W endswith then SecondStripSeq + [pm, cm] break end if end for if len SeconnStriip)eee) then SccondStripSeq <- [em, pm] end if for all m E &SeconSStiipSeg do
- noun lemma(W; S, D)
else tag = pronoun then
- pro le, D)
else if tag verb then
I, <- verb lemb_llemaa(1 (W, S, D)
else if tag adverb then
L + adverb BeeerrbbeemmaWWWee (W, S, D)
else tag adjective then
< adj Jew.llmmmaWW.... S, D)
else lag = posiposition then
+ Letipoosleemmaffe D)
else
+ W
end if lemmas.add(() end for lemmo sent < space_join return lemma_sent sent end procedure
W x- strip_smiiiispppp hatp_mmreerWWW... Sim), Dw))
if W E D.r then
return D [W]
end if end for
A.7 The Dictionary Format
return W end function
total, the dictionary contains 6 PoS clusters such as nouns, pronouns, verbs, adverbs, adjectives, and postpositions and consist of 46, 289, 499, 5, 366, 17, 040, 860, and 1, 353 word-lemma pairs, respec- tively. The dictionary format and organization is shown in Figure 6
made include at least 1% of sentences from each domain. This decision enabled us in- corporate sentences from domains that were under- represented in the dataset, such as circular, which accounted for only 0.09% of the entire corpus. Ad- ditionally, we included sentences for which we could not determine specific domain. Through- out this procedure, we maintained the uniqueness of the selected sentences.
A.8 Test Dataset Preparation
First, we divided the sentences into their respec- tive domains and randomly reshuffled them. Then, selected percentage of sentences from each domain based its contribution to the total per- centage of sentences in the entire raw text corpus. For example, we sampled 452 sentences from the Bangla news domain, which accounted for 48.18% of the entire dataset. During the selection process,
At this stage, we discovered that the test dataset had significant overlap of 38.27% with the words used during the inflected word analysis. To en- sure a more effective evaluation of our proposed

---

Algorithm 4 Pronoun lemmatization method
Algorithm 6 Adverb lemmatization method
Require: A pronoun word (W); Clustered marker list (S), and Dictionary (D)
Require: An adverb word (W), Clustered marker
list (S), and Dictionary (D)
Ensure:
The word is a pronoun. Suffix lists are clustered into markers in a hash map where the key is the marker name and the value a list of markers.
Ensure: The word is an adverb. Suffix lists are clustered into markers in hash map where the key is the marker name and the value is a list of markers.
function pro lemma((W W, S, D)
function ADVERB LEMMA(W, S, D)
Dr t D[pronouns]
D t D[adveró]
if TWE E Dw then return [W]
if TWE E Dw then return D=[W] [W] end if
end if
StripSeq t (em, cm, dm, pm]
StripSeq + [em)
for all m E StripSeg do
for all m E StripSeq do t -tripp marker(W, Slm], Dw)
W + strip_ ciiip_marker(M)) W, S(m], Dw)
W E Drc then return Dw [W]
if W E D' then return D [W]
end if end for
end if end for
return W
return W end function
end function
Algorithm 5 Adjective lemmatization method
Algorithm 7 Postposition lemmatization method
Require: A adjective word (W), Clustered marker list (S), and Dictionary (D)
Require: A postposition word (W), Clustered
marker list (S), and Dictionary (D)
Ensure: The word an adjective. Suffix lists are clustered into markers in a hash map where the key is the marker name and the value is a list of markers.
Ensure: The word is an adverb or postposition. Suffix lists are clustered into markers in a hash map where the key is the marker name and the value is list of markers.
function ADJ LEMMA(W, S, D)
function POSTPOS LEMMA(W, S, D)
D. D[adjectives] if W C Dw then return [w]
D + Dlpostposition) if W D then return D [W]
end
end if
StripSeq + [em, dgm]
StripSeq
+ [em]
for all m E StripSeg do
for all m E Strippeq do
W + haailpmarrrrWWW. W, S[m], Dw)
W + strip marker W, S(m], Dw)
if W c Dw then
if W c D., then
return [W]
return [W]
end if
end if
end for
end for
return W
return W end function
end function
rules, we aimed to reduce this overlap percent- age. Figure illustrates that the raw corpus con- sists of a large number of nouns, verbs, and adjec- tives. Therefore, during the selection test sen- tences, we excluded any sentence that contained any noun, verb, or adjective that was already in-
cluded in the analysis dataset. However, found that the sample sentences were not well formed as the number of verbs and adjectives is limited. So, finally, we attempt reduce the overlap by al- lowing all verbs and adjectives while controlling the overlapping of nouns. As a result, the final

---

{
Split
Precision
Recall
F1
nouns: {
Non-inflected
0.9784
0.9682
0.9733
word_1: 1: lemma_1,
Inflected 0.929
0.9512
0.9399
word_N N: lemma
Table 13:
Performance of the lemmatizer on non-
},
inflected and inflected words.
verbs: {
Word_N+1 Lemma_N+1.
A.10 Dataset Annotation Inconsistencies
From the dataset of BenLem, firstly we found that they labeled verb roots as lemmas, while we con- sider the dictionary form as the lemma. For ex- ample, they annotated the lemma of the word शरर (hobe) as (ha), whereas we lemmatize it शशशा (ho"a). Secondly, they converted colloquial pro- nouns to their classical forms. They labeled the lemma of Øich (tader; their) as the classical form करीाीक (tahader; their), whereas we consider the same colloquial form कालन (tader; their) as the lemma. Lastly, they made spelling changes to cer- tain words, such as transforming उन (b"alo) to डोमी (b"alo), which differs from approach.
word N+M: lemma_Ne N+M
PoS_PP P: {...}
}
Figure 6: The dictionary format utilized in the lemma- tizer implementation. It consists of hash map with PoS class names keys and another hash map as val- ues. The keys of each PoS class hash map are words and values are the corresponding word's lemma.
During our analysis of the BaNeL dataset, we discovered the following inconsistencies. Firstly, certain derivational markers were removed. Sec- ondly, pronoun forms were modified, converting पालर (eder; their) to दिरि (tini; he/she). Thirdly, spelling changes were made to some words, such as lemmatizing लररररलमा (Jebokad"omer) as लिककपकय (Jebokodfom), whereas consider it as लाककशय (Jebokadaam). Additionally, incor- rect lemmas were found in the dataset, where डोलनगगाालाा (b"alomanufer) was provided as the lemma for डोननगापाार (b aloomnufe)).. Furthermore, our lemmatizer has a limitation that produces incor- rect results when the actual word ends with suf- fix marker. For instance, the lemma of the word कककन (jeler) should be लिा (jele; fisherman), but our system incorrectly lemmatizes it as (Sfo (jel; prison).
test dataset had only 16. overlapping words with the analysis dataset, where we found 9.68% nouns overlaps with the analysis dataset. However, this reduced overlapping dataset allows to con- duct more robust evaluation. Finally, to complete the annotation process, we manually assigned PoS tags and lemmas to the words extracted from these sentences. We collaborated with an annotator who assigned the PoS tags and lemmas to each word. To ensure the accuracy and consistency of the an- notations, the assigned tags and lemmas were vali- dated by a validator who was linguistic expert.
A.9 Further Performance Evaluation
We were interested in evaluating how the lem- matizer works on the non-inflected and inflected words. For a non-inflected word, the word itself is the lemma. Except for the proper nouns, Usually, the non-inflected words are found in a dictionary, To conduct this experiment, we first lemmatize the sentences from the test dataset. Then separate the non-inflected and inflected words along with the lemmas. In this setup, there are total of 6906 non-inflected words and 3125 inflected words. We found that the lemmatizer achieves an Fl score of 0.9733 for non-inflected words and 0.9399 for in- flected words. Table 13 summarizes the analysis.
In the dataset of Chakrabarty al. (2017) They made changes to the gender of words, such as transforming शरड (juboţi; young girl) to शकक (jubok; young boy), altered negated forms to posi- tive forms, e.g., changing आरद (adur; not so far) to क (dur; far), and modified pronouns, e.g., लिमा (tomar; your) to शল (tumi; you). They also made derivational changes, such transforming কाলिकाक (banittik; commercial) to रािक (banitfo; trade), शसक (prokriçi; nature) to शश (prokriço; real), and so on. These discrepancies significantly impacted the performance of our lemmatizer.


Reviews:
[{"review_id": "MAH8OrfLQV", "reviewer": "Reviewer_5hHh", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful comment and kind words. We'll respond to each point separately below:\n1. The writing style of Bangala words with their English pronunciation is inconsistent. It is recommended to use a unified style, for example, (Bangala word + it transcription in IPA format + English meaning). This would help researchers of other languages to read the paper easily.\n\t\n   **Response:** Thank you for providing your valuable suggestions. We will ensure that they\n       \tare incorporated into the revised manuscript.\n2. There are some typos: A. Line 114: tas. B. Line 174: lemmatizat. C. Line 866: a letter is written as a square. D. Line 879: don't add comma separators in numbers since it conflicts with the comma of the written sentence itself.\n\n   **Response:** We sincerely apologize for the typographical errors. We will take the necessary steps to revise the paper and ensure that all mistakes are rectified.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors of this paper present a rule-based lemmatization system for one of the low-resource and rich morphology languages. They provide a detailed linguistic description of how they developed the rules depending, mainly, on the POS of each word where the POS is the key for the suffix markers to be stripped from the word and, then, check their developed dictionary. In addition, they analyzed a large corpus of Bangala texts that covers different sources and domains. They compare their work with the three previously published trials for predicting the lemma and they could achieve SOTA results against these systems. They achieve an accuracy of 96.67% and list the limitations of their system that they may work on it, in the future, for enhancing the results of their lemmatizer.", "reasons_to_reject": "No reasons", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a rule-based approach for predicting the lemma for one of the highly inflected languages; the Bangala language. Nevertheless, it is still one of the low-resource languages. For predicting the lemma of each word, they mainly depend on it Part-of-Speech (POS) where each POS has its rules that strip the suffixes in different ways. In addition, they utilize a look-up dictionary to check the presence of the predicted lemma after striping the suffixes. They analyze the suffix marker occurrence according to the morpho-syntactic values and then utilized sequences of suffix markers instead of entire suffixes. For developing their rules, they analyze a large corpus of Bangla text from various domains, sources, and time periods to observe the word formation of inflected words. They achieve SOTA results when tested against a manually annotated test dataset against the three previously published Bangla lemmatization datasets.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The paper is well-written and easy to read, nevertheless, I have some comments:\n\n1- The writing style of Bangala words with their English pronunciation is inconsistent. It is recommended to use a unified style, for example, (Bangala word + it transcription in IPA format + English meaning). This would help researchers of other languages to read the paper easily.\n\n2- There are some typos:\n       A. Line 114: tas.\n       B. Line 174: lemmatizat.\n       C. Line 866: a letter is written as a square.\n       D. Line 879: don't add comma separators in numbers since it conflicts with the comma of the written sentence itself."}}, {"review_id": "5MCsdpwDWh", "reviewer": "Reviewer_N1LB", "summary": "", "questions": "A. The paper mentions a unique approach of analyzing the suffix marker occurrence according to the morpho-syntactic values and then utilizing sequences of suffix markers instead of entire suffixes. How does this method differ from traditional methods, and what specific advantages does it offer for Bangla language processing?\n\nB. The reported accuracy of BanLemma on your own human-annotated test set is 96.36%, which is quite impressive. However, there's a discrepancy in performance when using the test dataset provided by Chakrabarty et al. (2017). What attributes of their dataset or your lemmatizer could contribute to this discrepancy?\n\nC. The methodology section highlights the importance of PoS tags for the lemmatizer to operate accurately. Given the dependency on the accuracy of the PoS tagger, are there plans to improve the PoS tagger or to incorporate multiple PoS tagging mechanisms to increase the robustness of the lemmatizer?\n\nD. The limitations section identifies issues with out-of-dictionary words. Considering this limitation, how does the BanLemma handle real-world scenarios where the system might frequently encounter such words, and what are the proposed strategies to mitigate this limitation in future iterations?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback, and we believe these will improve the quality of the paper. We are writing our response to each point separately below.\n\n**Our response to the ”Questions For The Authors\":**\n1. The paper mentions a unique approach of analyzing the suffix marker occurrence according to the morpho-syntactic values and then utilizing sequences of suffix markers instead of entire suffixes. How does this method differ from traditional methods, and what specific advantages does it offer for Bangla language processing?\n\n    **Response:** Below we explain how the proposed method differs including its specific advantages over other existing methods:\n     - The traditional method of stripping an entire suffix fails to cover the lemmatization of different Bangla word formations because of its highly inflected nature. Bangla words are inflected with four different markers (e.g., Case marker, Plural marker, Determiner, and Emphasis marker) that convey grammatical meanings and there are some specific sequences in their occurrence [1]. Moreover, the structure of Bangla words supports using multiple suffixes sequentially as well as with different combinations of the same word. To make it more challenging, the lemma may end with characters that are similar to the inflection markers [2]. This is why stripping the entire suffix fails even in the most common cases for Bangla lemmatization.\n\n          For example, ‘মেয়েরটা’ (meʲerta; girl’s) in this word lemma is মেয়ে (meʲe; girl) and inflected with র (r) and টা(ta). Another word মায়েরটা (maʲerta; mother’s) where মা (ma; mother)  is the lemma and inflected with য়ের (er) and টা (ta). Traditional lemmatizer would strip the য়েরটা (erta) in both cases resulting in invalid lemma মে (me) in মেয়েরটা (meʲerta) as the lemma ends with য়ে (ʲe). Here, BanLemma will strip টা (ta) first and র then cross-reference with the dictionary and return the accurate lemma মেয়ে (meʲe; girl). So stripping morpho-syntactic sequences of characters at a time produces better results than stripping entire suffixes. \n\n\n\n     - BanLemma covers almost all possible suffixes by using the combination of markers rather than just a list of fixed suffixes (No previous work found that could cover all the suffixes). Moreover, we will also release our suffix list upon acceptance of the paper.\n     - As markers are limited and generally very short in length, often 1-3 characters, it provides computational advantages rather than matching a long list of suffix strings. \n\n    **References:**\n\n     [1] Bhattacharya, Samit, et al. \"Inflectional morphology synthesis for bengali noun, pronoun and verb systems.\" Proc. of the National Conference on Computer Processing of Bangla (NCCPB 05). 2005.\n\n     [2] Islam, Rafiqul & Sarkar, Pabitra (2017). Bangla Academy Pramita Bangla Bhashar Byakaran. Dhaka: Bangla Academy.\n\n2. The reported accuracy of BanLemma on your own human-annotated test set is 96.36%, which is quite impressive. However, there's a discrepancy in performance when using the test dataset provided by Chakrabarty et al. (2017). What attributes of their dataset or your lemmatizer could contribute to this discrepancy?\n\n    **Response:** We have observed several discrepancies in Chakrabarty et al. (2017) efforts of annotating lemmatizing Bangla words. We have discussed the type of errors (to the best of our knowledge) in their work below:\n\n     - Chakrabarty et al. (2017) made changes to the gender of a set of words of their choice, such as transforming যুবতী (ɟubot̪i; young girl) to যুবক (ɟubok; young boy). However, according to [2] lemmatization does not intend to change the semantic meaning of a word. \n \n    - As discussed in [1], inflection does not involve adding prefixes and altering the word's meaning. But in their dataset, by stripping prefixes they altered negated forms to positive forms, such as changing অদূর (ɔd̪ur; not so far) to দূর (d̪ur; far).\n    \n    - They kept inconsistent annotations for pronouns, such as তোমার (your) is annotated as both তুমি (t̪umi; you) and তোমার (t̪omar; your). We always considered the lemma of তোমার (t̪omar; your) as তোমার (t̪omar; your).\n\n     - They made derivational changes, such as transforming বাণিজ্যিক (baniɟik; commercial) to বাণিজ্য (baniɟɟo; trade), প্রকৃতি (prokrit̪i; nature)  to প্রকৃত (prokrit̪o; real), and so on. But in study [2] authors discuss that lemmatizer does not involve derivational changes and we also do the same in BanLemma.\n     - They had errors, including misspelling and incorrect lemmatization of words such as transforming সংশোধনাগার (ʃɔŋʃod̪ʱonagar; reformatory) as সংশোধনঅগার (ʃɔŋʃod̪ʱonɔgar; )\n\n    To effectively evaluate our proposed system, we made an attempt to check and correct some erroneous lemmas. We took 52 sentences at random which contained about 695 words and corrected the lemma as well as annotated the PoS tags. Upon acceptance, we plan to open the sampled portion along with the corrected lemma and annotated PoS tags. On this sampled portion, we achieved an accuracy of 94.34% measured against the corrected lemmas and using the annotated PoS tags. This performance is reported in Table 7 of the paper.\n\n    **References:**\n\n     [1] Lieber, Rochelle. Introducing morphology. Cambridge University Press, 2021.\n\n     [2] Karwatowski, Michal, and Marcin Pietron. \"Context based lemmatizer for Polish language.\" arXiv preprint arXiv:2207.11565 (2022).\n\n\n3. The methodology section highlights the importance of PoS tags for the lemmatizer to operate accurately. Given the dependency on the accuracy of the PoS tagger, are there plans to improve the PoS tagger or to incorporate multiple PoS tagging mechanisms to increase the robustness of the lemmatizer?\n\n    **Response:** \n    Lemmatization considers a word's part of speech (PoS) in order to accurately ascertain its lemma, given that the lemma's meaning can undergo modifications based on its grammatical role [1] [2]. Our exploration of prior lemmatization research revealed that prominent contemporary methodologies, such as Trankit [3], TurkuNLP [4], and LEMMING [5], utilize PoS tags before the lemmatization process. Drawing inspiration from these valuable findings, we purposefully tailored our approach to align with this prevailing paradigm, particularly due to our involvement in an expansive initiative dedicated to advancing Bangla NLP research and its practical implementations. Concurrently, within this overarching framework, we are actively involved in enhancing the automated PoS tagging process. Notably, the present study is exclusively centered on refining the lemmatization methodology for the Bangla Language. \n\n    **References:**\n\n     [1] Bhattacharya, Samit, et al. \"Inflectional morphology synthesis for bengali noun, pronoun and verb systems.\" Proc. of the National Conference on Computer Processing of Bangla (NCCPB 05). 2005.\n\n     [2] Karwatowski, Michal, and Marcin Pietron. \"Context based lemmatizer for Polish language.\" arXiv preprint arXiv:2207.11565 (2022).\n\n     [3] Minh Van Nguyen, Viet Dac Lai, Amir Pouran Ben Veyseh, and Thien Huu Nguyen. 2021. Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural Language Processing. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 80–90, Online. Association for Computational Linguistics.\n\n     [4] Kanerva, Jenna, et al. “Universal Lemmatizer: A Sequence-to-Sequence Model for Lemmatizing Universal Dependencies Treebanks.” Natural Language Engineering, vol. 27, no. 5, 2021, pp. 545–574., doi:10.1017/S1351324920000224.\n\n     [5] Müller, Thomas, et al. \"Joint lemmatization and morphological tagging with lemming.\" Proceedings of the 2015 conference on empirical methods in natural language processing. 2015.\n\n4. The limitations section identifies issues with out-of-dictionary words. Considering this limitation, how does the BanLemma handle real-world scenarios where the system might frequently encounter such words, and what are the proposed strategies to mitigate this limitation in future iterations?\n\n    **Response:**\n\tThank you for your concern. In response to your query, an assessment of the generalizability of the proposed method was conducted through an out-of-dictionary (OOD) test. The procedure encompassed the retrieval of recent data from diverse sources (e.g., newspaper, facebook comment, blogpost). Subsequently, we measured the performance of the lemmatizer on the out-of-dictionary words only and found an accuracy of 79% where we used the BNLP automatic PoS tagger [1] to obtain the PoS tags. During analysis we found the following things:\n\n      - Though a word is not present in the dictionary, the inflection pattern is known for the particular word’s PoS class and sequential stripping of the markers leads to the correct lemma most of the time.\n\n      - The performance of the lemmatizer on OOD words can also increase upon using a more accurate or improved PoS tagger.\n\n\tOther than improving the PoS tagger, we plan to take the following steps:\n\n      - Enriching the dictionary: Using the vast data available in different online sources, we would take steps to identify keywords and phrases and enrich the dictionary.\n\n      - Hybrid system: Leveraging this system we plan to create a Bangla lemmatization data set and develop a neural network based or hybrid lemmatization system in future.\n\n     **References:**\n\n      [1] Sarker, Sagor. \"Bnlp: Natural language processing toolkit for Bengali language.\" arXiv preprint arXiv:2102.00405 (2021).\n\n\n**Our response to the “Reasons To Reject (Weakness)\":**\n1. The paper notes a significant decrease in performance when used with an automatic PoS tagger. A lemmatizer that relies heavily on the accuracy of an external PoS tagger may suffer in real-world applications if the PoS tags aren't accurate.\n\n    **Response:** Kindly refer to question 3 in \"Questions to Authors\" section, where we have discussed the dependency of PoS tagger and outlined our strategies for addressing them in upcoming iterations.\n2. The described limitation where in the lemmatizer strips suffixes from words that are already lemmas but end with a suffix substring is a major concern for real-world texts with evolving vocabularies.\n\n    **Response:** Kindly refer to question 3 in \"Questions to Authors\" section, where we have discussed the case regarding our lemmatizer performance on Out of Dictionary (OOD) words.\n3. The lemmatizer demonstrates notably lower performance on the dataset provided by Chakrabarty et al. (2017). Such variance in performance on different datasets indicates potential overfitting to certain data types or lack of generalizability.\n\n    **Response:** In question 2 of “Questions to Authors” we have discussed the case performance discrepancy on Chakrabarty et al. (2017) dataset.\n4. While the paper mentions that the rules were derived from rigorous analysis, the specifics of the rules and their formation aren't deeply discussed. This makes it hard to gauge the lemmatizer's adaptability to nuanced linguistic challenges.\n\n    **Response:** Thank you for your feedback. We have already discussed our proposed rules in detail with their formation in our article (please see section 3.1 and appendix A.3). However we have a plan to write up an in-detailed document and release it to the public through GitHub along with code upon acceptance.\n5. The examples where the lemmatizer fails are linked to the PoS tagger's inaccuracies. A more comprehensive analysis of the failure cases, beyond PoS errors, would have provided a holistic view of the lemmatizer's weaknesses.\n\n    **Response:** In the limitations section of the paper, we covered instances where the lemmatizer encounters challenges, encompassing scenarios like Out of Dictionary words, words with ambiguous semantic meanings etc.\n\n\n**Our response to the “Reasons To Reject (Risks)\":**\n\n1. If presented at a conference, attendees may question the lemmatizer's generalizability, given its varied performance on different datasets.\n\n    **Response:** Thank you for your valuable response. In our experimentation we have taken utmost care for achieveing generalizability of our proposed BanLemma. From our in-depth experiments, we have observed that, the baseline datasets we use has several grammatical errors. Below we will mention the key errors we have found from our experimentations, however the detail analysis can be found in our submitted article (Section: Appendix (A.9), from line 930).\n\n    - BenLem dataset: Labeled verb roots differently from dictionary forms; converted colloquial to classical pronouns; spelling changes impacted lemmatization.\n\n    - Banel: Inconsistent derivational marker removal; altered pronoun forms; spelling changes affected outcomes; contained incorrect lemmas.\n\n    - Chakrabarty et al. (2017): Modified gender, negated forms, and pronouns; applied derivational changes impacting lemmatization, erroneous PoS labeling.\n\n    It is worth noting that, inspite of this error, we have achieved competitive results on BaNeL and BenLem dataset. As we have mentioned earlier in terms of data discrepancies, we have randomly choose 52 sentences (more than 700 words) from that dataset and corrected their mistakes before testing with our proposed method. By correcting the random samples from Chakrabarty et al. (2017) we have gained a 3% performance improvement. The aforementioned statement has already been mentioned in the Results section (See table 6, 7 and line 550-572).\n\n     **References:**\n\n     [1] Islam, M.A., Towhiduzzaman, M., Bhuiyan, M.T.I. et al. BaNeL: an encoder-decoder based Bangla neural lemmatizer. SN Appl. Sci. 4, 138 (2022). https://doi.org/10.1007/s42452-022-04985-2\n\n\n2. The heavy dependency on PoS taggers could raise questions about the stand-alone efficacy of BanLemma. If other researchers attempt to reproduce results or integrate the lemmatizer into different systems, the variance in PoS tagging quality may produce inconsistent results.\n\n    **Response:** We would like to thank you for your feedback. The use of parts of speech lemmatization has proven to be very useful in achieving state-of-the-art results (e.g., Trankit [1], TurkuNLP [2], and LEMMING [3]). Thus we believe that reproducing the results should not have any problem since we have used publicly available resources such as BNLP PoS tagger [1] which is available online to ensure that we can avoid inconsistency in our results reported in this study. Additionally, we will publish our code, dictionary, and all other resources related to this study through GitHub upon acceptance of this paper.\n\n     **References:** \n\n     [1] Minh Van Nguyen, Viet Dac Lai, Amir Pouran Ben Veyseh, and Thien Huu Nguyen. 2021. Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural Language Processing. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 80–90, Online. Association for Computational Linguistics.\n\n    [2] Kanerva, Jenna, et al. “Universal Lemmatizer: A Sequence-to-Sequence Model for Lemmatizing Universal Dependencies Treebanks.” Natural Language Engineering, vol. 27, no. 5, 2021, pp. 545–574., doi:10.1017/S1351324920000224.\n\n     [3] Müller, Thomas, et al. \"Joint lemmatization and morphological tagging with lemming.\" Proceedings of the 2015 conference on empirical methods in natural language processing. 2015.\n \n\n3. Without clear details on rule formation and limitations, other researchers may misapply or misinterpret the lemmatizer's capabilities, leading to unintended consequences in subsequent research or applications.\n\n    **Response:** We have made an effort to provide a comprehensive explanation of our rules in the manuscript (see section 3 and Appendix A.3). With due respect to your feedback, we will create elaborate descriptions of each lemmatizing rule and release those through GitHub. We will also open source all our code related to this study upon acceptance of this paper.\n\n\n\n**Regarding Typos Grammar Style And Presentation Improvements:**\nWe sincerely apologize for the typographical errors. We will take the necessary steps to revise the paper and ensure that all mistakes are rectified.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strengths:\n\n1. The paper introduces BanLemma, based on a detailed and rigorous analysis of a large Bangla text corpus, allowing for a comprehensive understanding of Bangla word formation.\n   \n2. Instead of focusing on entire suffixes, BanLemma analyzes suffix marker sequences, addressing the limitation of previous lemmatization methods.\n   \n3. Achieving an accuracy of 96.36% on a human-annotated test dataset is commendable and shows the potential and efficiency of BanLemma.\n   \n4. The paper demonstrates the ability of BanLemma to perform competitively on various datasets, surpassing other lemmatizers on certain test sets.\n    \n5. The paper delves deeply into its methodology, detailing the creation of rules and dictionary for BanLemma, and the extensive sources they utilized.\n\nBenefits to the NLP community:\n\n1. The paper tackles lemmatization in Bangla, a less-represented language in NLP. As Bangla is morphologically rich, addressing lemmatization in this context is essential for many higher-level NLP tasks.\n\n2. The comprehensive dataset, rules, and methods introduced can be a foundational resource for future researchers working on Bangla NLP.\n    \n3. By combining linguistic rules with NLP techniques, the paper demonstrates the potential benefits of such hybrid methods, possibly encouraging more linguistically-informed approaches in the field.", "reasons_to_reject": "Weaknesses:\n\n1. The paper notes a significant decrease in performance when used with an automatic PoS tagger. A lemmatizer that relies heavily on the accuracy of an external PoS tagger may suffer in real-world applications if the PoS tags aren't accurate.\n \n2. The described limitation wherein the lemmatizer strips suffixes from words that are already lemmas but end with a suffix substring is a major concern for real-world texts with evolving vocabularies.\n\n3. The lemmatizer demonstrates notably lower performance on the dataset provided by Chakrabarty et al. (2017). Such variance in performance on different datasets indicates potential overfitting to certain data types or lack of generalizability.\n\n4. While the paper mentions that the rules were derived from rigorous analysis, the specifics of the rules and their formation aren't deeply discussed. This makes it hard to gauge the lemmatizer's adaptability to nuanced linguistic challenges.\n\n5. The examples where the lemmatizer fails are linked to the PoS tagger's inaccuracies. A more comprehensive analysis of the failure cases, beyond PoS errors, would have provided a holistic view of the lemmatizer's weaknesses.\n\nRisks:\n\n1. If presented at a conference, attendees may question the lemmatizer's generalizability, given its varied performance on different datasets.\n\n2. The heavy dependency on PoS taggers could raise questions about the stand-alone efficacy of BanLemma. If other researchers attempt to reproduce results or integrate the lemmatizer into different systems, the variance in PoS tagging quality may produce inconsistent results.\n\n3. Without clear details on rule formation and limitations, other researchers may misapply or misinterpret the lemmatizer's capabilities, leading to unintended consequences in subsequent research or applications.", "questions_to_authors": "A. The paper mentions a unique approach of analyzing the suffix marker occurrence according to the morpho-syntactic values and then utilizing sequences of suffix markers instead of entire suffixes. How does this method differ from traditional methods, and what specific advantages does it offer for Bangla language processing?\n\nB. The reported accuracy of BanLemma on your own human-annotated test set is 96.36%, which is quite impressive. However, there's a discrepancy in performance when using the test dataset provided by Chakrabarty et al. (2017). What attributes of their dataset or your lemmatizer could contribute to this discrepancy?\n\nC. The methodology section highlights the importance of PoS tags for the lemmatizer to operate accurately. Given the dependency on the accuracy of the PoS tagger, are there plans to improve the PoS tagger or to incorporate multiple PoS tagging mechanisms to increase the robustness of the lemmatizer?\n\nD. The limitations section identifies issues with out-of-dictionary words. Considering this limitation, how does the BanLemma handle real-world scenarios where the system might frequently encounter such words, and what are the proposed strategies to mitigate this limitation in future iterations?", "paper_topic_and_main_contributions": "The paper addresses the challenge of lemmatization for the Bangla language, an essential task in natural language processing (NLP) and linguistics. \n\nMain contributions:\n\n1. The paper presents BanLemma, a lemmatization system specifically tailored for Bangla. This system shows superior performance when compared to existing Bangla lemmatization methods.\n\n2. The study provides a set of linguistic rules that explain how Bangla-inflected words are derived from their respective base words or lemmas. These rules result from a rigorous analysis of an expansive Bangla text corpus.\n\n3. The linguistic rules are based on a comprehensive analysis of a Bangla text corpus of 90.65M unique sentences. This corpus includes 0.5B words, of which 6.17M are distinct. The study manually sampled and analyzed 22,131 words from this corpus to understand the formation of inflected words.\n\n4. The paper establishes an evaluation framework, conducting intra-dataset and cross-dataset evaluations. BanLemma achieved an accuracy of 96.36% on intra-dataset testing and demonstrated robust performance across different datasets.\n\n5. BanLemma outperformed several recently published methodologies in cross-dataset evaluations, indicating performance improvements ranging from 1% to 11%.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. The word \"processs\" in the sentence \"Figure 1 shows an illustration of our lemmatization processs...\" should be \"process\".\n2. In the sentence \"However, due to the highly inflected nature and morphological richness, lemmatization in Bangla text poses a complex challenge.\", the use of both \"highly inflected\" and \"morphological richness\" seems redundant. These terms are closely related, and the sentence could be made more concise.\n3. In the sentence \"We have curated a human-annotated test set annotated by trained linguists.\", the word \"annotated\" is repeated. It would be clearer to say, \"We have curated a human-annotated test set by trained linguists.\"\n4. In the sentence \"By embracing the marker sequence striping approach, we are able...\", the word \"striping\" should be corrected to \"stripping\"."}}, {"review_id": "bGkGFzDB98", "reviewer": "Reviewer_ATsq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback, and we believe these will improve the quality of the paper. We are writing our response to each point separately below.\n1. Dicitionary-based approach for lemmatization has some limitations: the words out of the dictionary may incorrectly lemmatized.\n\n    **Response:** We have conducted an Out of Dictionary (OOD) test to measure the performance for words that are not present in the dictionary. The procedure encompassed the retrieval of recent data from diverse sources (e.g., newspaper, facebook comment, blogpost). Subsequently, we measured the performance of the lemmatizer on the OOD words only and found an accuracy of 79% where we used the BNLP automatic PoS tagger [1] to obtain the PoS tags. During analysis we found the following things:\n    - Though a word is not present in the dictionary, the inflection pattern is known for the particular word’s PoS class and sequential stripping of the markers leads to the correct lemma most of the time.\n\n    - The performance of the lemmatizer on OOD words can also increase upon using a more accurate or improved PoS tagger.\n\n    Thus, our method may produce incorrect results for OOD words, however, the sequential stripping proves to be an effective process to achieve generalizability for both OOD and dictionary words.\n\n     **References:**\n\n     [1] Sarker, Sagor. \"Bnlp: Natural language processing toolkit for bengali language.\" arXiv preprint arXiv:2102.00405 (2021).\n\n2. The authors evaluate their lemmatizer on the correct lemmas (only accuracy).\n\n    **Response:** We conducted additional experiments following your feedback to assess our lemmatizer's performance using our original test dataset. We evaluated its accuracy on two types of data: non-inflected words (these are usually the words found in the dictionary, for example exceptions may occur for proper nouns), inflected words (these words need to be lemmatized). In these considerations, there are 6906 non-inflected words and 3125 inflected words.\n\n     Our experimental results include a macro precision of 0.9537, macro recall of 0.9597, macro F1 score of 0.9567 and overall accuracy of 0.9636 (also reported in the paper) for these two categories of words. These findings will be included in the appendix of the final manuscript. We include the detail result for both the non-inflected and inflected words below:\n\n     **Non-inflected:**\n\n     - Precision : 0.9784245583550536\n\n     - Recall : 0.9681902851411377\n\n     - F1 Score : 0.9732805185451926\n\n     **Inflected:**\n\n     - Precision : 0.92896\n\n      - Recall : 0.9511795543905636\n\n      - F1 Score : 0.9399384814634937", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "BanLemma achieved a high accuracy 96.36% compared to existing state-of-the-art Bangla lemmatization methods. The lingustic rules were derived from analysis of Bangla text corpus of 90.65M unique sentences. ", "reasons_to_reject": "Dicitionary-based approach for lemmatization has some limitations: the words out of the dictionary may incorrectly lemmatized. The authors evaluate their lemmatizer on the correct lemmas (only accuracy). ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper is about lemmatization for Bangla. The authors propose rule- and dictionary-based approach for lemmatization to design BanLemma lemmatization system. The authors develop new method based on the suffix marker sequences considering of the part-of-speech class of the word. ", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a rule-based Bangla lemmatizer. It strips suffix markers and validates the predicted lemma against a dictionary.

It is somewhat unexpected to encounter a rule-based language analyzer in 2023. However, in comparison to prior research, this paper demonstrates relatively strong performance. The authors conducted comprehensive experiments and transparently outlined the study's limitations. Although it may not reach a broad audience, this paper deserves publication in some form.
Decision: Accept-Findings