On Evaluation of Bangla Word Analogies
Authors: Mousumi Akter
Souvika Sarkar
Shubhra Kanti Karmaker Santu
Venue: EMNLP 2023 Main (2023)
Keywords: Bangla
Word Analogy
Evaluation
TL;DR: This paper presents a benchmark dataset of Bangla word analogies for evaluating the quality of existing Bangla word embeddings.

Abstract: This paper presents a benchmark dataset of Bangla word analogies for evaluating the quality of existing Bangla word embeddings. Despite being the 7th largest spoken language in the world, Bangla is still a low-resource language and popular NLP models often struggle to perform well on Bangla data sets. Therefore, developing a robust evaluation set is crucial for benchmarking and guiding future research on improving Bangla word embeddings, which is currently missing. To address this issue, we introduce a new evaluation set of 16,678 unique word analogies in Bangla as well as a translated and curated version of the original Mikolov dataset (10,594 samples) in Bangla. Our experiments with different state-of-the-art embedding models reveal that current Bangla word embeddings struggle to achieve high accuracy on both data sets, demonstrating a significant gap in multilingual NLP research.

On Evaluation of Bangla Word Analogies
Mousumi Akter, Souvika Sarkar, Shubhra Kanti Karmaker ("Santu")
Big Data Intelligence (BDI) Lab
Department of Computer Science & Software Engineering Aubur University, Alabama, USA (mza0170, szs0239, sks0086) @auburn.edu
Abstract
contrary, there is clear absence of word anal- ogy data sets for evaluating the syntactic and se- mantic properties of Bangla word embeddings, al- though such word analogy evaluation data sets are available for other languages Like Turkish, Ger- man, Spanish, Arabic, etc. (Gurevych, 2005; Has- san and Mihalcea, 2009; Sak et al., 2010; Joubarne and Inkpen, 2011; Panchenko et al., 201
This paper presents a benehmark dataset of Bangla word analogies for evaluating the qual- ity of existing Bangla word embeddings. De- spite being the 7th largest spoken language in the world, Bangla is still low-resource lan- guage and popular NLP models often struggle perform well on Bangla data sets. Therefore, developing robust evaluation set is crucial for benchmarking and guiding future research improving Bangla word embeddings, which is currently missing. To address this issue, we introduce a new evaluation set 16,678 unique word analogies in Bangla as well translated and curated version of the original Mikolov dataset (10,594 samples) in Bangla. Our experiments with different state-of-the-art embedding models reveal that current Bangla word embeddings struggle achieve high ac- curacy on both data sets, demonstrating sig- niticant gap multilingual NLP research.
As word embeddings have direct impacts on the performance of downstream NLP tasks, creat- ing a high-quality evaluation test set for Bangla word embeddings will enable researchers to bench- mark the performance of existing Bangla embed- ding models and guide further research. With this motivation, present a Mikolov-style (Mikolov et al., 2013) high-quality word-analogy evalua- tion set exclusively for Bangla, with a sample size of 16,678!. To the best of our knowl- edge, we are the first ones to do so. In Addi- tion, we translated and curated Mikolov's origi- nal dataset for Bangla, resulting in 10,594 more analogies. For both these data sets, we also pro- vide analysis of the performance of several state- of-the-art Bangla/multtlinguaal embedding models: Word2Vec (Mikolov al., 2013), GloVe (Pen- nington et al., 2014), fastText (Bojanowski et al., 2017), LASER (Artetxe and Schwenk, 2019), LaBSE (Feng et al., 2022), bnBERT (Bhattachar- jee et al., 2022a), bnBART (Wolf et al., 2020).
Introduction
The Bangla language, having over 300 million na- tive speakers and ranking as the seventh most spo- ken language in the world, still regarded as a lan- guage with limited resources (Joshi et al., 2020). Despite the breakthrough in the field of Natural Language Processing (NLP) recently, it has been reported that popular NLP models often fail to per- form well for low-resource languages like Bangla while showing human-like performance in high- resource languages like English. As such, there growing need for high-quality Bangla word embed- dings, and several efforts have been made in this direction by training embeddings on large Bangla corpora (Bhattacharjee et al., 2022a; Artetxe and Schwenk, 2019; Feng et al., 2022). For evalua- tion, there have been few efforts create Bangla benchmark data sets for downstream NLP tasks like Sentiment Analysis, Machine Translation, and Summarization (Hasan et al., 2020, 2021; Akil et al., 2022; Bhattacharjee et al., 2022b). On the
Our experiments with different statc-of-the-art embedding models reveal that current Bangla word embeddings struggle achieve high accuracy on both data sets, signifying that Bangla has its own unique characteristics, and further rescarch war- ranted for low-resource languages like Bangla. Therefore, we suggest that future research on Bangla word embeddings should report accuracy using this new benchmark data set in order to track consistent research progress in this direction.
analogy http:://wwppowwwtioood..coddaaca wttp:///wwwdooddddddddeeeeeeeeee

---

Type
Relationship
Subgroup
Sample !
Example
Word
Division-
2160
DRF (Dhaka)
कह (Shariatpur)
Re2a (Rangpar)
TلमÍ (Kurigram)
Districi
Somantic
Gender
करी (Father)
xI (Mother)
5T (Uncle)
115 (Aunt)
Ordinal
380
4 (One)
244 (First)
91 (Five)
भकн (Fifth)
Number
Date
930
(One)
भाकत (First)
7 (Twe)
09सय0 (Second)
Female
(One)
(First)
(Twe)
रिका (Second)
Comparative
552
कोेर (Cool)
भरeाडz (Cooler)
कां (Long)
काोिक (Longer)
Superaative
600
ANd (Long)
#Nअअर (Longest)
(Low)
गतीнम (Lowest)
Adjective
म|री (Expensive)
*08 (Chenp)
н (Addicted)
Heकरiee (Desperate)
Antoryym
Misc.
3782
anaae (Arrival)
E (Departure)
Bank (Upstream)
wfß (Downsrream)
Plural
Noun
506
(Sailor)
शालाा (Sailons)
उ (Student)
म (Students)
Object
(Compusition)
कहमारीी (Compositions)
сार (Rule)
मारारीी (Rules)
Syntactic
Tonse
144
soffk (Continued)
bikk (Continuing)
yid (Will continue)
bVic (Will Continne)
Prefix
66
s64 (Knowledge)
kara (Seience)
CRR (Fnd)
RDR (Special)
Sullix
54
ER (Alijude)
вाया (Aitjude)
<]मा (Malier)
रामााराा (Maller)
Affix
38
<ीa (Night)
sface (Night)
*f (Elephant)
мffece (Hlephant)
Noun
(Pish)
(Fish)
žś (Hlephant)
अोड (Elephant)
Standard
Pronoun
110
काेा (Ilss)
सтоP (His)
टाहर (Ilis)
शार (Ilis)
Colloquial
Verb
462
टेनार (Become)
शिस (Become)
कहककरर (Du)
पााा (Do)
Conjunction
अगती (Although)
*ffi (Althhough
eтaM (Oftee)
सtअi (Oftzn)
Table 1: This table shows the statistics of the dataset and examples the semantic and syntactic relationship sets, Relationships highlighted in cyan represent unique linguistic forms for Bangla, while highlighted in red represent somewhat unique forms with different syntax compared to English.
2 Dataset
literature, stories, and novels. A total of 2,844 word pairs were formed that reflect these unique characteristics of the Bangla language. Further- more, 3,776 word pairs were introduced in the eval- uation set that is absent in Mikolov's word analogy data set, such as division-district pairs and number pairs with different forms. These additional word forms further demonstrate the diverse and complex nature of the Bangla language.
To evaluate the quality of word vectors, the au- thors (who are also native Bengali speakers) inde- pendently first proposed different types of relation- ships for the Bangla Language. Subsequently, they created a list of related word pairs for each relation- ship and then formed analogies by grouping two pairs. The pairs were then independently reviewed by other annotators and removed if there was any disapproval from the annotator. Thus, a compre- hensive test set was developed that contains three types of semantic analogies and nine types of syn- tactic analogies, shown in Table 1. Overall, our testing set contains 5,036 semantic and 11,642 syn- tactic analogies, providing comprehensive col- lection for evaluating the quality of Bangla word vectors. For example, we made list of 7 divi- sions in Bangladesh and 64 districts that belong to these divisions and formed 2,160 analogies by picking every possible division-district pair. Fol- lowing Mikolov's approach (Mikolov et al., 2013), word analogies were created by using the format wordg- word, wordp worde. The goal was to determine the wordp that is similar to worde in the same way that words is similar to word,-
Additionally, translated Mikolov's 's original dataset² and manually removed Linglish words that do not have Bangla translations. We also removed word pairs that translate to duplicated terms in Bangla, such as present participles and plural verb forms. This cleaning step resulted in a dataset of 10,594 samples from the original 19,544 sam ples. In summary, while our new dataset focused on the linguistic specifics of the Bangla language, the translated Mikolov's dataset is more focused on common words both Bangla and English. The translation of Mikolov's dataset provides a useful resource for cross-lingual research and analysis.
Experimental Setup
We evaluated the quality of pretrained word vec- tors trained using both traditional and transformer- based models on both data sets. For classical models, we used Word2 Glo and fastText, while for transformer-basee models, we employed LASER, LaBSE, baBERT, and buBART, to eval- uate the quality of word embeddings.
While creating this test set, we also took into account the unique characteristics of the Bangla language. For example, the Bangla language has different forms for numbers, including date forms, female forms, and also with prefixes and suffixes that change the meaning of the word. Additionally, Bangla has colloquial forms that are often used in
wwww.fit.wutb..com cz/ imikoloo//mmlm//wortcero

---

Embedding
Vec
GloVe
Tast
LaBSE
bnBERT
LASER
bnBART
Dimension
100
100
300
768
768
1024
1024
Table Dimensions of different embedding used
For GloVe, we used the Bengali GloVe model that was trained on Bangla Wikipedia' and Bangla news articles. Specifically, we utilized the model trained with 39M tokens. For Word2Vee, we used the Bengali Werd2Vec model, also trained with Bengali Wikipedia data (Sarker, 2021). We also used fastText, which provides pretrained Bangla word embeddings (Bojanowski et al., 2017).
Bengali language was trained on Wikipedia dump datasets. It consists of 20 million words, vocabu- lary size of 1110111 and was trained for epochs with a 300-dimensional embedding.
LASER": LASER model performs zero-shot cross-lingual transfer across 90+ languages. It is open-source and supports efficient language pro- cessing and understanding.
For transformer-based models, we used LASER (Artetxe and Schwenk, 2019), LaBSE (Feng al., 2022), bnBERT (Bhat- tacharjee et al., 2022a), bnBART (Wolf et al., 2020), which provide sentence embeddings. To obtain the word embedding for a particular word, we passed the word to the model and collected the contextualized token embeddings for all the tokens of that word, which were then averaged to obtain the word embedding. This way, we created an exhaustive embedding dictionary for 178,152 Bengali words and used them directly to perform the word analogy task. For experiments, we used NVIDIA Quadro RTX 5000 GPUs. Table shows the dimensions of different embedding used. In summary, we performed comprehensive evaluation of pretrained word vectors (using both traditional and transformer-based embedding models) on our two data sets, following the same word analogy task setup used by Mikolov al. (2013).
LaBSE LaBSE is a BERT-based model trained on 109 languages for sentence embedding, combin- ing masked language modeling and translation lan- guage modeling. It provides multilingual sentence embeddings.
bnBERT BanglaBERT an ELECTRA discrim- inator model pretrained with the Replaced Token Detection (RTD) objective, suitable for finetuning on diverse downstream tasks in Bengali.
bnBART bnTransformer is built with trans- formers for various transformer-based inference tasks in Bengali. We pass facebook/bart-laggee model from the huggingface model hub to get the embeddings.
4 Results
Our first set of results is focused on our hand- crafted Bangla analogy set (presented in Table I). In all our experiments, we strictly matched against the target word from our dataset compute Top- 1, Top-3, Top-5, and Top-10 accuracy numbers. Because of strict matching, achieving a very high accuracy was not expected. However, as demon- strated in Figure 1, the accuracy numbers turned out to be even worse than expected. For example, the best overall top-5 accuracy was obtained by bnBART (a 1024-dimensional embedding), which turned out to be only 20.9%. LaBSE 768- dimensional embedding) yielded the best Top-5 ac- curacy for semantic analogies (around 12%), while bnBART gave the best Top-5 syntactic accuracy (around 25%). Top-1 accuracy was generally low in all cases, except for bnBART achieving around 21% Top-1 accuracy for syntactic analogies. Next, we compare our handcrafted dataset
3.1 Bangla Word Embedding
Word2vec The Wikipedia dump datasets were used for training Word2Vec word embeddings of Bangla with dimension of 100. The minimum count of words was set to 5, the window size was 5, and the training performed over 10 epochs. G:oVe The Bangla GloVe Vectors offer pre- trained glove vectors for the Bengali language. The model Trained using Wikipedia dump datasets and crawled bangla news articles with spe- cific parameters including a vector size of 100, a window size of 15, and a maximum iteration of 15, With a vocabulary size of 0.18 million.
fastText The fastText pre-trained model for the
bn. wiw.ieidiii.org//
"github.com wwaiiloooooccccbbecc htw.f/cccoookeeeacttLSSSSEE
"huggingface.c basaagonaakkerbbnglla word2vee
$ "huggingface.co wetpi//gface.c///eeeeeee Senteence-raansfoorrsssnnnn..
'ugginnffcc aaagoosarterbbnglggglloee
9 "huggingface,col/ coleesebuemlppbbanglabbe
github.co mithub.ccm//ee fhnecbookkrssacccffaaaattttt

---

16
Top-1
Top-1
Top-3
Top-3
Top-5
Top-5
Top-10
Top-10
Accuracy (%)
20
Accuracy
15
8-
6-
10-
4-
2-
0
Word2Vec GloVe fastiext LaBSE enBERT LASER bABART
Word2Vec GloVe
fastText LaBSE bnBERT LASER BOBART Embedding
Embedding
(a) Semanntc Accuracy
(b) Syntactic Accuracy
Figure I: Top- Top-3(%)), Top-5(%) and 1op-10(%) accuracy for different embeddings dataset.
Top 5% Performance Break- Down for Different Categories
Type
Relationship
Word2Vee
GloVe
fastText
LaBSE
bnBERT
LASER
bnBART
Division-District
9.8
9.5
0.0
0.1
0.6
Semantic
Gender
11.2
8.5
3.3
21.9
0.1
12.7
12.6
Number
5.1
0.6
16.3
0.4
3.4
Antonym
11.0
7.6
14.5
3.5
0.7
1.0
3.3
Tense
6.3
2.8
4.2
13.9
0.0
16.7
14.6
Comparative
5.1
1.1
6.5
12.5
2.2
5.3
14.7
Superlative
3.5
1.3
3.3
10.2
0.0
6.3
5.2
Syntactic
Prefix
0.0
3.0
0.0
Suffix
0.0
0.8
0.8
30.4
0.0
22.8
53.6
Affix
21.1
23.7
5.3
63.2
0.0
50.0
79.0
Plural
7.3
2.5
1.5
41.6
0.0
18.4
11.1
Standard-Collopue
9.8
3.9
1.9
23.5
0.0
6.6
9.4
Overall Accuracy
7.7
5.6
3.7
20.5
0.3
12.2
20.9
Top 10% Performance Break-Down for Different Categories
Type
Relationship
Word2
GloVe
fastText
LaBSE
bnBERT
LASER
bnBART
Division-Distred
21.5
17.5
0.0
0.1
0.7
0.0
0.0
Semantic
Gender
13.3
11.6
6.4
27.5
0.1
19.4
14.3
Number
9.6
9.8
1.7
21.7
1.1
5.3
Antonym
14.6
10.5
22.6
5.3
0.8
1.6
4.7
Tense
9.7
3.5
5.6
16.0
0.0
20.1
18.8
Comparative
6.2
2.2
10.0
13.0
3.1
8.3
15.2
Superlative
1.3
5.5
12.5
0.0
8.3
Syntactic
Prefix
1.5
1.5
6.1
10.6
0.0
4.6
43.9
Suffix
0.0
0.8
1.7
37.3
0.0
31.2
53.6
Afīx
21.1
26.3
7.9
73.7
0.0
63.2
81.6
Plural
11.3
3.7
2.6
53.8
0.0
27.7
14.6
Standard-
Standard-Colloquial Colloquial
13.6
5.9
3.3
0.0
Overall Accuracy
10.6
7.9
6.1
25.2
0.5
16.6
22.3
Table Top-5(%) and 10 (%) accuracy on three types of semantic and nine types of syntactic relationship set. The highest accuracy for each category is bolded. Results indicate that certain embeddings perform strongly, moderately strongly, weakly depending on the relationship type, with green, blue, and red highlights, respectively.
against the translated Mikolov dataset in Figure and 3. Again, LaBSE and baBART were the over- all winners, and both methods performed better on the handcrafted dataset than the translated one.
multiple downstream tasks helped models bet- capture the syntactic relations among Bangla words. For example, both LaBSE and LASER, which achieved high accuracy on syntactic analo- gies, were trained with multilingual data, where LaBSE was trained on data from 109 languages, and LASER was trained on data from 97 lan-
4.1 Discussion
An interesting observation from the results that training embeddings on multilingual corpora and

---

12
Mikelov-trams Dalaset
17.5
Mikelov-sanss Dalasel
Our Dataset
Our Dataset
Accuracy 06
12.5
Accuracy
7.5
4-
5.0
2.5
0.0
Werd2Vec
GloVe
fastiioxt
LaBSE
bnBERT
LASER
brBART
Werd2Vec
GloVe
LaBSE biBERT
LASER
brBART
Embedding
Embedding
(a) Top-1 (%) Accuracy
(b) Top-5 (%) Accuracy
Figure 2: Comparison of ((%) and Top-5(%) accuracy translated Mikolov datasct and our dataset.
Mikolov-trans. Dataset
Mikolov-trans. Dataset
Dataset
20
Dataset
Accuracy (%6)
10-
Accuracy
10-
Word2Vec
LaBSE Embedding
bnBART
WerdZVes
LaBSE
Embedding
(a) Top-3 (%) Accuracy
(b) Top-10 (%) Accuracy
Figure 3: Comparison of Top-3 (%) and Top-10 10 (%) accuracy on translated Mikolov dataset and our dataset.
guages (Feng et al., 2022). On the other hand, bnBART, another high-performing embedding on syntactic analogies, was finetuned on multiple downstream inference tasks for the Bengali lan- guage, such as QA, NER, MASK generation, Translation, and Text generation. In contrast, Word2 GloVe, and fastTText achieved lower ac- curacy on syntactic analogies and were trained on only Bangla Wikipedia and crawled Bangla news articles.
iments with different word embeddings reveal that current word embeddings for Bangla still struggle to achieve high accuracy on both data sets. To im- prove performance, future research should focus training embeddings with larger data sets while taking into account the unique morphological char- acteristics of Bangla.
Limitations
is important to acknowledge that our study has certain limitations. Firstly, we tried to provide an exhaustive list of all possible word analogies from the Bangla language. However, it is possible that may unintentionally miss some analogies and the data sets can be extended by adding those analo- gies. Additionally, we created the translated data from English word analogies only, and we did not consider other languages. Future research can be conducted with the translated dataset from other languages to explore the potential differences in word analogy relations across languages. More- over, the evaluation of word analogy with propri- ctary large language models, such as ChatGPT and Bard,
For semantic analogies, LaBSE performed the best, followed by Vec. Table 3 demonstrates the category-wise performance in terms of Top-5 and Top-10 accuracy, where LaBSE and baBART turned out to be the overall winners. However, with 20.9% and 25.2% best Top-5 and Top-10 accuracy scores, respectively, we can safely con- clude that state-of-the--rt Sangla/multilingur em- beddings struggle a lot to capture the semantic and syntactic relations among Bangla words properly.
5 Conclusion
In this paper, have introduced high-quality dataset for evaluating Bangla word embeddings through word analogy tasks, similar Mikolov's word analogy data set for English. Addition- ally, we translated Mikolov's original dataset into Bangla, contributing a second data set. Our exper-
explored in this study. Rather, the focus was entirely on models from which Word embeddings could be directly obtained and sub- sequently tested. Further research may be under- taken this direction with large language models.

---

7 Acknowledgements
Second International Joint Conference, Jeju Island, Korea, October 11-13, 2005, Proceedings, volume 3651 of Lecture Notes in Computer Science, pages 767-778. Springer.
This work has been partially supported by the Na- tional Science Foundation (NSF) Standard Grant Award #2302974 and Air Force Office of Sci- entific Research Grant'Cooperative Agreement Award -FFA9550-23-1-044 thank Auburn University College of Engincering and the Department of CSSE for their continuous support through Student Fellowships and Faculty Startup Grants.
Tahmid Hasan, Abhik Bhattacharjee, Wasi Uddin Ah- mad, Yuan-Fang Li, Yong-Bin Kang, and Rifat Shahriyar. 2021. Crosssum: Beyond english- centric cross-lingual abstractive text summarization for 1500+ language pairs. CoRR, abs/2112.08804.
We would also like to
Tahmid Hasan, Abhik Bhattacharjee, Kazi Samin, Ma- sum Hasan, Madhusudan Basak, M. Sohel Rahman, and Rifat Shahriyar. 2020. Not low-resource any- more: Aligner ensembling, batch filtering, and new datasets for bengali-english machine translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 2612 2623. Association for Computational Linguisties.
References
Ajwad Akil, Najrin Sultana, Abhik Bhattacharjee, and Rifat Shahriyar. 2022. Banglaparaphrase: high- quality bangla paraphrase dataset. In Proceedings of the 2nd Conference of the Asia-Pacific Chap- ter of the Association for Computational Linguis-
Samer Hassan and Rada Mihalcea. 2009, Cross-lingual semantic relatedness using encyclopedic knowledge. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, EMNLP 2009, 6-7 August 2009, Singapore, A meeting of SIG- DAT, a Special Interest Group of the ACI., pages 1192-1201. ACL
tics and the 12th International Joint Conference on Natural Language Processing, AACI/IJCNI. 2022, pages 261-272. Association for Computational Lin- guistics.
Mikel Artetxe and Holger Schwenk. 2019,
Mas- sively multilingual sentence embeddings for zero- shot cross-lingual transfer and beyond. Trans. Assoc. Comput. Linguistics, 7:597-610.
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, Juline,July 5-10 2020, pages 6282-632 0292-6293.44 As- sociation for Computational Linguistics.
Abhik Bhattacharjee, Tahmid Hasan, Wasi Ahmad, Kazi Samin Mubasshir, Md Saiful Islam, Anindya Iqbal, M. Sohel Rahman, and Rifat Shahriyar. 2022a. Banglabert: Language model pretraining and bench- marks for low-resource language understanding eval- uation in bangla. In Findings of the Association for Computational Linguistics: NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 1318- 1327. Association for Computational Linguistics.
Colette Joubarne and Diana Inkpen. 2011. Compari- son of semantic similarity for different languages us- ing the google n-gram corpus and second-order co- occurrence measures. In Advances in Artificial In- telligence - 24th Canadian Conference on Artificial Intelliennee Canadian Al 2011, St. John's is, Canada, May 25-27, 2011. Proceedings, volume 6657 of Lec- ture Notes in Computer Science, pages 216-221. Springer.
Abhik Bhattacharjee, Tahmid Hasan, Wasi Uddin Ah- mad, and Rifat Shahriyar. 20226. Banglanlg: Bench- marks and resources for evaluating low-resource natural language generation in bangla. CoRR, abs/2205. 11081.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Trans. Assoc. Comput. Lin- guistics, 1:5--1146.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word represen- tations in vector space. In Ist International Con- ference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings.
Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. Language- agnostic BERT sentence embedding. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), ACI. 2022, Dublin, Ireland, May 22-27, 2022, pages 878-891. 891, Association for Computational Lin- guistics.
Alexander Panchenko, Dmitry Ustalov, Nikolay Arefyev, Denis Paperno, Natalia Konstantinova, Natalia V. Loukachevitch, and Chris Biemann. 2016. Human and machine judgements for russian semantic relatedness. In Analysis of Images, Social Networks and Texts 5th International Conference, AIST 2016, Yekaterinburg, Russia, April 7-9, 2016, Revised Selected Papers, volume 661 of Communi- cations Computer and Information Science, pages 221-235.
Tryna Gurevych. 2005. Using the structure of con- ceptual network in computing semantic relatedness. In Natural Language Processing- - IJCNLP 2005,

---

Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Confer- ence on Empirical Methods in Natural Language Processing. EMNLP 2014, October 25-29, 2014. Doha, Qalar, A meeting of SIGDAT, Special Inter- est Group of the ACI., pages 1532-1543. ACL.,
Hasim Sak, Murat Saraclar, and Tunga Güngör. 2010. Morphology-basee and sub-word language model- ing for turkish speech recognition. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing. ICASSP 2010, 14-19 March 2010, Sheraton Dallas Hotel, Dallas, Texas, USA, pages 5402-5405, IEEE
Sagor Sarker. 2021.
BNLP: natural language pro- cessing toolkit for bengali language. CoRR, abs/2102.00405.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- rie Cistac, Tim Rault, Rémi Louf, Morgan Fun- towicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jemite, Julien Plu, Can- wen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural lan- guage processing, In Proceedings of the 2020 Con- ference Empirical Methods in Natural Language Processing: System Demonstrations.


Reviews:
[{"review_id": "xAAyxzeCFg", "reviewer": "Reviewer_ZKu3", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "“Although the paper presents a good trial for developing two datasets for Bangala, they couldn't achieve high results in their experiments. This may be due to the low representativeness of their compiled and translated dataset or the unbalance of these datasets. In addition, in their experiments, they experiment on each dataset separately, but they can merge the two datasets which may enhance the results. They need to enrich their datasets with more word analogies and try different experiments to achieve best results.”\n\n- The main goal of our paper is to create a benchmark word analogy dataset (the first of its kind for Bangla) to facilitate future Bangla NLP research and evaluation. Although the results reported in this paper do not show a high degree of performance, we want to emphasize that achieving a high performance is NOT the goal of this work. In fact, achieving high performance is an orthogonal goal, and to achieve this goal, we first need an evaluation benchmark, which does not exist today for Bangla word analogy tasks. To address this issue, we created two different datasets. The first dataset keeps the distinct linguistic characteristics that distinguish Bangla from English and make it unique (see Table 1). The second dataset is a translation of Mikolov's original word analogy dataset that has been meticulously manually filtered to ensure alignment with the Bangla language. Our analyses show that both of these datasets' performance is noticeably limited, indicating significant room for future improvement in Bangla embeddings. However, optimizing for the best results is out of the scope of our paper.\n\n\n\n“you need to clarify what you mean by top-1, top-3, top-5, and top-10. This is used several times later and it is not clear.”\n\n- To clarify, these terms indicate whether the correct answer is present within the top 1, 3, 5, and 10 outputs based on the best cosine similarity scores with other words in the vocabulary.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The real contribution of this paper is the datasets developed by the authors for the Bangala language which is considered one of the low-resource languages although it is spoken by over 300 million native speakers.", "reasons_to_reject": "Although the paper presents a good trial for developing two datasets for Bangala, they couldn't achieve high results in their experiments. This may be due to the low representativeness of their compiled and translated dataset or the unbalance of these datasets. In addition, in their experiments, they experiment on each dataset separately, but they can merge the two datasets which may enhance the results. They need to enrich their datasets with more word analogies and try different experiments to achieve best results.\n ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors of this paper tried to build a benchmark dataset of Bangala word analogies, in addition to translating the word analogies of another English dataset; Mikolov dataset. They used both datasets in evaluating the quality of existing Bangala word embeddings. Unfortunately, their experiments with different state-of-the-art embedding models reveal that current Bangla word embeddings couldn't achieve high accuracy on both datasets. ", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Lines 169-173, you need to clarify what you mean by top-1, top-3, top-5, and top-10. This is used several times later and it is not clear."}, "extra_text": {}}, {"review_id": "lQ1uVb8JTZ", "reviewer": "Reviewer_5yeX", "summary": "", "questions": "- When the dataset is released, please retain mappings to the original Mikolov dataset. \n- Line 139: Are you using an existing Glove embedding or have you trained one?\n- Given that the NLP world is now focused on contextual embeddings and in-context learning capabilities, the utility of a dataset studying linear transformations of word embeddings seems limited. A comment on the current relevance of the dataset would be helpful. ", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- “There is limited learning from the paper about word analogy in Bangla beyond reporting accuracies on a few pre-trained embeddings. It would have been interesting to answer questions like: (a) how does Bangla word analogy compare with English word analogy results? This is possible since the authors have also translated Mikolov's original dataset (b) If we use a model that is also trained on related languages (like IndicBERT or IndicBART) would that improve model performance?”\n\n    - Regarding a) We have shown the comparison of the Bangla word analogy with the translated English word analogy in Figure 2, and we are not sure what other comparisons we are missing here. Could you please provide more clarification here? Regarding b) We appreciate your comment and will be happy to include more models specifically trained on related languages (like IndicBERT or IndicBART) in the camera-ready version (if accepted).\n\n\n- “Details about the bnBART model are missing. The citation (line 71) is wrong. It is not clear what dataset this model is trained on. Line 202 seems to suggest that this model is trained on many NLP tasks - however, there is no paper mentioning this and no information in the git repo.”\n\n    - Thank you for this thorough and specific comment. The exact bnBART model we have reported is from https://github.com/sagorbrur/bntransformer/ (line 202). This model has been pretrained on different inference tasks for the Bangla language, and we have used the BART model from Huggingface (https://github.com/huggingface/transformers) as a tokenizer. Unfortunately, no paper is available to cite on the GitHub link; therefore, we have cited the paper from here (https://github.com/huggingface/transformers) and provided the link to the original trained model on Bengali (line 202). Regarding the concern of the trained dataset, it’s included at the bottom of the GitHub link, also including it here.\n      - Question Answering (https://huggingface.co/sagorsarker/mbert-bengali-tydiqa-qa)\n      - Name Entity Recognition (https://huggingface.co/neuropark/sahajBERT-NER)\n      - Mask Generation (https://huggingface.co/sagorsarker/bangla-bert-base)\n      - Translation (https://huggingface.co/Helsinki-NLP/opus-mt-bn-en)\n      - Text Generation (https://huggingface.co/flax-community/gpt2-bengali)\n\n- “Are you using an existing Glove embedding, or have you trained one?”\n\n    - We are using pretrained Glove embedding, which was trained on Bangla Wikipedia datasets and crawled Bangla news articles. The details are provided in the appendix.\n\n- “Given that the NLP world is now focused on contextual embeddings and in-context learning capabilities, the utility of a dataset studying linear transformations of word embeddings seems limited. A comment on the current relevance of the dataset would be helpful.”\n\n    - This is an excellent comment and, indeed, worth discussing. Note that a word is still the foundation of any language, and word embeddings directly impact the performance of downstream NLP tasks. Although contextual embeddings and in-context learning capabilities are the central focus of NLP now, those contextual embeddings are usually derived from nonlinear transformations of context-free (classic) embeddings and hence, improving classic context-free embeddings will directly/indirectly improve contextual embeddings and in-context learning capabilities, which in turn, will also enable improvement in sentence/document-level and downstream tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Creation of a word analogy dataset for Bangla that is diverse.\n- Translation of an English word analogy dataset can help cross-lingual studies.  \n", "reasons_to_reject": "- There is limited learning from the paper about word analogy in Bangla beyond reporting accuracies on a few pre-trained embeddings. It would have been interesting to answer questions like: (a) how does Bangla word analogy compare with English word analogy results? This is possible since the authors have also translated Mikolov's original dataset (b) If we use a model that is also trained on related languages (like IndicBERT or IndicBART) would that improve model performance?\n- Details about the bnBART model are missing. The citation (line 71) is wrong. It is not clear what dataset this model is trained on. Line 202 seems to suggest that this model is trained on many NLP tasks - however, there is no paper mentioning this and no information in the git repo. ", "questions_to_authors": "- When the dataset is released, please retain mappings to the original Mikolov dataset. \n- Line 139: Are you using an existing Glove embedding or have you trained one?\n- Given that the NLP world is now focused on contextual embeddings and in-context learning capabilities, the utility of a dataset studying linear transformations of word embeddings seems limited. A comment on the current relevance of the dataset would be helpful. ", "paper_topic_and_main_contributions": "In this paper, the authors create a new word analogy dataset for Bangla. A diverse set of relations are covered spanning many synactic and semantic relation categories. They also translate and filter the original Mikolov dataset. They compare various models (classical word embedding models and sentence embeddings models) on this dataset. \n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ous5f2D70m", "reviewer": "Reviewer_Wxve", "summary": "", "questions": "* See reasons to reject section.\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- “The description of how the new dataset was built (first half of section 2) is a bit brief, some details are missing. For example, how were the list of related word pairs acquired, how were two pairs selected to form a group (while the mentioned division-district example is helpful, other groups are not clear), how many annotators (if any) and with what background worked on the dataset, agreement rate, etc.? Additionally, some of the Bangla specific examples in table 1 are not clear for non Bangla speakers, e.g., how are the two words in the Fish-Fish pair different? These specifics should be explained in more detail.”\n\n\n  - Thank you for your comment. Yes, you are right, the highlighted pairs in cyan are hard to distinguish for non-native Bengali speakers, but they are unique to the Bangla language, making our dataset unique for research in Bangla. Due to space limitations, we could not provide more details on data-set creation in the main paper and would be happy to include them in the extra page/appendix in the camera-ready version (if accepted). For now, 3 native Bengali speakers who are also NLP researchers independently first manually proposed different types of relationships for the Bangla Language. Subsequently, they created a list of related word pairs for each relationship. Next, they checked the pairs created by other annotators. If other annotators disagree, we dropped that word pair. Finally, the two words in the Fish-Fish pair are different because one of them is the standard Bangla form (mostly used in formal writing), and the other one is the Colloquial form (used for daily speaking purposes).\n\n- “The authors mention a few already existing resources for Bangla which are mainly related to sentence or document level tasks, such as sentiment classification or summarization. This paper made me wonder about the importance of word level tasks, such as analogy, which is not discussed in the paper. However, I think it would be important to motivate the importance of the task, since most recent NLP work focus on large language models and sentence/document level tasks.”\n\n  - This is an excellent comment and, indeed, worth discussing. Note that a word is still the foundation of any language, and word embeddings directly impact the performance of downstream NLP tasks. Although contextual embeddings and in-context learning capabilities are the central focus of NLP now, those contextual embeddings are usually derived from nonlinear transformations of context-free (classic) embeddings and hence, improving classic context-free embeddings will directly/indirectly improve contextual embeddings and in-context learning capabilities, which in turn, will also enable improvement in sentence/document-level and downstream tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper introduces a new evaluation dataset for the low-resource Bangla\nlanguage which will support other researches in the development of NLP tools for\nthe language.\n* Various word representation methods are evaluated and compared, including\ntheir fine-grained evaluation on the 12 word analogy relationship types.\n* The paper is well written and easy follow.\n", "reasons_to_reject": "* A: The description of how the new dataset was built (first half of section 2) is\na bit brief, some details are missing. For example, how were the list of related\nword pairs acquired, how were two pairs selected to form a group (while the\nmentioned division-district example is helpful, other groups are not clear), how\nmany annotators (if any) and with what background worked on the dataset,\nagreement rate, etc.? Additionally, some of the Bangla specific examples in\ntable 1 are not clear for non Bangla speakers, e.g., how are the two words in\nthe Fish-Fish pair different? These specifics should be explained in more\ndetail.\n* B: The authors mention a few already existing resources for Bangla which are\nmainly related to sentence or document level tasks, such as sentiment\nclassification or summarization. This paper made me wonder about the importance\nof word level tasks, such as analogy, which is not discussed in the paper.\nHowever, I think it would be important to motivate the importance of the task,\nsince most recent NLP work focus on large language models and sentence/document\nlevel tasks.\n", "questions_to_authors": "* See reasons to reject section.\n", "paper_topic_and_main_contributions": "This paper introduces two word analogy datasets for the low-resource Bangla\nlanguage: i) by translating the existing English resource (Mikolov et al., 2013)\nand ii) building a dataset from scratch, keeping Bangla linguistic properties in\nmind. The authors evaluate various classical, such as Word2Vec, and transformers\nbased word embedding models, and find that the latter group of models\noutperforms the former and that the overall performance of the models is low,\nindicating that further work is needed to improve Bangla word representation\nquality. The new dataset, which is the main contribution of the paper, is an\nimportant resource to help the evaluation of future methods on the Bangla\nlanguage.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors create a new word analogy dataset for Bangla. A diverse set of relations are covered spanning many synactic and semantic relation categories. They also translate and filter the original Mikolov dataset. They compare various models (classical word embedding models and sentence embeddings models) on this dataset.

Reasons To Accept:
- The datasets developed by the authors for the Bangala language which is considered one of the low-resource languages although it is spoken by over 300 million native speakers.
- Creation of a word analogy dataset for Bangla that is diverse
- Translation of an English word analogy dataset can help cross-lingual studies.
- Various word representation methods are evaluated and compared, including their fine-grained evaluation on the 12 word analogy relationship types.
- The paper is well written and easy to follow.

Reasons To Reject:
- Although the paper presents a good trial for developing two datasets for Bangala, they couldn't achieve high results in their experiments. 
- There is limited learning from the paper about word analogy in Bangla beyond reporting accuracies on a few pre-trained embeddings.

I agree with the authors when they write (during the rebuttal): "The main goal of our paper is to create a benchmark word analogy dataset (the first of its kind for Bangla) to facilitate future Bangla NLP research and evaluation. Although the results reported in this paper do not show a high degree of performance, we want to emphasize that achieving a high performance is NOT the goal of this work."
Decision: Accept-Main