IndiSocialFT: Multilingual Word Representation for Indian languages in code-mixed environment
Authors: Saurabh Kumar
Ranbir Singh Sanasam
Sukumar Nandi
Venue: EMNLP 2023 Findings (2023)
Keywords: Indian Languages
Multilingual Word Embedding
Code-mixed
Social Media Text

Abstract: The increasing number of Indian language users on the internet necessitates the development of Indian language technologies. In response to this demand, our paper presents a generalized representation vector for diverse text characteristics, including native scripts, transliterated text, multilingual, code-mixed, and social media-related attributes. We gather text from both social media and well-formed sources and utilize the FastText model to create the "IndiSocialFT" embedding. Through intrinsic and extrinsic evaluation methods, we compare IndiSocialFT with three popular pretrained embeddings trained over Indian languages. Our findings show that the proposed embedding surpasses the baselines in most cases and languages, demonstrating its suitability for various NLP applications.

IndiSocialFT: Multilingual Word Representation for Indian languages in code-mixed environment
Saurabh Kumar, Sanasam Ranbir Singh and Sukumar Nandi
Department of Computer Science and Engineering Indian Institute of Technology Guwahati (saurabh1003) ranbir, sukumar)@iitg ac.
Abstract
studies mostly focus on monolingual corpora with limited reach to social media setups. Motivated by these observations, this paper focuses on de- veloping a more generalized representation vector suitable for the text with different characteristics written in native scripts, transliterated text, multi- lingual, code-mixed, and other social media related characteristics.
The increasing number of Indian language users on the internet necessitates the develop- ment of Indian language technologies. In re- sponse to this demand, paper presents generalized representation vector for diverse text characteristies, including native scripts, transliterated text, multilingual, code-mixed, and social media-related attributes. We gather text from both social media and well-formed sources and utilize the FastText model to cre- ate the "IndiSocialFT" embedding. Through intrinsic and extrinsic evaluation methods, compare IndiSocialFT with three popular pre- trained embeddings trained over Indian lan- guages. Our findings show that the proposed embedding surpasses the baselines most cases and languages, demonstrating its suitabil- ity for various NLP applications.
With a target to incorporate diverse characteris- tics of user-generated contents on social media and also well formed text, we consider texts collected from two forms of sources social media text (Twitter, Facebook and YouTube), and well-formed text collected from Samanantar Dataset (Ramesh et al., 2021). Dakshina Dataset (Roark al., 2020), Manipuri-Engliin comparable corpus (Laitonjam and Singh, 2023) and Wikipedia (including 20 schedule languages written in respective native scripts). Due to computing resource constraints at our end, the embedding of the words present in the corpus has been built using FastText (Bo- janowski et al., 2017) model named IndiSocialFT?. To evaluate the quality of obtained embedding. we compare it with three popular publicly avail- able works for Indian languages namely Face- book's FastText (Wiki+CommonCraw)) (Grave et al., 2018), two models by IndicNLPSuite (Kak- wani et al., 2nd0)-In and IndicBERT, and Google's ' MuRIL (Khanuja et al., 2021) using both intrinsic and extrinsic evaluation methods. From various experimental observations, the proposed embedding outperforms all the baseline embedding for almost all the cases and languages.
1 Introduction
Considering the growing interest of developing lan- guage technologies (with Indian languages support) due to vase Indian language base internet users (expected to cross 650 millions',, ), development of pre-trained Indian language text representation suit- able for various NLP applications is becoming an important task. Though India is home to vast linguistic landscapes, encompassing 1,369 ratio- nalized languages and dialects (INDIA, 201 1), the majority of the Indian language related NLP re- search focus primarily on few scheduled languages (only 22 languages are scheduled). Further, in the context of developing Indian language technology, the widespread use of transliterated text, creative acronyms, multilingual, code-mixed text etc. social media should also be taken into account. To mitigate the above challenges, few studies (Kak- wani et al., 2020; Conneau et al., 2020; Khanuja et al., 2021) have initiated the development of word embedding for Indian languages. However, these
Related Work
Facebook FastText project (Grave et al., 2018) has published monolingual word embedding 157 languages. Their FastText-based approach utilized
Wiki+CommonCraw// data to create high-quality
"Statistica-200 (0tssiica-00222anom www. www.ssttttst.comm
Moodel will be made available upon decision of the paper

---

Native Multilingual Code-mixed IIang(IN)
#tokens Sources
FT-WC
17
Common Crawl and Wiki
IndicFT
11
8.8 B
News Crawls
IndicBERT
12
8.8 B
News Crawls
OSCAR corpus, Wiki,
MuRIL
16
11.0
PMIndia, Dakshina
Social Media, Samanantar and Dukshina Dataset, Wiki
IndiSocialFT
20+
11.0
Table Summarization of different model support and corresponding training dataset
word embeddings that encompasses both semantic and syntactic information.
nested comments, resulting in 14.8 million tokens. The dataset also incorporates 0.4 million comments from YouTube, comprising 3.8 million tokens.
Expanding on the growing interest in Indian lan- guage representation, authors of IndicNLPSuite (Kakwani et al., 2020) have developed two differ- ent pre-trained models, IndicFT - a set of 11 mono- lingual pre-trained FastText embedding models, and IndicBERT multilingual ALBERT model trained on their corpora, referred as IndicCorp.
To ensure balanced distribution, we have also included 20 Indian languages in their native scripts from various publicly available datasets. We have incorporated Assamese (as), Bengali (bn), Gujarati (gu), Hindi (hi), Kannada (kn), Malayalam (ml), Marathi (mr), Oriya (or), Pun- jabi (pn), Tamil (ta), and Telugu (te) from the Samanantar dataset (Ramesh al., 2021), Sindhi (sd), Sinhala (si), and Urdu (ur) from the Dak- shina Dataset (Roark al., 2020), and Manipuri (mni) from the Manipuri-English comparable cor- pus (Laitonjam and Singh, 2023). We have also included Sanskrit (sa), Bhojpuri (bho), Nepali (ne), Maithili (mai), and Angika (ang) languages in their native script text, which we have crawled from Wikipedia. Additionally, we have also included En- glish (en) language text in our dataset. These native script datasets have added 0.3 billion sentences the social media dataset. Summarization of dataset is presented in Table 1.
To address the code-mixed cross-lingual transfer tasks, a few works like Conneau et (2020) use transliterated data in training, but limit to including naturally present web crawl data.
A more recent development in this field is the MuRIL (Khanuja et al., 2021), which focuses on multilingual representations for 17 Indian lan- guages. MuRIL is pre-trained multilingual lan- guage model based on the BERT framework, specif- ically built for Indian languages. Its effectiveness has been demonstrated across a wide range of NLP tasks for Indian languages.
3 Dataset
3.1 Data Collection
3.2 Model Pre-training Details
We have crawled tweets, retweets, and replies us- ing Twitter's API, focusing on Indian languages of a three-year duration spanning from 2019 to 2022. Our curated dataset primarily comprises text sourced from Twitter, totaling 0.6 billion tweets, in- cluding quoted retweets and replies. These tweets are filtered by location (India) and amount to 5.5 billion tokens. Additionally, we have collected posts and comments from Facebook profiles of well-known Indian individuals and news media, as well as comments on videos uploaded by popular news and entertainment channels on YouTube. The content from Facebook includes total of 0.8 mil- lion posts, which also encompass comments and
With the curated dataset, we have trained 300- dimensional embeddings model using FastText. We have selected FastText for this task due its ability handle morphologically rich languages, such as Indian languages, by incorporating sub- word information in the form of character n a-gram embeddings during the training process.
We have run the training for 15 epochs, utilized window size of 5, and set a minimum token count of 5 for each instance. These hyperparameters are chosen to optimize the performance of the embed- dings while considering the specific linguistic char- acteristics of the Indian languages in our dataset. The resulting multilingual word embeddings are

---

FT-WC
IndicFT
IndiSocialFT
Lang
FT-WC
IndicFT
IndiSocialFT
Lang
Anto
Syno
Anto
Syno
Anto
Syno
pa 95.53
96.47
95.51
bn 97.57
97.71
97.14
as
12.8
27.4
13.7
22.1
7.5
13.7
bn
15.4
29.6
12.0
14.6
10.1
11.2
or
96.20
98.43
97.23
gu 94.63
99.02
99.51
mni
20.5
31.0
NA
NA
9.7
12.0
15.0
NA
NA
NA
17.2
NA
mr
97.07
99.37
98.74
kn 96.53
97.43
96.36
ta
18.2
41.6
15.4
27.3
12.4
17.3
te 98.08
99.17
99.04
22.0
4.7
NA
NA
13.1
16.4
ml 89.18
92.83
90.50
Table 2: Average rank score Antonyms(Anto) and Synonyms(Syno) pair of different languages by taking top 50 similar words.
ta 95.90
97.26
96.20
Average
95.63
97.52
96.69
Table 4: Accuracy score (in percentage) on IndicGLUE News category testset in different languages.
Lang FT-WC
IndieFT
IndiSocialFT
pa 0.384
0.445
0.683
hi 0.551
0.598
0.664
and English each of set having 100-150 pairs, and performed ranking- intrinsic evaluation. Let Sim4(w) (w;) be the set of the top most similar & words of w;. have used cosine similarity to es- timate similarity between embedding of two words. In the ranking-based approach, given a word pair (w,, wy), the rank of word wy is defined as the posi- tion of word wj in the set Sim; (wx)). As the words in the antonyms pairs and the synonyms pairs are semantically similar to each other, their ranks are expected to be low. The average rank (k: 50) for different language datasets of antonyms and synonyms pairs, considering various pre-trained FastText- based models, is tabulated in Table 2. For most of the languages the average rank result pro- duced by our embedding for pairs of words is lower than the result produced by the other monolingual embeddings. The lower ranked result reveals that embedding is better at representing the seman- tically similar words.
gu 0.521
0.600
0.665
0.544
0.509
0.624
te 0.543
0.578
0.662
ta 0.438
0.422
0.691
ur 0.248
NA
0.624
Average 0.461
0.525
0.659
Table 3: Word Similarity results for different pre-trained embeddings. (a) FT-WC: FastTexx Wikipedia + Com- monCrawl, (b) IndicPT: IndicNI.Suite (c) IndiSo- cialFT.
expected to capture semantic and syntactic similar- ities across the various Indian languages present in the dataset, thereby enabling the development of more effective natural language processing appli- cations tailored this diverse set languages.
4 Evaluation
We have also conducted another word similarity based intrinsic evaluation using the IIT--Hyderabad word similarity dataset (Akhtar et al., 2017). The word similarity assessment examines the relation- ship between the distances of word embedding and the semantic similarity perceived by humans (Wang et al., 2019). This helps to determine how well the word embedding representations capture human-like understanding of similarity, and sup- ports the idea that word's meaning is connected the context it appears in. Higher the word similar- ity score generated by the word embedding model for semantic similar words implics the better word embedding representations. Following the similar-
4.1
Evaluation Texts with Native Scripts
the native script setting, we have compared our embeddings (referred to as IndiSocialFT) with two pre-trained embeddings: one released by the Fast- Text project, trained on Wiki+Commnnnraww (FT- WC) (Grave et al., 2018), and the other released by IndicNLPSuite, IndicFT. Evaluation is done with two sctups intrinsic and extrinsic.
Intrinsic Evaluation: For intrinsic evaluation, we have created set of semantically related word pairs (antonyms and synonyms word pairs) for six languages (five Indian and English) Assamese, Bengali, Manipuri, Urdu(only antonyms), Tamil,

---

Lang
#train
#test
task
#class
handling the semantic and contextual information of text in the native script.
YouTube Cookery Dataset
4.2 Evaluation on Multilingual Code-Mixed Texts
Hinglish(hi-en)
7839
1960
SA
Hinglish-TOP Datase
We have assessed our word embeddings in a code- mixed multilingual environment by conducting var- ious text classification tasks, including (a) senti- ment analysis, (b) offensive language detection, and (c) domain classification, using publicly avail- able code-mixed datasets.
Hinglish(hi-en)
8716
2179
DC
Dravidian-CodeMMx-FIIRe
2021
5549
1388
SA
5
Manglish(ml-en)
16006
4002
OfD
35214
8804
SA
Tanglish(ta-en)
Dataset- For evaluation, we have utilized the Cravidian-CodeMix-Fiii
35133
8784
OfD
2021 (Priyadharshini et al., 2021) dataset, YouTube cookery channels viewer comments in Hinglish (Kaur et al., 2019), and the human-annotated code-switched semantic parsing dataset (Hinglish-TOP Dataset) (Agarwal et al., 2022). The statistics of these datasets are provided in Table 5.
6135
1534
SA
Kanglish(kn-en)
6216
1554
OfD
Table Statistics of different multilingual Code-mixed datasets used for evaluating the model. Here, SA indi- cates Sentiment Analysis, DC indicates Domain Classi- fication and OID indicates Offensive language detection
Classifier Training- In line with Kakwani et al. (2020), we have employed k-NN (k (k= classifier, ensuring that classification performance directly reflects the effectiveness of the embedding space in capturing text semantics and contextual informa- tion.
ity evaluation method outlined by Kakwani et al. (2020), we have assessed the similarity scores on the IITT-Hyderabad word similarity dataset. The word similarity scores are presented in Table 3.Our findings demonstrate that across various languages, our embedding model outperforms the other mod- els in terms of word similarity scores. These higher scores suggest that the word embeddings generated by our model in the native script are more effective capturing human-like perception of similarity.
Results- As our embedding model is trained over a large datasct containing text from social me- dia platforms, it inherently contains the code-mixed and multilingual text. We have compared our classi- fication results with a baseline model trained using IDF vectorizer, FastText, IndicFT, IndicBERT and MuRIL (Khanuja et al., 2021). The text clas- sification results in terms of accuracy and Macro Fl score are presented in Table 6. As our model is trained over the both native and code-mixed text, we can observe that in most of the dataset our model outperform all other models. The higher average accuracy score of 0,691 and higher Macro Fl score of 0.504 for various text classification tasks on the code-mixed multilingual texts, using our embedding model, demonstrates that our model more proficient in managing contextual informa- tion in code-mixed and multilingual settings also.
Extrinsic Evaluation: We have further con- ducted extrinsic evaluation of our model by em- ploying text classification tasks on the IndicGLUE Datasets (Kakwani et al., 2020). IndicGLUE is a comprehensive dataset containing news articles classified into various categories, covering nine different Indian languages, each represented in its native script, offering diverse linguistic landscape for evaluation and analysis. We have adopted the k-NN (k = 4) classifier based evaluation module outlined by Kakwani et al. (2020) to assess our embeddings. As this approach is non-parametric, the classification performance directly illustrates the efficacy of the embedding space in capturing the semantic and contextual information of of cach word in the text. The accuracy score of the trained classifier on IndicGLUE Datasets is presented in Table 4. We have got an average accuracy score of 96.69%. This remarkable accuracy score high- lights the effectiveness of our embedding model in
5 Conclusion and Future Work
In this paper, we have addressed the challenge of representing text in a multilingual code-mixed so- cial media environment by developing FastText- based embedding model. Our model trained on a diverse dataset collected from various social me- dia platforms, supplemented with native script text

---

Dataset
TF-IDF
MuRIL
IndicBERT
IndiSocialFT
acc
F1
acc
F1
acc
F1
acc
F1
hTT)
0.579
0.551
0.652
0.641
0.606
0.591
0.661
0.661
hi-en(TOP))
0.839
0.836
0.864
0.867
0.764
0.758
0.922
0.912
ml-en(SA)
0.144
0.068
0.531
0.465
0.510
0.410
0.539
0,463
ml-en(OfD)
0.893
0.315
0.925
0.398
0.923
0.384
0.926
0.389
ta-en(SA)
0.579
0.208
0.564
0.421
0.538
0.374
0.528
0.427
ta-en(OID)
0.731
0.176
0.734
0.349
0.725
0.321
0.740
0.381
kS-en(SA)
0.487
0.288
0.546
0.440
0.522
0,409
0.556
0.427
kn-en(Off)
0.617
0.222
0.677
0.361
0.656
0.320
0.652
0.368
Average
0.609
0.333
0.686
0.493
0.656
0.446
0.691
0.504
Table 6: Accuracy (acc) and Macro-F1 (FI) score of Text Classification task on different code-mixed dataset for k-NN (k=4) using (a) TF-IDR (b) MuRIL, (c)IndicBERT and, (d) IndiSocialFT.
from publicly available corpora to ensure balance. We have assessed the performance of our trained embeddings in both monolingual native seript and code-mixed multilingual enviromments, employing a range of intrinsic and extrinsic evaluation tech- niques. The results demonstrate the effectiveness of our model's embedding space in capturing the semantic and syntactic information of text in both native monolingual and code-mixed multilingual contexts. This trained embedding model can be utilized address various NLP challenges in the social media context in the Indian region.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised lingul representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Ar- mand Joulin, and Tomas Mikolov. 2018. Learning word vectors 157 languages. In Proceedings of the Fleventh International Conference on Language Resources and Evaluation (LRFC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
As for future work, we plan to further improve the quality of our embeddings by incorporating additional data sources and exploring transformer- based pre-training techniques. We also aim to ex- tend the applicability of our embeddings to wider range of NLP tasks and evaluate their performance in more diverse linguistic scenarios.
INDIA. 2011. Census of india.
Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, Gokul N.C., Avik Bhattacharyya, Mitesh M. Khapra, and Pratyush Kumar. 2020. IndicNLPSuite: Monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for Indian languages. Findings of the Association for Com- putational Linguistics: EMNLP 2020, pages 4948- 4961, Online. Association for Computational Lin- guistics.
References
Anmol Agarwal, Jigar Gupta, Rahul Goel, Shyam Upad- hyay, Pankaj Joshi, and Rengarajan Aravamudhan. 2022. Cst5: Data augmentation for code-switched semantic parsing, arXiv preprint arXiv:2211.075..
Gagandeep Kaur, Abhishek Kaushik, and Shubham Sharma. 2019. Cooking is creating emotion: A study hinglish sentiments of youtube cookery channels using supervise approach. Big Data and Cog- nitive Computing, 3(3).
Syed Sarfaraz Akhtar, Arihant Gupta, Avijit Vajpayee, Arjit Srivastava, and Manish Shrivastava. 2017. Word similarity datasets for Indian languages: An- notation and baseline systems. In Proceedings of the 11th Linguistic Annotation Workshop, pages 91- 94, Valencia, Spain. Association for Computational Linguistics.
Simran Khanuja, Diksha Bansal, Sarvesh Mchtani, Savya Khosla, Atreyee Dey, Balaji Gopalan, Dilip Kumar Margam, Pooja Aggarwal. Rajiv Teja Nagipogu, Shachi Dave, et al. 2021. Muril: Multi- lingual representations for indian languages. arXiv preprint arXiv:2103. 107.00
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Associa- tion for Computational Linguistics, 5:135-146.
Lenin Laitonjam and Sanasam Ranbir Singh. 2023. Manipuri-english comparable corpus for cross-

---

lingual studies. Language Resources and Evaluation. 57(1):377-413,
Ruba Priyadharshini, Bharathi Raja Chakravarthi, Sajeetha Thavareesan, Dhivya Chinnappa, Durairaj Thenmozhi, and Rahul Ponnusamy. 2021. Overview of the dravidiancodemix 2021 shared task on senti- ment detection in tamil, malayalam, and kannada. In Forum for Information Retrieval Evaluation, pages 4-6.
Gowtham Ramesh, Sumanth Doddapaneni, Aravinth Bheemaraj. Mayank Jobanputra, Raghavan AK, Ajitesh Sharma. Sujit Sahoo, Harshita Diddee, Ma- halakshmi J, Divyanshu Kakwani, Navneet Kumar, Aswin Pradeep, Srihari Nagaraj, Kumar Deepak, Vivek Raghavan, Anoop Kunchukuttan, Pratyush Ku- mar, and Mitesh Shantadevi Khapra. 2021. Samanan- tar: The largest publicly available parallel corpora collection for 11 indic languages.
Brian Roark, Lawrence Wolf-Sonkin, Christo Kirov, Sabrina J. Mielke, Cibu Johny, Işin Demiryahin, and Keith Hall. 2020. Processing South Asian languages written in the Latin script: Dakshina dataset. In Proceedings of The 12th Language Resources and Evaluation Conference (LREC), pages 2413-2423.
Bin Wang, Angela Wang, Fenxiao Chen, Yuncheng Wang, and C.-C. Jay Kuo. 2019. Evaluating word embedding models: methods and experimental re- sults. APSIPA Transactions Signal and Informa- tion Processing, 8:e19.


Reviews:
[{"review_id": "fUpcTUedv9", "reviewer": "Reviewer_T51M", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer's constructive feedback and acknowledgment of the strengths in our work in terms of:\n* Extensive Language Coverage,\n* Soundness of Evaluation Methodology, and\n* Performance Improvement.\n\nWe also greatly appreciate the reviewer's insightful comments regarding potential weaknesses in our work. We have considered the feedback and concerns expressed in the rejection reasons, and we would like to offer our rebuttal below: \n\n1. Coverage of Evaluated Languages: We acknowledge the concern that not all languages covered in our model dataset are evaluated. Although our model dataset consists of more than 20 languages, the choice of languages in the evaluation of the paper are aligned to the set of languages present in the publicly available datasets$^{4,5}$ that we use for evaluations, for ease of cross comparisons. We have employed different evaluation approaches (intrinsic as well as extrinsic evaluation considering both well-formed text and social-media text) to cover most of the languages reported in the evaluation in some aspect.  \n\n2. Evaluation Data Size and Statistical Significance: The datasets that we use for evaluating the models are widely used in several other similar studies such as IndicFT$^{4}$ and IndicBERT. The constraint on the dataset size is due to the choice of the evaluation dataset.  \n\n3. Usefulness for General NLP Community and Novelty: We appreciate the acknowledgment of the potential utility of our work for the Indian NLP community. We also acknowledge your concern on the relevance of the model to broader NLP community. However, considering the recent increase in attention of NLP researchers globally $^{1,2,3}$ toward Indian languages due to its  vast linguistics landscape, and unique challenges due to the presence of transliterated text, creative acronyms, multilingual, code-mixed text etc., especially on social media data, we believe that the proposed model will be a value addition to the larger NLP community. \n\n We extend our genuine appreciation for your comprehensive evaluation of our work's strengths and weaknesses.  Your insights have been invaluable in shaping the trajectory of our research. We eagerly welcome further engagement to ensure our work continually uplifts NLP community. \n\nThank you once again for your valuable feedback. \n\nReferences: \n\n[1] Malmasi, Shervin, et al. \"Semeval-2022 task 11: Multilingual complex named entity recognition (multiconer).\" Proceedings of the 16th international workshop on semantic evaluation (SemEval-2022). 2022. \n\n[2] Piskorski, Jakub, et al. \"Semeval-2023 task 3: Detecting the category, the framing, and the persuasion techniques in online news in a multi-lingual setup.\" Proceedings of the the 17th International Workshop on Semantic Evaluation (SemEval-2023). 2023. \n\n[3] Pei, Jiaxin, et al. \"Semeval 2023 task 9: Multilingual tweet intimacy analysis.\" arXiv preprint arXiv:2210.01108 (2022). \n\n[4] Kakwani, Divyanshu, et al. \"IndicNLPSuite: Monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for Indian languages.\" Findings of the Association for Computational Linguistics: EMNLP 2020. 2020. \n\n[5] Akhtar, Syed Sarfaraz, et al. \"Word similarity datasets for Indian languages: Annotation and baseline systems.\" Proceedings of the 11th Linguistic Annotation Workshop. 2017.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The number of languages covered in this work is more than similar previous work.\n\n2. The evaluation methodology is sound.\n\n3. In some experiment, IndiSocialFT can outperform other pre-trained embeddings available for Indian languages.\n", "reasons_to_reject": "1. Not all languages covered in IndiSocialFT are evaluated.\n\n2. Some evaluation data are relatively small. For example, word similary task. On the other hand, several experiments only show slight model performance difference among IndiSocialFT and other models. Significant statistical tests are needed to verify whether the model improvement is achieved.\n\nA news dataset from IndicGLUE may have been saturated to benchmark the model since the baseline performance is already high.\n\n3. Even though this resource can be useful for Indian NLP community, it is difficult to find anything useful from this work for general NLP community. Unfortunately, this work has very limited novelty.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents IndiSocialFT, that is word embedding for 20 Indian languages built using FastText. IndiSocialFT combines the various characteristics of user-generated content on social media as well as well-formed text. The dataset used to construct embeddings includes native scripts, transliterated text, multilingual, code-mixed, and social media-related attributes. IndiSocialFT is evaluated using intrinsic (i.e., word similarity task) and extrinsic methods (i.e., several text classification tasks).\n", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "D7zTYQLh50", "reviewer": "Reviewer_tX8d", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our heartfelt appreciation to the reviewer for their thoughtful evaluation of our work. We greatly appreciate the reviewer's insightful comments regarding potential weaknesses in our work. The feedback provided is invaluable, and we appreciate the opportunity to address the concerns in the following response:  \n\n1. **Clarity on Dataset Composition**: We understand the reviewer's concern regarding the clarity of our dataset composition. We apologize for any ambiguity and offer a comprehensive clarification. The curated dataset primarily comprises of text sourced from Twitter, amounting to 0.6 billion tweets (encompassing quoted retweets and replies), totalling 5.5 billion tokens. Additionally, content from Facebook includes a total of 0.8 million posts, which also encompasses comments and nested comments, resulting in 14.8 million tokens. Furthermore, the dataset incorporates 0.4 million comments from YouTube, comprising 3.8 million tokens. This clarification aims to provide transparency and context to our dataset creation.  \n\n2. **Philosophical Contribution of the New Corpus**: We understand the reviewer's query about the thought process behind our new corpus creation. Our approach emphasizes the unique linguistic dynamics of social media content, encompassing a diverse array of language styles, including code-mixing and creative writing.  These widespread use of transliterated text, creative acronyms, multilingual expressions, code-mixed content, etc., on social media poses a significant challenge for processing data in various NLP downstream tasks such as sentiment analysis, sarcasm detection, fake news detection, and more. Models trained solely on well-formed text are inadequate for addressing the varied characteristics of user-generated content on social media. Our choice to curate a dataset that bridges social media and well-formed text aims to address these challenges and offer versatile representations applicable to diverse text types. \n\n3. **Language-wise Data Composition and Code-Mixing**: We appreciate your feedback about the need for language-wise statistics. We have considered the data crawled from the social media platform as the code-mixed data which is contributing the half of the curated data set.  We have crawled the Twitter based on the Indian location (tweet having India as location) from a span of three years starting from year 2019 to 2022. This crawled text consists of 0.6 billion tweets originating from almost every corner of India. After preprocessing this collection amounts to 59.3GB of text.  We have removed the language tag while preprocessing. We are not able to provide the exact language composition in this short span of time.  The same will be provided in the camera-ready version of the paper.   \n\n4. **Comparison with Other Indian Language Models**: We appreciate your suggestion to include a broader range of language models for comparison, such as IndicBERT. We have incorporated IndicBERT in this comparison and same to reflect in the camera-ready version of the manuscript. \n\n5. **Choice of Evaluation Metric - Accuracy vs. Macro-F1**: We appreciate reviewer's insight regarding the preference for macro-F1 score over accuracy in the multi-class classification setup. The point is well taken, and we have re-evaluated our choice of evaluation metric to ensure it aligns with best practices and provides a more comprehensive assessment of model performance. Performance in terms of Macro-F1 is tabulated below: \n\n|          | Muril | IndicBert | **_IndiSocialFT_** |\n|----------|-------|-----------|--------------|\n| hi-en(YT) | 0.641 |   0.591     | **0.661**        |\n| hi-en(TOP)| 0.867 |   0.758     | **0.912**        |\n| ml-en(SA) | **0.465** |   0.410     | 0.463        |\n| ml-en(OfD)| **0.398** |   0.384     | 0.389        |\n| ta-en(SA) | 0.421 |   0.374     | **0.427**        |\n| ta-en(OfD)| 0.349 |   0.321     | **0.381**        |\n| kn-en(SA) | **0.440** |   0.409     | 0.427        |\n| kn-en(OfD)| 0.361 |   0.320     | **0.368**        |\n\n6. **Clarity on Model Performance Explanation**: The reviewer's feedback regarding the explanation of our model's strong performance is noted. Our primary focus is to develop a more generalized representation vector suitable for text with various characteristics, including those written in native scripts, transliterated text, multiple languages, code-mixed content, and other attributes related to social media. We have conducted a comprehensive evaluation of the language model trained on the curated dataset, considering different contexts, such as texts with native scripts and texts consisting of multilingual code-mixed content. As suggested by state-of-art work$^1$, we have used a non-parametric classifier (k-NN) for the Extrinsic Evaluation, which directly illustrates the efficacy of the embedding space in capturing the semantic and contextual information of each word in the text. Therefore, the model's performance relies entirely on the embeddings generated by the trained model on the curated dataset. We have also conducted Intrinsic evaluation, where the model's performance is dependent on the representations of word embeddings. \n\nOnce again, our heartfelt gratitude for your insights. \n\nReference(s):\n\n[1]. Kakwani, Divyanshu, et al. \"IndicNLPSuite: Monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for Indian languages.\" Findings of the Association for Computational Linguistics: EMNLP 2020. 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(i) Very timely study. Can have broader applicability for downstream tasks. \n(ii) For many tasks, they perform better as compared to famous language models like MuRIL and FastText. \n(iii) Covers more language than MuRIL and other baseline models", "reasons_to_reject": "(i) I understand the only difference between IndiSocialFT and FastText is the new corpora that the authors have created. Yet there is not much clarity on the dataset aspect. Some questions are (a) How many posts and tokens are from Facebook, and similarly, how much data (tokens/sentences) is from News media etc? \n\n(ii) What is the thought process behind this new corpus? Is the only difference that authors preferred social media posts instead of Wikipedia + common crawl data? If my intuition is correct, then there is very little philosophical contribution in this paper. \n\n(iii) Language-wise statistics of data composition are also not given. Further, how much code-mixed data is used is also not clear. \n\n(iv) Since a lot of tasks are included for evaluation, authors need to include other language models trained on Indian language corpora too, e.g. IndicBERT, IndicBART etc. (check this link: https://github.com/AI4Bharat/indicnlp_catalog#PreTrainedLanguageModels)\n\n(v) I don’t understand why authors have preferred to go with accuracy instead of macro-F1 score. This is crucial in the multi-class classification setup. \n\n(vi) So, my takeaway is authors have developed a language model that performs well in some tasks, but there is no clue why it works well. This seems to be a half-baked work.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a new language model called IndiSocialFT, which is a fast-text based model trained on new corpora. The new corpora consist of (i) 0.6 billion tweets filtered by location, (ii) Facebook posts and comments from the profiles of well-known Indian personalities, (iii) News media and (iv) comments on videos posted by popular news and entertainment channels.  The corpora cover 20 Indian languages, as mentioned in section 3.1. Through an intrinsic and extrinsic evaluation, authors show that their model performs better than baseline models, i.e. IndicFT, FastText based on Wikipedia and common crawl and TF-IDF based models.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ExNC5oAds0", "reviewer": "Reviewer_NDo2", "summary": "", "questions": "-", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our sincere appreciation to the reviewer for dedicating time and attention to evaluating our work. Your thoughtful feedback is highly valuable, and we address each point raised in the spirit of enhancing our manuscript: \n\n1. **Missing References** (line 202 - IIT-Hyderabad Word Similarity Dataset): We apologize for the unintended missing of this reference. Thank you for bringing it to our attention. We will promptly include the appropriate reference in the camera-ready version.\n\n2. **Typos, Grammar, Style, and Presentation Improvements**: We apologize for any inconsistencies in our writing and appreciate your vigilance. We will diligently address these instances, including those highlighted (instances of misplaced spaces) at lines 103, 158, 164, and within Table 3 and 6, to ensure a polished and cohesive manuscript. \n\nOnce again, our heartfelt gratitude for your insights.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "IndiSocialFT is more complex in terms of the number of languages addressed and and also number of samples, than the already existing FastText corpuses, which are available for 11 Indian languages: Assamese, Bengali, English, Gujarati, Hindi, Kannada, Malayalam, Marathi, Oriya, Punjabi, Tamil, Telugu.", "reasons_to_reject": "-", "questions_to_authors": "-", "paper_topic_and_main_contributions": "The authors use FastText to generate the \"IndiSocialFT\" embedding by gathering text from mainly social media, in different Indian languages, and also English. The languages are: Assamese (as), Bengali (bn), Gujarati (gu), Hindi (hi), Kannada (kn), Malayalam (ml), Marathi (mr), Oriya (or), Punjabi (pn), Tamil (ta), Telugu (te), Sindhi (sd), Sinhala (si), Urdu (ur), Manipuri (mn), Sanskrit (sn), Bhojpuri (bo), Nepali (ne), Maithili (mt), Angika (ag), and English (en).", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "line 202: IIT-Hyderabard word similarity dataset", "Typos_Grammar_Style_and_Presentation_Improvements": "There are numerous instances of misplaced spaces in writing. For example, at lines: 103, 158, 164, etc. and Table 3, 6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces "IndiSocialFT," a word embedding model developed using FastText for 20 Indian languages. This model is unique as it incorporates diverse textual characteristics from both structured and unstructured sources. The data used to train this embedding is derived from a variety of platforms, including 0.6 billion location-filtered tweets, Facebook posts and comments from notable Indian personalities, news media articles, and comments on popular video channels. The languages covered range from major ones like Hindi and Bengali to regional ones like Maithili and Angika, and also include English. The model's effectiveness is demonstrated through both intrinsic (word similarity) and extrinsic (text classification) evaluation methods. The results indicate that IndiSocialFT outperforms baseline models, including IndicFT and other FastText and TF-IDF based models, showcasing its potential for diverse linguistic applications in the Indian context.

The paper is appreciable for its extensive coverage of languages, surpassing similar previous studies. Specifically, IndiSocialFT addresses more languages than other renowned models like MuRIL and FastText. This comprehensive approach enhances the model's applicability and potential for various downstream tasks. Secondly, the evaluation methodology employed is robust and well-structured. The results from these evaluations are promising, with IndiSocialFT outperforming other pre-trained embeddings designed for Indian languages in several experiments.

However, there's a notable lack of comprehensive evaluation across all languages covered in IndiSocialFT. Some datasets used for evaluation, like the word similarity task, are relatively small. Moreover, the performance difference between IndiSocialFT and other models in several experiments is marginal, necessitating significant statistical tests to validate any claims of improvement. The use of a news dataset from IndicGLUE for benchmarking is also questionable due to the already high baseline performance. The paper lacks detailed language-wise statistics, clarity on the amount of code-mixed data used, and a comparison with other significant models like IndicBERT and IndicBART. The paper seems to lack a clear explanation of why their model performs well.
Decision: Accept-Findings