Late Fusion of Transformers for Sentiment Analysis of Code-Switched Data
Authors: Gagan Sharma
R Chinmay
Raksha Sharma
Venue: EMNLP 2023 Findings (2023)
Keywords: Code-switched
Transformer
Sentiment Analysis
GLUECoS benchmark dataset
Late Fusion
Neural Network
TL;DR: Sentiment Analysis of Code-Switched Data

Abstract: Code-switching is a common phenomenon in multilingual communities and is often used on social media. However, sentiment analysis of code-switched data is a challenging yet less explored area of research. This paper aims to develop a sentiment analysis system for code-switched data. In this paper, we present a novel approach combining two transformers using logits of their output and feeding them to a neural network for classification. We show the efficacy of our approach using two benchmark datasets,  viz., English-Hindi (En-Hi), and English-Spanish (En-Es) availed by Microsoft GLUECoS. Our approach results in an F1 score of 73.66% for En-Es and 61.24% for En-Hi, significantly higher than the best model reported for the GLUECoS benchmark dataset.

Late Fusion of Transformers for Sentiment Analysis of Code-Switched Data
Gagan Sharma and R Chinmay and Raksha Sharma
gagan_ set_s@cc...ittc..ii and r_c@cs. cc and raksha.sharma www.acc.......... is.iitr.ac.i Indian Insitute of Technology Roorke Roorkee, Uttarakhand, India
Abstract
Hindi
Romanized
नपा (new)
naya nya
Code-switching is a common phenomenon in multilingual communities and often used on social media. However, sentiment anal- ysis of code-switched data is a challenging yet less explored area of research. This pa- per aims to develop sentiment analysis sys- tem for code-switched data. In this paper, we present novel approach combining two trans- formers using logits of their output and feeding them to a neural network for classification. show the efficacy of our approach using two benchmark datasets, viz., English- (En- Hi), and English-Spamist (En-Es) availed by Microsoft GLUECoS. Our approach results in an F1 score of 73.66% for En-Es and 61 .4% for En-Hi, significantly higher than the best model reported for the GLUECoS benchmark dataset.
336H (end)
khatam khtm khatm bas bus
(sufficient)
Figure 1: Hindi words with multiple possible Roman- ization
the seven tasks present in the GLUECoS evaluation benchmark, which is aimed at classifying sentiment in a code-switched sentence as positive, negative, or neutral. The task consists of English-Spanish and English-Hindi code-switched language pairs. The English-Spanise task is useful for studying the performance of models in code-switched environ- ment of closely related languages, and the English- Hindi task is for largely different languages.
The code-switched data analysis comes with its challenges. Traditional techniques trained for one language (monolingual NLP models) quickly break down when input is mixed in from another lan- guage. The performance of these models goes down the amount of mixed-language present increases (Aguilar al., 2019). On the other hand, there scarcity of high-quality owie-swithhed datasets (Sitaram al., 2020). Conversations on so- cial media usually use slang, unknown references, mentions, improper grammar, spelling errors, etc. Models trained on correct grammar and spelling will not perform well on such datasets immediately (Dhingra al., 2016).
1 Introduction
Code-Switching refers to using two more lan- guages in single conversation. This linguistic feature is prevalent in multilingual communities all over the world. Multilingual communitics share vast amounts of unstructured code-switched data on social media in text and speech. Sentiment analysis of data shared on social media has proved valu- able in market predictions and analysis (Darapa- neni al., 2022), mental health assessments (Calvo al., 2017), and intelligent agents (Rambocas and Pacheco, 2018). There have been enormous studies conducted on multilingual NLP, but, despite such prevalence code-switching in real life, very few studies have been conducted in the field.
In addition to this, there is also the case of multi- ple Romanization. A code switch statement is usu- ally found in the Roman script because of standard English keyboards' vast popularity and availability. Multiple Romanization leads to words in foreign languages written in English alphabets with vary- ing spellings, shown in English- English-Hindi examples in Figure 1.
Wang et al., (2018) proposed Generalized Lan- guage Evaluation Benchmark (GLUE) to evaluate embedding models on various language understand- ing tasks. Inspired by this, Microsoft Research published GLUECoS (Khanuja al., 2020), an evaluation benchmark on various tasks focused on code-switched text. Sentiment Analysis is one of
The cause of such different spellings is usu- ally the difference in pronunciations which gets

---

reflected in the absence addition of vowels in the Romanized foreign words.
The statement infers joy, encour- agement, likes, sarcasm, or excitement. For example, I am amazed by the work done by the VFX team.
Positive:
Our contribution to the sentiment analysis of code-switched data research includes the perfor- mance improvement from the combination (late fusion) of different pre-trained models without ing ensemble techniques. Since pre-trained models are trained on different datasets and have different architectures, they produce varying results on the same dataset. However, a unison model, which in- cludes multiple pre-trained models, each fine-tuned on whole training data, improves performance com- pared to ensemble other voting techniques, We show that unisom model has improved the weighted-F1 score by 10% for the English-Spanish about 3. 7.73% for the English Hindi sentiment anal- ysis task.
The statement infers sadness, dis- couragement, dislikes, sadism, or indifference. For example, The food service was horrible at our hotel.
Negative:
The statement is mostly factual and does not demonstrate much of an opinion. For example, The Taj Mahal is one of the World's Seven Wonders.
Neutral:
Dinlish-Hiidii
Sentiment En-Es
En-Hi
Positive
433
3202
The rest of the paper is organized as follows. In Section 2, we briefly describe the details regarding the datasets used. Section describes some recent related work. Section 4 describes our approach. Section provides the experimental setup. Section 6 lists the results obtained, and Section 7 concludes the paper.
Negative
379
2319
Neutral
711
4559
Total
1523
10080
Table I: Training Set Distribution
Sentiment
En-Es
En-Hi
2 Dataset Description
Positive 55
399
Negative
283
GLUECoS is a reasonably significant evaluation benchmark for the six code-switched NLP tasks. A high performance sentiment analysis system for this dataset opens the scope for improvement in the performance of other code-switched NLP tasks. In this paper, we present results the GLUECoS benchmark dataset. The dataset is generated by running script that scrapes tweets of some pre- saved ids annotated beforehand (Khanuja et al., 2020). The English-Spanish dataset consists of 1523 tweets for training. 184 for validation, and 232 for testing. The English-Hindi dataset con- sists of 10080 tweets for training, 1260 for valida- tion, and 1261 for testing. We have referred to the best results and method (mBERT) reported on the GLUECoS w for the task. The objective of the paper is improve upon the state-of-the-art method, that is, mBERT adapted for code-switched sentiment analysis.
Neutral
578
Total
184
1260
Table 2: Validation Set Distribution
Literature Survey
Several research and industrial works have been done on sentiment analysis on textual data, espe- cially in the previous decade. Recently, the re- search focused on code-switched data has been growing in popularity since large share of posts available social media or general conversations includes two or more languages (Winata et al., 2022). Before 2019, most sentiment analysis re- search revolved around techniques such as lexical sentiment analysis (like Bag of- Words technique) (Kasthuriarachchy et al., 2014), semi semi-supervised or unsupervised algorithms (Maas et al., 2011 1), volution models (Jianqiang al., 2018), recurrent models (Arras al., 2017). etc. These models failed capture the context of the sentences and needed to more accurate compared to transform- ers (Ran et al., 2018). Transformer-ban models
Every data item is labeled to be expressing a pos- itive, negative, and neutral sentiment. The detailed distribution of the annotations are listed in Table and 2. Below is the definition of assigned labels.

---

replaced these classical models since the availabil- ity of labeled data was the main bottleneck for training neural networks (Devlin et al., 2019). Semi-supervised algorithms such as Label Prop- agation (LP) algorithms (Wang et al., 2015) were used predict sentiments in Code-switched data, using both statistical machine translation and senti- ment information for words. In 2016, Wang et al., (2016) proposed a Bilingual Attention Net- work for code-switched emotion Prediction. They used Bi-LSTM to capture the context of words and both monolingual and bilingual attention vectors to predict the overall emotion of the input sentence. However, transformers (Vaswani et al., 2017) and pre-trained language models produced much better results than traditional techniques (Luo and Wang. 2019), with significantly less training data avail- able. Fine-tuning the models with small amounts of available labeled data produced much better results than earlier methods. Recent research includes the generation of synthetic code-switched data (for bet- ter fine-tuning), intermediate task training (Prasad al., 2021), continued pre-training (Li al., 2022) (masked Language modeling using code-switched texts), late fusion (Mundra al., 2021) (using multiple pre-trained transformer-based models and combining their outputs using some logic to im- prove overall performance). and Data Augmenta- tion (Mundra et al., 2021) both Random Augmen- tation (RA) and Balanced Augmentation (BA), and custom attention models (Li et al., 2021). Other techniques are used to improve sentiment analysis results in code-switched texts specific language this paper, we propose novel upproach to enhance the late fu- sion technique using logits from models instead of the output.
network with three nodes in the output layer, one for each sentiment class. The neural network post- model training handles assigning weights to the results generated by each model. It thus gives more weight to the models as and when one performs better than the other and thus produce a new model which has the better aspects of each of the contribut- ing model. The output neuron the output layer with the highest value determines the sentiment class for the given input.
Logit is described as the output of neuron with- out applying the activation function. Logit for a class is defined using the probability of the class being the predicted output. The mathematical defi- nition of the logit is provided below.
p
Logit(p) = logg (1 p)
clear from the mathematical description that a larger logit value means a larger probability for the said sentiment class.
Experimental Setup
This section provides the experimental details to replicate the work.
5.1 Data Pre-processing
The dataset contains words that written in differ- scripts. For the English-Hindi task, need transliterate words written in the Devanagari script to the Roman script to make it suitable for further processing. This is done using libindic's indic-tran trans (Bhat et al., 2015) library.
pairs or dataset specifications.
5.2 Hyper-parameters of the models
We have implemented BERT (Devlin et al., 2019). mBERT, XLM-R (Conneau et al., 2019) models in our work. For English-Spanish Task, the hyper- parameters are mentioned in Table 3. We used AdamW Optimizer (Loshchilov and Hutter, 2019) for all models. Other values default values from the Hugging-Face library (refer BERT, mBERT, XLM-R).
4 Methodology
Due to the complementary behavior of different sentiment prediction models, we use the late fusion (Colneric and Demar, 2020) technique to combine results from different models as a unison model. The unison model, which includes multiple pre- trained models, each fine-tuned on whole training data, improves performance compared to ensemble other voting techniques. We combine multiple models using their logit prediction for each senti- ment. The description of such a model is given in Figure 2. The logit values for each sentiment class generated from two models are fed to a neural
Model BERT mBERT
gain
epsilon
batch
epoch
5e-5
1e-8
16
4e-5
1e-8
16
5
XLM-R
3e-5
1e-8
16
5
Table 3: Hyper-parameters for En-Es models
Table mentions the parameters for the English- Hindi task. For this task, we have used Hi-BERT

---

Posstive
Model
Nogalive
Nectral Ioge
INPUT
Postive
Model
Nogalive
Neutrol loge
Figure Late Fusion Neural Network
which is an ELECTRA model (Clark et al., 2020) trained on Hindi Wikipedia data and Hindi Com- monCrawl dataset. For tokenization, used Bert- Tokenizer. Other values are the default from the Hugging-Face library (refer Hi-BERT).
Model
F1
mBERT (Khanuja et al., 2020)
63.02%
BERT
65.05%
XLM-R
69.42%
mBERT BERT (Late Fusion)
69.40%
XLM-R BERT (Late Fusion)
70.9%
Model
gain
epsilon
batch
epoch
XLM-R mBERT (Late Fusion)
73.66 %
BERT
5e-5
le-8
32
3
mBERT
4e-5
le-8
32
3
Table 5: FI Scores Obtained for English-Spanish Senti- ment Analysis.
XLM-R
3e-5
le-8
16
5
Hi-BERT
5e-5
le-8
32
3
Table Hyper-paramette for En-Hi models
suitable for multiple languages text (viz., mBERT, XLM-R, Hi-BERT). Tables 5 and 6 present the results of our approach and various other meth- ods for English- and English-Hindi code- switched dataset for sentiment analysis. Khanuja et al., (2020) presented mBERT these datasets' pest-performig model. We further produced re- sults with BERT, XLM-R for both the datasets and Hi- Hi-BERTTEE for the English-Hindi dataset. The results obtained from the late fusion of two models are better than those obtained from individual models. In our late fusion approach, the neural network post-model training handles assigning weights the results generated by each model. It thus gives more weight to the models as and when one per- forms better than the other and thus produce a new model which has the better aspects of each the contributing model. The late fusion technique re- duces the predictions' dispersion, making the end model more robust. We obtained the best F1 score of 73.66% for Saglish-Spanish with the late fusion of XLM-R and mBERT. While for English-Hindi, mBERT and BERT produced the best Fl score of 61.21%.
5.3 Evaluation Metric
We used the weighted Fl score as metric of per- formance measure. The evaluation is done through pull request at the GLUECoS GitHub repository. GitHub Actions run evaluation script and gener- ates the score.
5.4 Late Fusion
The two models generate logtis for each sentiment class. These values are fed to neural network with six input neurons (Figure 2). Further, the model has hidden layer with six hidden neurons and output layer with three output neurons (one for each sentiment category). The neural network is created using the MLPClassifier of SKlearn. Other values are the default from the SKlearn implementation.
6 Results
BERT has proven to be state-of-the-art for many NLP tasks, including sentiment analysis. For senti- ment analysis of code-switched data, a multilingual version of BERT, that is. mBERT (Khanuja et al., 2020), is reported state-of-the-art for the GLUE- CoS benchmark dataset. In this paper, we have tried to present a more accurate system by mak- ing an ensemble multiple BERT-based models
'The majority voting-bused ensemble method results were worse than the independent use of BERT/BEERTXXXM models. Hence, we focused on showing improvement over the individual model.
Frefer MLPClassifier.

---

Model
F1
Named entity recognition on code-switched data: Overview of the cales 2018 shared task.
XLMMR
28.5%
Hi-BERT
55.90%
Leila Arras, Grégoire Montavon, Klaus-Roberi Müller, and Wojeiech Samek. 2017. Explaining recurrent neural network predictions in sentiment analysis.
mBERT (Khanuja et al., 2020)
57.51%
BERT
59.05%
BERT Hi-BERT (Late Fusion)
59.73%
Irshad Ahmad Bhat, Vandan Mujadia, Aniruddha Tam-
mBERT Hi-BERT (Late Fusion)
59.97%
mewar, Riyaz Ahmad Bhat, and Manish Shrivastava, 2015. litt- system submission for fire2014 shared task on transliterated search. In Proceedings of the Forum for Information Retrieval Evaluation, FIRE '14, pages 48 53, New York, NY, USA. ACM.
mBERT BERT (Late Fusion) 61.24%
Table 6: Fl Scores Obtained for English- Hindi Senti- ment Analysis.
Rafael Alejandro Calvo, David N. Milne, M. Sazzad Hussain, and Helen Christensen. 2017. Natural lan- guage processing in mental health applications using non-clinical textst. Natural Language Engineering. 23:649 685.
7 Conclusion
This paper presents novel approach that uses late fusion for sentiment analysis of code-switched data. The results show that mBBETTBERT and XLMR-mBERT fusion models perform the best for Eiglish--indl and English-Spanist Sentiment Analysis tasks, respectively. The tools and models used in our work are largely non-language specific and thus could helpful in Sentiment Analysis of code-switched data of different languages.
Kevin Clark, Minh-Thang Luong, Quoc Lc, and Christopher D. Manning. 2020. Electra: Pre-training text encoders discriminators rather than genera- tors.
Niko Colneric and Janez Demar. 2020. Emotion recog- nition on twitter: Comparative study and training unison model. IEEE Transactions on Affective Com- puting, 1 11:4334444
Limitations
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2019. Unsupervised cross- tross-lingual representation learning at scale. In An- nual Meeting of the Association for Computational Linguistics.
This paper presents results for code-switched data in English-Hindi and English-Spanisee The lan- guages involved in the work are widely spoken lan- guages across the world over social media. How- ever, we have not observed the performance of fusion-based approach for code-switched data where one of the languages low resource lan- guage such as English and Dravidian (Malayalam, Tamil, Kannada). requires further investigation and experimentation.
Narayana Darapaneni, Anwesh Reddy Paduri, Himank Sharma, Milind Manjrekar, Nutan Hindlekar, Pranali Bhagat, Usha Aiyer, and Yogesh Agarwal. 2022. Stock price prediction using sentiment analysis and deep learning for indian markets.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing.
Ethics Statement
The work reported in the paper complies with the ACL Ethics Policy. The output of our work does not have any kind of negative impact on society; it is for the good of society.
Bhuwan Dhingra, Zhong Zhou, Dylan Fitzpatrick, Michael Muchl, and William W. Cohen. 2016. Tweet2vec: Character based distributed representa- tions for social media.
Acknowledgements
Zhao Jianqiang, Gui Xiaolin, and Zhang Xuejun. 2018. Deep convolution neural networks for twitter senti- ment analysis. IEEE Access, 6:23253-23260.
We grateful Dr. Rudra Murthy from IBM Research Bangalore for his guidance throughout the execution of the project.
Buddhika Kasthuriarachehy. Kasun De Zoysa, and Lalith Premaratne. 2014. Enhanced wfrrds model for phrase-level sentiment analysis.
References
Simran Khanuja, Sandipan Dandapat, Anirudh Srini- vasan, Sunayana Sitaram, and Monojit Choudhury. 2020. Gluecos An evaluation benchmark for code- switched nlp.
Gustavo Aguilar, Fahad AIGhamdi, Victor Soto, Mona Diab, Julia Hirschberg, and Thamar Solorio. 2019.

---

Bin Li, Yixuan Weng, Qiya Song, Bin Sun, and Shutao Li. 2022. Continuing pre-trained model with mul- tiple training strategies for emotional classification. In Proceedings of the 12th Workshop on Computa- tional Approaches to Subjectivity, Sentiment & Social Media Analyxis, pages 233--388, Dublin, Ireland. As- sociation for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for nat- ural language understanding, In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. pages 353-355, Brussels, Belgium. Association for Com- putational Linguistics.
Ruifan Li, Hao Chen, Fangxiang Feng. Zhanyu Ma, Xi- aojie Wang, and Eduard Hovy. 2021. Dual graph convolutional networks for aspect- sspect-based sentiment analysis. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 63 19-6922, Online, Association for Computa- tional Linguistics.
Zhongqing Wang, Sophia Lee, Shoushan Li, and Guodong Zhou. 2015. Emotion detection in code- switching texts via bilingual and sentimental infor- mation. In Proceedings of the 53rd Annual Meet- ing of the Association for Computational Linguistics and the 7th International Joint Conference on Natu- ral Language Processing ( (Volume 2: Short Papers), pages 763-768, Beijing, China. Association for Com- putational Linguistics.
Nya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization.
Li Luo and Yue Wang. 2019. Emotionx-hsu: Adopting pre-trained bert for emotion classification. ArXiv, abs/1907.09669.
Zhongqing Wang. Yue Zhang, Sophia Lee, Shoushan Li, and Guodong Zhou. 2016. A bilingual attention network for code-switcher emotion prediction. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1624-1634, Osaka, Japan. The COL- ING 2016 Organizing Committee.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Anmual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA. Association for Computational Lin- guistics.
Winata, Aji, Yong, and Solorio. 2022. The decades progress on code-switching research in nlp: A sys- tematic survey on trends and challenges.
Jay Mundra, Rohan Gupta, and Sagnik Mukherjee. 2021. WASSA @IITK at WASSA 2021: Multi-task learn- ing and transformer finetuning for emotion classifica- tion and empathy prediction. In Proceedings of the Eleventh Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 112-116, Online. Association for Computa- tional Linguistics.
Archiki Prasad, Mohammad Ali Rehan, Shreya Pathak, and Preethi Jyothi. 2021. The effectiveness of intermediate-tase training for code-switched natural language understanding. Proceedings of the 1st Workshop on Mulrilingual Representation Learning. pages 176-190. Punta Cana, Dominican Republic. Association for Computational Linguistics.
Meena Rambocas and Barney Pacheco. 2018. Online sentiment analysis in marketing research: review. Journal of Research in Interactive Marketing, 12.
Li Ran, Lianzhe Zheng, Lin Hailun, Wang Weiping, and Meng Dan. 2018 Text emotion analysis: survey. Journal of Computer Research and Develop- ment, 55:30.
Sunayana Sitaram, Khyathi Raghavi Chandu, Sai Kr- ishna Rallabandi, and Alan W Black. 2020. A survey of code-switched speech and language processing.


Reviews:
[{"review_id": "WnpBZeM0fe", "reviewer": "Reviewer_ZSXs", "summary": "", "questions": "- The reported results do not match the ones in the original CLEUCos paper. Can you elaborate on why this is the case?\n- It seems that you first fine-tune the models on the whole dataset and then utilise the fusion. This was not mentioned clearly in the methodology; can you confirm that?\n- Can you justify why you only experimented with three models?\n- Can you justify using the suggested dataset? Why did you not include other datasets, such as SentiMix?\n- Is there any more recent works that report on the SAIL dataset (the original CLEUCos dataset)?\n", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for reading the paper thoroughly and providing valuable comments. Due to the limitation of 4 pages, we had to confine the information. We want to provide more clarification with the help of the following points through rebuttal. We will incorporate these details in the camera-ready version. We request the reviewer to increase the score of the paper considering the clarification provided in the rebuttal.\n\n1. The reported results do not match the ones in the original GCLEUCos paper. Can you elaborate on why this is the case?\n\n- We have referred to the best results and method (mBERT) reported on the GLUECoS website for the task: https://microsoft.github.io/GLUECoS/. The objective of the paper is to improve upon the SOTA method, that is, mBERT adapted for code-switched sentiment analysis.\n\n2. It seems that you first fine-tune the models on the whole dataset and then utilise the fusion. This was not mentioned clearly in the methodology; can you confirm that?\n\n- Yes, please see the ‘Our contribution’ section at line number – 083.\n\n3. Can you justify why you only experimented with three models?\n\n- BERT has proven to be state-of-the-art for many NLP tasks, including sentiment analysis. For sentiment analysis of code-switched data, a multilingual version of BERT, that is, mBERT (Khanuja et al., 2020), is reported as state-of-the-art for the GLUECoS benchmark dataset. In this paper, we have tried to present a more accurate system by making an ensemble of multiple BERT-based models suitable for multiple languages text  (viz., mBERT, XLM-R, Hi-BERT). We have shown a comparison among six systems (Table 5) for English-Spanish setup and seven systems (Table 6) for English-Hindi setup.  We believe this suffices the list of experiments to support the paper's objective.\n\n4. Can you justify using the suggested dataset? Why did you not include other datasets, such as SentiMix?\n\n- GLUECoS is a reasonably significant evaluation benchmark for the six code-switched NLP tasks. A high performance sentiment anlysis system for this dataset opens the scope for improvement in the performance of other code-switched NLP tasks.\n\n5. Is there any more recent works that report on the SAIL dataset (the original CLEUCos dataset)?\n\n- No, to the best of our knowledge.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper shows that there is an improvement on the used dataset.\n- The approach seems reproducible.", "reasons_to_reject": "- The paper lacks enough empirical evidence of the effectiveness of the proposed model. Since the main contribution of the paper is the model, there should be more experiments to support that. To achieve this, it is suggested to experiment with more models and test on other datasets. \n- The paper reports on a single dataset and compares the performance against one model from 2020.\n- The paper lacks some clarity in explaining the methodology. It is not clear how the models are fine-tuned before using the fusion.\n", "questions_to_authors": "- The reported results do not match the ones in the original CLEUCos paper. Can you elaborate on why this is the case?\n- It seems that you first fine-tune the models on the whole dataset and then utilise the fusion. This was not mentioned clearly in the methodology; can you confirm that?\n- Can you justify why you only experimented with three models?\n- Can you justify using the suggested dataset? Why did you not include other datasets, such as SentiMix?\n- Is there any more recent works that report on the SAIL dataset (the original CLEUCos dataset)?\n", "paper_topic_and_main_contributions": "The paper targets the problem of sentiment analysis for code-switch data. The paper provides an overview of the sentiment analysis task and focuses on the challenges that arise when dealing with code-switch data, such as multiple Romanizations of the same word. The authors aim to improve the performance on sentiment analysis of code switch data. This is done through a model based on late fusion, where multiple models' output is combined. The hypothesis is that since pre-trained language models are trained on varying data, they would provide different results on the same dataset. In this work, the authors fine-tune two models for the task and then utilise their feed their logits to a feed-forward network, which produces the final prediction. The authors experimented with the GLUECos benchmark datasets, specifically the English-Spanich and English-Hindi datasets.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "- SemEval-2020 Task 9: Overview of Sentiment Analysis of Code-Mixed Tweets (Patwa et al., SemEval 2020). Also, the participating teams are relevant.", "Typos_Grammar_Style_and_Presentation_Improvements": "- The methodology section needs improvement. It is not clear that you fine-tune the models separately, and then utilise them for the final model.\n- lines 83, 87 (unison vs unisom)."}, "extra_text": {}}, {"review_id": "kkheNfLsHX", "reviewer": "Reviewer_FBU8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Due to the limitation of 4 pages, we had to confine the information. We want to provide more clarification with the help of the following points through rebuttal. We will incorporate these details in the camera-ready version. We request the reviewer to increase the score of the paper considering the clarification provided in the rebuttal.\n\n1. mBERT is a BERT model suitable to process datasets having multiple language words. GLUECoS is a significant evaluation benchmark for the six code-switched NLP tasks. We have referred to the best results and method (mBERT) reported on the GLUECoS website for the task: https://microsoft.github.io/GLUECoS/. The paper's objective is to improve upon the SOTA method, that is, mBERT adapted for code-switched sentiment analysis. A high-performance sentiment analysis system for this dataset opens the scope for improvement in the performance of other code-switched NLP tasks.\n\n2. Line 080 - 'Since pre-trained models are trained on different datasets and have different architectures, they produce varying results on the same dataset. However, a unison model, which includes multiple pre-trained models, each fine-tuned on whole training data, improves performance compared to ensemble or other voting techniques.'\n\n3. The majority voting-based ensemble method results were worse than the independent use of BERT/mBERT/XLM-R models. Hence, we focused on showing improvement over the individual model.\n\n4. Line 141: 'Transformer-based models replaced these classical models since the availability of labeled data was the main bottleneck for training neural networks (Devlin et al., 2019).' The statement suggests there needed to be more labeled data in the past to utilize neural networks with full capacity. However, at the current time, we can deploy transformers with total capacity due to the availability of a significant amount of labeled datasets.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors presents a clear and direct approach to solve the proposed problem of sentiment analysis in code-switched text. \n- Their approach sounds reproducible and uses a well known public available data. Their methodology seems to be reproducible with proper hyper-parameters discussion.\n- Not just their solution outperforms their baseline, but the proposed approach is easy to understand and well justified for the problem.", "reasons_to_reject": "- The paper sounds like an incremental traditional sentiment analysis paper without anything new. One may expect a solution that could leverage the context of the problem to solve it instead of just combining two models' outputs to outperform a selected baseline.\n- It needs to be clarified why not using ensemble techniques is an advantage. Explain it.\n- In the literature review, you state that data availability was the bottleneck for transformers, but in the following paragraph, you stated that transformers produced better results than traditional techniques with significantly less training data. These sentences are contradictory.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors present a sentiment analysis classifier for code-switched (mixed languages) datasets. They leverage two public datasets, one with English & Spanish and another with English-Hindi. Their approach involves combining two state-of-the-art transformers models by training a third model responsible for merging their output classification, a Late Fusion technique. Their solution presents a better performance in terms of F1 score compared to other single-model approaches.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "-The text would greatly be enriched if you share in the text examples of code-switch cases.\n-\"availed\"?\n- unison vs unisom"}, "extra_text": {}}, {"review_id": "lEKPqa6fbr", "reviewer": "Reviewer_L6it", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for reading the paper in depth. Due to the limitation of 4 pages, we had to confine the information in all sections, including the results section. We want to provide more clarification with the help of the following points through rebuttal. We request the reviewer to increase the score of the paper considering the clarification provided in the rebuttal.\n\n- The majority voting-based ensemble method results were worse than the independent use of BERT/mBERT/XLM-R models. Hence, we focused on showing improvement over the individual model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The concept of the paper is based on an innovative method. They introduce a new fusion technique instead of an ensemble majority voting model.", "reasons_to_reject": "An addition that could be done is a conventional majority voting technique and comparing it with their late fusion technique.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors in this paper presented an approach for Sentiment Analysis of Code-Switched Data. They combined two transformers using logits of their output and feeding them to a neural network for classification. They evaluated the efficacy of their approach using two benchmark datasets of English-Hindi and English-Spanish.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses sentiment analysis for code-switched data, emphasizing the challenges posed by mixed languages. The authors propose a late fusion model that combines the outputs of two fine-tuned transformers and demonstrates improved performance on English-Spanish and English-Hindi datasets.

Reasons to accept:
The paper demonstrates a clear improvement on the dataset used for sentiment analysis.
The approach is presented in a clear and direct manner, making it easy to understand and reproduce.
The methodology is reproducible, utilizing well-known and publicly available data with proper hyperparameter discussion.
The proposed approach not only outperforms the baseline but is also well-justified for the problem at hand.

Reasons to reject:
- Insufficient empirical evidence to support the effectiveness of the proposed model, with a lack of experiments involving additional models and diverse datasets.
- Limited focus on a single dataset and comparison against a single 2020 model, raising concerns about generalizability.
-Lack of clarity in explaining the methodology, particularly in terms of how the models are fine-tuned before fusion.
-The paper is perceived as an incremental traditional sentiment analysis work without introducing novel solutions that leverage context for problem-solving.
-The advantage of not using ensemble techniques is not adequately explained.
-Contradictory statements in the literature review regarding data availability and transformer performance with less training data.
Decision: Accept-Findings