Can Large Language Models understand Code-Mixing?
Authors: Pavan Sai Balaga, NAGASAMUDRAM KARTHIK, Challa Vishwanath, Rudra Murthy, Raksha Sharma


Reviewer paper summaries: The paper explores the capability of LLMs on handling Hindi-English code-mixing tasks on two datasets, i.e., Hindi-English Sentiment Analysis from LINCE (Aguilar et al., 192 2020) and Hindi-English Question Answering data from GLUECoS (Khanuja et al., 2020). The paper compares several LLMs: LLaMA-3 8B and two Indian language LLMs, OpenHaathi and Airavata. The paper and also incorporate fine-tuned PLMs as baselines covering BERT, mBERT, XLMR and MuRIL. The results shows that LLMs still perform worse than PLMs, while Q-LoRA-tuned LLMs can outperforms fully fine-tuning on smaller PLMs.  There are many problems with the works, specifically, the relateds work cover very little body of knowledge in code-switching. More specifically, LLMs on code-switching in various languages have been studied in prior works (Yong et al., 2023; Zhang et al., 2023) while many others have been studied for smaller PLMs and even before PLMs, but none of these paper is covered in related works. Furthermore, the claim and contributions of the paper is unclear, especially since the code-switching capabilities of LLMs on both understanding and generation in multiple tasks and evaluation have been explored in Yong et al. (2023) and Zhang et al (2023).
Summary of strengths: - The paper showcases that Q-LoRA fine-tuned LLMs outperform fully fine-tuned on smaller PLMs, nonetheless it is unclear what are the number of trainable parameters when using Q-LoRA.
Summary of weaknesses: - The claim and contributions of the paper is unclear, especially since the code-switching capabilities of LLMs on both understanding and generation in multiple tasks and evaluation have been explored in Yong et al. (2023) and Zhang et al (2023).
- The related work covers very little body of knowledge in code-switching. Specifically, LLMs on code-switching in various languages have been studied in prior works (Yong et al., 2023; Zhang et al., 2023) while many others have been studied for smaller PLMs and even before PLMs (Solorio and Liu, 2008; Winata et al., 2019; Tan and Joty, 2021; Liu et al., 2020; Adilazuarda et al., 2022; Pratapa et al., 2018; Rizvi et al., 2021, Santy et al., 2021, Appicharla et al., 2021; Gautam et al., 2021; Jawahar et al., 2021; Dowlagar and Mamidi, 2021), but none of these paper is covered in related works. 
- The paper lacks of some details, such as the rank used in Q-LoRA, what are the size of the PLM used in the study (e.g., BERT base/large, XLMR base/large/xl/xxl, etc)
- The paper lacks of novelty in various aspects including method, analysis, and dataset

Reference for Prior works on Code-Switching in LLMs:
1. Yong et al., 2023 => https://aclanthology.org/2023.calcs-1.5
2. Zhang et al., 2023 => https://aclanthology.org/2023.emnlp-main.774
Comments, suggestions and typos: - Please add proper references that are related to code-switching studies in the related work
Overall assessment scores: 1.5