CS-Sum: A Benchmark for Code-Switching Dialogue Summarization and the
  Limits of Large Language Models
Authors: name: Sathya Krishnan Suresh, name: Tanmay Surana, name: Lim Zhi Hao, name: Eng Siong Chng

Code-switching (CS) poses a significant challenge for Large Language Models
(LLMs), yet its comprehensibility remains underexplored in LLMs. We introduce
CS-Sum, to evaluate the comprehensibility of CS by the LLMs through CS dialogue
to English summarization. CS-Sum is the first benchmark for CS dialogue
summarization across Mandarin-English (EN-ZH), Tamil-English (EN-TA), and
Malay-English (EN-MS), with 900-1300 human-annotated dialogues per language
pair. Evaluating ten LLMs, including open and closed-source models, we analyze
performance across few-shot, translate-summarize, and fine-tuning (LoRA, QLoRA
on synthetic data) approaches. Our findings show that though the scores on
automated metrics are high, LLMs make subtle mistakes that alter the complete
meaning of the dialogue. To this end, we introduce 3 most common type of errors
that LLMs make when handling CS input. Error rates vary across CS pairs and
LLMs, with some LLMs showing more frequent errors on certain language pairs,
underscoring the need for specialized training on code-switched data.