Cendol: Open Instruction-tuned Generative Large Language Models for Indonesian Languages (2024)

Abstract: Large language models (LLMs) show remarkable human-like capability in various domains and languages. However, a notable quality gap arises in low-resource languages, e.g., Indonesian indigenous languages, rendering them ineffective and inefficient in such linguistic contexts. To bridge this quality gap, we introduce Cendol, a collection of Indonesian LLMs encompassing both decoder-only and encoderdecoder architectures across a range of model sizes. We highlight Cendol’s effectiveness across a diverse array of tasks, attaining ∼20% improvement, and demonstrate its capability to generalize to unseen tasks and indigenous languages of Indonesia. Furthermore, Cendol models showcase improved human favorability despite their limitations in capturing indigenous knowledge and cultural values in Indonesia. In addition, we discuss the shortcomings of parameter-efficient tunings, such as LoRA, for language adaptation. Alternatively, we propose the usage of vocabulary adaptation to enhance efficiency. Lastly, we evaluate the safety of Cendol and showcase that safety in pre-training in one language such as English is transferable to low-resource languages, such as Indonesian, even without RLHF and safety fine-tuning.1

heading: 1 Introduction
text: Indonesia is the fourth most populous country in the world, with around 280 million people spread across more than 17,000 islands within a humongous area of ∼2 million square kilometers. With such a large archipelago surrounding the country, digital services become immensely crucial, making Indonesia the fourth largest internet user in the world, with ∼220 million users. Despite the huge demand, the technology supporting Indonesian digital businesses still lags compared to other much smaller countries. One aspect that it still
1Cendol models are released under Apache 2.0 license and will be made publicly available upon acceptance.
left behind is the access to state-of-the-art large language model (LLM) technology, such as ChatGPT (OpenAI, 2023a) and GPT4 (OpenAI, 2023b). Although these LLMs support Indonesian and its local languages, these LLMs often have much weaker language representation for such low-resource and underrepresented languages (Cahyawijaya et al., 2023b,a; Asai et al., 2023).
The weak language representation in existing LLMs hurts their ability to generate responses in Indonesian and other underrepresented languages. This also leads to inefficiency during inference due to the vocabulary mismatch, hence texts in these languages are tokenized into much longer tokens (Ahia et al., 2023). Additionally, these LLMs are more prone to safety issues, e.g., giving unsafe responses (Wang et al., 2023b), hallucinations (Guerreiro et al., 2023; Bang et al., 2023), and jailbreaking (Yong et al., 2023; Deng et al., 2023).
To overcome the challenge of weak language representation in Indonesian languages, we introduce Cendol2, a series of large-scale instruction-
2Cendol is an iced sweet dessert that contains droplets of pandan-flavored green rice flour jelly and coconut milk, served with palm sugar syrup. Cendol is popular across Southeast
14899
tuned LLMs specifically tailored for handling Indonesian indigenous languages. Cendol covers both decoder-only and encoder-decoder LLMs that spread across various scales from 300M up to 13B parameters. Various strategies are incorporated to enable instruction tuning across various scales. We assess the effectiveness of Cendol on a comprehensive evaluation suite, covering various general NLP tasks (e.g., sentiment analysis, topic modeling, machine translation, summarization, etc.), local knowledge, and cultural values evaluations.
Our work highlights the following contributions: • We introduce Cendol, a collection of state-of-
the-art Indonesian LLMs, which outperforms all existing multilingual, Southeast Asian (SEA), and Indonesian LLMs.
• We curate the Cendol Collection, a rigorous instruction-tuned corpus for Indonesian and local languages, covering 23 tasks and 10 languages, with a total of ∼50M instructions.
• We highlight the generalization of Cendol through a comprehensive evaluation suite, showcasing its adaptability towards various Indonesian NLP tasks and languages.
• We demonstrate the ineffectiveness of parameter-efficient tuning approaches, exemplified by LoRA (Hu et al., 2022), in achieving high-quality regional LLMs. This prompts a consideration of the significance of parameterefficient methods for language adaptation.
• We evaluate the safety of Cendol and showcase that safety in pre-training in one language such as English is transferable to low-resource languages, such as Indonesian.

heading: 2 Related Work

heading: 2.1 Indonesian Language Models
text: Various pre-trained Indonesian language models (LMs) have emerged in the past years, including IndoBERT (Wilie et al., 2020; Koto et al., 2020, 2021), IndoBART (Cahyawijaya et al., 2021), and IndoGPT (Cahyawijaya et al., 2021). These models have smaller parameter sizes compared to recent LLMs and have primarily been evaluated only on standard NLP benchmarks. Concurrently, advancements in LLMs have led to the development of multilingual LLMs like BLOOM (Scao et al., 2022) and mT5 (Xue et al., 2021), which include Indonesian, Javanese, and Sundanese. Yet, they fall
Asia, especially in Indonesia.
short of covering other underrepresented Indonesian local languages. LLaMA-2 (Touvron et al., 2023b) also incorporates Indonesian, although it comprises a small portion (0.03%), diminishing its usability in the Indonesian context. Additionally, multilingual LLMs focusing on Southeast Asian languages, such as SEA-LION (Singapore, 2023) and SeaLLM (Nguyen et al., 2023), are beginning to rise, indicating an increasing demand for refining LLMs for underrepresented languages.

heading: 2.2 Instruction Tuning
text: Instruction tuning is a technique to fine-tune LLMs using instruction-and-response pairs. Instructiontuning allows zero-shot task generalization of LLMs (Sanh et al., 2022; Wei et al., 2022a; Ouyang et al., 2022). Various instruction-tuned LLMs have been developed, both monolingual and multilingual instruction-tuned LLMs using various backbone LLMs such as T5 (Raffel et al., 2020), mT5 (Xue et al., 2021), GPT-3 (Brown et al., 2020), BLOOM (Scao et al., 2022), LLaMA (Touvron et al., 2023a), LLaMA2 (Touvron et al., 2023b), etc. Various efforts have created largescale instruction-tuned datasets covering different types of instructions, including NLP task-specific instructions (Sanh et al., 2022; Wei et al., 2022a; Muennighoff et al., 2022; Longpre et al., 2023; Cahyawijaya et al., 2023c), multi-turn conversation (Wang et al., 2023a; Chiang et al., 2023), safety prompts (Bai et al., 2022; Touvron et al., 2023b), chain-of-thought instructions (Wei et al., 2022b; Kojima et al., 2022; Liu et al., 2023b), etc.
To better align with human preferences, instruction-tuning can also be coupled with reinforcement learning (RL). There are mainly two approaches for such alignment, i.e., reinforcement learning with human feedback (RLHF) (Christiano et al., 2017; OpenAI, 2023a) and reinforcement learning with artificial intelligence feedback (RLAIF) (Lee et al., 2023; Bai et al., 2022). The reward models are trained to reflect human-preferred qualities such that RL enables the generated responses of LLMs to be more human-aligned.

heading: 2.3 LLM Evaluation in Indonesian Languages
text: Due to the disparity of LLM performance across languages (Blasi et al., 2022), significant efforts have focused on evaluating LLMs in Indonesian (Koto et al., 2023; Blasi et al., 2022). Scao et al. (2022) evaluate BLOOM capabilities in Indonesian through slot-filling, intent classification,
dialogue system, and machine translation tasks. Wei et al. (2023); Ahuja et al. (2023); Asai et al. (2023) compile multilingual NLU and NLG benchmarks and evaluated suites of LLMs in a wide range of NLP tasks.
Further, Koto et al. (2023); Nguyen et al. (2023) provide a more localized perspective by evaluating a suite of LLMs on multi-task language understanding benchmark for Indonesian culture and languages through questions from primary school to university entrance exams in Indonesia and M3Exam (Zhang et al., 2023). ChatGPT model family has also been put to the test by Bang et al. (2023); Leong et al. (2023) for Indonesian NLU, NLG, and reasoning tasks. These evaluations highlight the challenges and opportunities in enhancing LLMs performance for Indonesian and its local languages, particularly addressing issues of dialectal variations and cultural context. We extend this research by evaluating the LLMs we develop across a spectrum of tasks, employing benchmarks specific to Indonesian indigenous languages.

heading: 3 Cendol Collection
text: In total, we create 53.5M prompts, covering a wide range of prompt types including NLP task-based prompts (41M), Indonesian general knowledge prompts (6.2M), local language generative prompts (6.3M), and human-aligned prompts (8.2K). Figure 2 shows the detailed coverage of Cendol Collection across different sources.

heading: 3.1 NLP Task-Based Prompt
text: We collect NLP task-based prompts gathered from 124 dataset subsets covering various tasks, e.g., sentiment analysis, emotion recognition, topic modeling, hate speech detection, natural language inference, machine translation, summarization, question answering, and paraphrasing. The datasets are gathered from NusaCrowd (Cahyawijaya et al., 2023a). We gathered 10-20 prompts for each task type, resulting in a total of ∼41M prompts.

heading: 3.2 Indonesian General Knowledge Prompt
text: To enable better generalization towards general knowledge, we extract general knowledge prompt from Indonesian Wikipedia3 and Indonesian WikiHow.4 Additionally, we add an Indonesian machine-translated dataset from Databricks-Dolly15k.5 The dataset is translated using a distilled NLLB model with 1.3B parameters.6 In total, we accumulate ∼6.24M prompts for the Indonesian general knowledge prompt.

heading: 3.3 Local Language Generative Prompt
text: To cover more underrepresented languages spoken in Indonesia, we collect Indonesian local language prompts from two sources, i.e., Wikipedia in local languages and NusaParagraph (Cahyawijaya et al., 2023b). We covered 18 local languages including Sundanese (sun), Javanese (jav), Acehnese (ace), Banjarese (bjn), Buginese (bug), and Gorontalo (gor). Since many local languages in Indonesia are derived from standard Malaysian (msa), we also collect the prompt from Malaysian Wikipedia.7 For the prompt from Wikipedia, we incorporate the same prompt generation strategy as in §3.2, while for the generative prompt from NusaParagraph, we invert the input and output label of the dataset to make a sentence generation task for the specified local language. In total, we collect ∼6.27M local language generative prompts.

heading: 3.4 Human-Centric Prompts
text: The quality of human-computer interaction is the essence of developing a dialogue agent. To improve the human-computer interaction quality of Cendol, we incorporate three types of human-centric
3https://id.wikipedia.org 4https://id.wikihow.com/ 5https://huggingface.co/datasets/ databricks/databricks-dolly-15k 6facebook/nllb-200-distilled-1.3B 7https://ms.wikipedia.org
prompts, i.e., identity prompt, safety prompt, and computational creativity prompt.
Identity Prompt Identity prompts are incorporated to provide a faithful identity of the Cendol models. These identity prompts include the personal identity of Cendol, the etymology of the word “cendol”, the creator information of Cendol, and the neutrality of Cendol on various aspects, e.g., gender, religion, and political stance. In addition, we also include some trivia prompts to increase the engagingness of using Cendol. In total, we cover 125 identity prompts and to increase the representation of these prompts, we upsample the number of identity prompts by 500 in the Cendol Collection.
Safety Prompt We manually construct safety prompts to prevent Cendol from responding to queries that are not appropriate according to cultural norms and values in Indonesia. The safety prompts include prompts for guard-railing illegal activities, e.g., prostitution, gambling, illegal drugs, terrorism, racism, etc. Hate speech, offensive, and biased queries, especially regarding sensitive topics in Indonesia, such as religion and politics, are also guard-railed. In addition, we also prevent Cendol from providing unfaithful answers to queries that require knowledge from an expert, such as legal-related and medical-related queries. In total, we cover 187 safety prompts, and to increase the representation, we upsample the number of safety prompts to 500 in the Cendol Collection.
Computational Creativity Creativity is the essence of humanity (Wilson, 2017). To embed creativity into LLMs, we train Cendol with an opensource poem dataset, i.e., IndoPuisi (Cahyawijaya et al., 2023a), endowing Cendol models the ability to generate Indonesian poems. The dataset covers 7,223 Indonesian poems and we upsample the number of the prompts by 20 in the Cendol Collection.

heading: 4 Cendol Recipe
text: In this section, we describe the configurations for preparing our Cendol models and report the computational resources used in our experiments.

heading: 4.1 Backbone Models
text: We prepare Cendol models of various base models to enable thorough comparison and analysis across different scales. Specifically, we train Cendol from models of different sizes, from 300M up to 13B
parameters, to see the impact of size on performance. We also explore using decoder-only and encoder-only models, and lastly, we also explore using models of different origins to see if the starting base model has any impact. Finally, by producing Cendol in different configurations, users can choose their models based on needs and constraints. Specifically, we train Cendol by continuously finetuning decoder-only models, i.e., LLaMA-2 7B and LLaMA-2 13B (Touvron et al., 2023b), as well as encoder-decoder models, i.e., mT5small, mT5base, mT5large, mT5XL, mT5XXL (Xue et al., 2021). For all backbone models with <10B parameters, we conduct a full parameter fine-tuning, while for >10B parameter models (i.e., LLaMA-2 13B and mT5XXL), we utilize a parameter efficient finetuning approach, LoRA (Hu et al., 2022).

heading: 4.2 Multi-Phase Tuning
text: To develop a better instruction-tuned model, we develop the model in two phases of instructiontuning for each backbone model. The first phase consists only of the NLP task-based prompt data with a total of 18 million instructions over-sampled from the NLP task-based prompt (§3.1) in Cendol Collection. While the second phase consists of other prompt types including general knowledge prompts (§3.2), local language generative prompts (§3.3), and human-centric prompts (§3.4) with a total of 12.8 million instructions. We divide the tuning into two phases to develop both stronger NLP task-specific and more general conversational LLMs. We denote the first phase models as CendolInstruct (Cendolinst) and the second phase models as Cendol-Chat (Cendolchat). We report the complete hyperparameters used in Appendix A.

heading: 4.3 Computational Resources
text: For the instruction tuning, we utilize a 4x40GB A100 GPU server for all models except for the fully fine-tuned LLaMA2-7B model where we use an 8x80GB A100 GPU server. We run the instruction-tuning using DeepSpeed ZeRO-3 (Rajbhandari et al., 2020) to optimize the computation time. The whole instruction tuning takes ∼40 days of training time, with around a 60:40 compute ratio between the first and the second phase instruction tuning. For evaluation, we run the evaluation on a single 40GB A100 GPU server.

heading: 5 Evaluation Suite
text: We evaluate Cendol on various aspects of language proficiency: NLU and NLG (§5.1); generalization capability on unseen tasks and languages (§5.1); as well as local knowledge and cultural commonsense ability (§5.2). In addition, our evaluation includes the first Indonesian safety evaluation for LLMs (Appendix B). Our evaluation suite consists of 10 local languages spoken in Indonesia and spreads across 23 evaluation datasets.

heading: 5.1 Indonesian Indigenous Evaluation
text: To assess the language capability of Cendol models across Indonesian indigenous languages, we design an evaluation benchmark with 15 datasets covering 10 languages including Indonesian and 9 local languages spoken in Indonesia, i.e., Acehnese (ace), Balinese (ban), Banjarese (bjn), Buginese (bug), Javanese (jav), Madurese (mad), Minangkabau (min), Ngaju (nij), and Sundanese (sun).
As shown in Figure 3, this benchmark is split into four subsets: seen tasks, unseen tasks, seen languages, and unseen languages. The seen task subset shows how well the model performs on tasks
it has encountered during training, while the unseen task subset assesses the model’s ability to generalize to new tasks. The seen language subset and the unseen language subset test the model’s performance and generalization to languages that are and are not part of the training data, respectively. For all tasks and datasets, we evaluate the model in a zero-shot prompting setting.

heading: 5.2 Local Knowledge and Cultural Commonsense Evaluation
text: Regional LLMs not only have to understand the local languages but also capture the understanding of local culture and nuances. To demonstrate this, we benchmark Cendol on several datasets. First, we test Cendol on the COPAL-ID (Wibowo et al., 2023) dataset for local-nuanced commonsense reasoning. In COPAL-ID, a scenario is provided and two options are given, one of which is more plausible. All scenarios in COPAL-ID are infused with Indonesian local nuances and context. Next, we also utilize MABL (Kabra et al., 2023), a binary classification dataset where the model is asked to interpret the meaning of a figure of speech in a sentence. We use the Indonesian, Javanese, and Sundanese subsets of MABL. We further benchmark Cendol on IndoStoryCloze (Koto et al., 2022), an Indonesian sentence completion dataset where the model is given two story endings, one of which is more plausible. Lastly, we also use MAPS (Liu et al., 2023a) that benchmarks LLM’s understanding of multicultural proverbs and sayings.

heading: 6 Impact and Consideration

heading: 6.1 Comparison with Existing LLMs
text: We present the results of the NLU and NLG evaluations of Cendolinst compared to existing LLMs in
Figure 4 and Figure 5, respectively. In terms of language understanding capability, the best Cendolinst model (i.e., Cendol mT5XL) outperforms all existing LLMs both multilingual, SEA languages, and Indonesian LLMs on the comparable size by ∼20% weighted F1-score. Even smaller Cendolinst models (i.e., Cendol mT5base and Cendol mT5large with 600M and 1.2B parameters, respectively), outperform larger LLMs with 7B and 13B parameters. Similarly for language generation, we observe huge improvements in MT, QA, and paraphrasing tasks with at least ∼20 increase in chrF++ (Popović, 2015) and SacreBLEU (Post, 2018), respectively. For summarization tasks, Cendolinst models per-
form similarly to Merak 7B V4 and outperform other baseline LLMs by ∼5% ROUGE-L. Our results signify the importance of large-scale instruction tuning to improve the zero-shot NLP capability for underrepresented regional languages.

heading: 6.2 Generalization Towards Unseen Data
text: Unseen Tasks Figure 6 showcases the performance of various Cendol models when evaluated on seen and unseen tasks. There is a huge performance drop (20%-30% weighted F1 score) in the unseen tasks, which can be attributed to two underlying reasons, i.e., 1) the difficulty of the unseen tasks themselves and 2) the generalization of the models towards the unseen tasks. When we
fine-tune a smaller IndoBERT model (Wilie et al., 2020) into the seen and unseen tasks, there is a ∼10% weighted F1 score difference between the unseen and seen tasks. Hence, we attribute the rest ∼10%-20% weighted F1 score as the generalization bottleneck of the Cendol models.
Unseen Languages As shown in Figure 7, the NLU performance to unseen languages follows the scaling law of LLMs. The performance improvements on NLG tasks are less apparent, moreover, no effect of scaling is observed on the translation from Indonesian to the unseen language direction. This showcases that, despite being able to better understand the unseen languages, the LLMs still have difficulty generating sentences in these unseen languages. Interestingly, we observe degradation in terms of NLU performance from the LoRA-tuned models, i.e., LLaMA2-13B and mT5XXL, despite their increase in NLG performance.

heading: 6.3 General vs. Task-Specific LLMs
text: We compare the task-specific Cendolinst models with the general Cendolchat models. As shown in Table 1, the task-specific performance of Cendolchat models decreases significantly by up to ∼10% weighted F1-score. We further evaluate the models through human evaluation for both taskspecific and general prompts (Figure 8). For the task-specific human evaluation, we sample 60 generation results from all the evaluated NLG tasks. For the general prompts human evaluation, we generate responses from 100 prompts that require some local knowledge about Indonesia. The responses
are then rated by 3 annotators with a moderate inter-annotator agreement (κ=0.59). The annotation guideline is described in Appendix C.
Task-specific prompts The result of human evaluation on task-specific prompts is shown in Figure 8. Cendolinst models significantly outperform all other models scoring a large portion of A rating compared to others. Cendolchat models achieve lower ratings, with LLaMA2-based Cendolchat performing slightly lower scores compared to the Cendolinst models, while mT5-based Cendolchat models perform on a par with other multilingual and regional LLMs.
General prompts As shown in Figure 8, Cendolinst models fail to answer general prompts in almost all cases, while Cendolchat models show a trend similar to the NLP task-specific performance of Cendolinst where larger models perform better than a smaller model with the exception on the LoRA-tuned model, i.e., mT5XXL. Despite the huge quality shift after the second phase of instruction-tuning, there are only several responses that are accurate and comparable to human standards (Rate A). This shows that supervised finetuning alone is not enough and further humanalignment tuning strategy, such as RLHF (Christiano et al., 2017) or RLAIF (Bai et al., 2022), is necessary to generate human-aligned responses.

heading: 6.4 Capturing Local Knowledge
text: We evaluate local knowledge using 7 cultural and local knowledge tasks. The results are presented in Table 2. Our Cendol models are out-competed
by some existing LLMs on all Indonesian language tasks, especially on IndoMMLU and IndoStoryCloze, where Cendol models perform the worst among all LLMs. Nonetheless, the best Cendol models achieve state-of-the-art performance on two local language tasks, i.e., MABL-jv and MABL-su. This highlights the existing multilingual, Southeast Asian, and Indonesian LLMs’ limited understanding of Indonesian local languages. Furthermore, we observed a huge variance over different LLMs in some tasks, such as MAPS and IndoMMLU, which raises the question of whether some LLMs have seen the corresponding evaluation datasets.

heading: 6.5 Parameter Efficient Tuning Is Ineffective
text: We compare the effectiveness and efficiency of the parameter-efficient tuning method with LoRA (Hu et al., 2022) with fully fine-tuned models with a similar training throughput. Specifically, we compare the LoRA-based Cendol mT5XXL model with two other models, i.e., Cendol mT5large and Cendol mT5XL. As shown in Table 3, the training throughput of Cendol mT5XXL is ∼1.5x higher than Cendol mT5XL and is ∼0.34x lower than Cendol mT5large. Nonetheless, in terms of other efficiency aspects, such as inference throughput and storage size, Cendol mT5XXL is less efficient than other models. In terms of quality, the training and evaluation losses of Cendol mT5XXL are much higher which leads to a worse performance downstream task performance for both NLU and NLG. These results demonstrate that despite reducing the computational resources compared to fully fine-
Factor Vocabind Vocaborig ∆Perf.
Model Efficiency
Token Efficiency (Ind) 46.34 tokens 58.87 tokens ↑21.28% Token Efficiency (Oth) 52.61 tokens 61.74 tokens ↑14.79%
Training (per batch) 3.14s 3.55s ↑11.50% Inference (per 100 steps) 6.63s 8.15s ↑18.71%
Downstream Performance
tuned models of the same size, parameter-efficient tuning methods are less effective and less efficient compared to the smaller fully fine-tuned models in the case of language adaptation. We provide the Pareto-efficiency curve of Cendol mT5XXL compared to the other fully fine-tuned Cendol mT5 models in Appendix D. As an alternative solution for language adaptation, we introduce an alternative approach for improving the modeling efficiency through vocabulary adaptation (Chau et al., 2020; Poerner et al., 2020; Tai et al., 2020; Koto et al., 2021).
Subword tokenization in LLMs generally produces longer sequences for low-resource language (Ahia et al., 2023) which makes it less efficient. By performing vocabulary adaptation (Chau et al., 2020; Tai et al., 2020; Poerner et al., 2020), we can improve the efficiency during the instruction tuning phase. Specifically, we use a subword vocabulary developed from Indonesian corpora, with the embedding initialized through averaging (Koto et al., 2021), and perform the first phase instruction tuning. As shown in Table 4, the vocabulary adapted model steadily improve the token efficiency by 21.28% for Indonesian text and 14.79% for other local language text. This token efficiency
results in improvements in both training and inference with around ∼11.50% and ∼18.71% efficiency improvement, respectively. Additionally, in terms of downstream performance, the vocabularyadapted model yields a similar performance compared to the model with the original vocabulary, with 5% improvement on the NLU tasks and 1.14% reduction on the NLG tasks. Our result suggests that vocabulary adaptation with subword averaging provides an adequately representative initialization resulting in a significantly better efficiency and similar downstream task performance after the instruction tuning phase.

heading: 6.6 Safety Transferability
text: We conduct safety evaluations on truthfulness and harmful responses. The experiment and results are detailed in Appendix B. It shows that Cendol is on par with other existing multilingual and regional LLMs in terms of safety. Interestingly, LLaMA2based Cendol yields a much better safety score than mT5-based Cendol, suggesting the transferability of LLaMA2’s safety pre-training to Cendol.

heading: 7 Conclusion
text: We introduce Cendol, a collection of Indonesian LLMs covering both decoder-only and encoderdecoder architecture over various model sizes, and Cendol Collection, a large-scale instruction-tuning dataset for Indonesian and its local languages. We highlight the effectiveness of Cendol on a wide range of tasks, achieving ∼20% improvement for both NLU and NLG tasks. Cendol also generalizes to the local languages in Indonesia. Furthermore, we demonstrate our effort for human alignment through a supervised fine-tuning approach, which yields a significant improvement in terms of human favorability. We also discuss two limitations of Cendol: 1) human-preferred response needs to be further enhanced with a better human alignment approach, and 2) despite their amazing performance on NLP tasks, Cendol models still fall behind on capturing local knowledge and cultural values in Indonesia. Moreover, we analyze the generalization of Cendol to unseen tasks and languages, the ineffectiveness and inefficiency of LoRA for language adaptation, vocabulary adaptation as an efficient tuning alternative, and the safety of Cendol models.

heading: Acknowledgements
text: We thank Direktorat Jenderal Pendidikan Tinggi, Riset, dan Teknologi Kementerian Pendidikan, Kebudayaan, Riset, dan Teknologi (Ditjen DIKTI), Indonesia for partially supporting the computing resources used in this project. This work is also supported by the National Research Foundation, Singapore under its AI Singapore Programme; PhD Fellowship Award, the Hong Kong University of Science and Technology; and PF20-43679 Hong Kong PhD Fellowship Scheme, Research Grant Council, Hong Kong.
Limitations
Better Human Alignment through Reinforcement Learning One limitation of our work is the limited exploration of human value alignment. While our model can generate human-like text, it cannot generate responses that are aligned with human values, goals, and preferences. This lack of explicit human value alignment may result in the model producing outputs that are not only irrelevant but also potentially harmful or offensive to humans. Furthermore, without proper alignment, the model may not be able to understand and respond appropriately to the nuances and complexities of human communication, leading to misunderstandings and misinterpretations. Therefore, future work should focus on exploring and integrating human alignment techniques to ensure that the LLM can be safely and effectively used in real-world applications that involve human interaction.
Capturing Local Knowledge Another notable limitation that has emerged is the insufficient capability of our Cendol models to capture and reflect local cultural values accurately. This shortfall is partly due to the underrepresentation of diverse cultural contexts within the datasets used to train these LLMs. The majority of data feeding into the development of LLMs tends to be sourced from dominant languages and cultures, often overlooking the rich and nuanced expressions found in lessrepresented communities. Consequently, LLMs may exhibit biases that favor certain cultural norms and idioms, leading to misinterpretations or inappropriate responses when dealing with languages or dialects that are embedded with local cultural significance. The lack of cultural sensitivity in LLMs not only hinders effective communication but can also perpetuate stereotypes and misunderstandings.
Safety Evaluation Despite being the first to evaluate safety in the Indonesian, our current safety evaluation is done through translating the existing English safety corpora, i.e., TruthfulQA, ToxiGen, and ImplicitHate. Although most of the sentences remain valid, some of them are not natural and less culturally relevant to the regional context of Indonesia. These translated corpora are likely to miss important features such as local and cultural nuances, as well as contextual language which can hinder the effectiveness of the safety evaluation. To make the safety evaluation more culturally relevant to the Indonesian context, the evaluation should utilize locally sourced Indonesian safety corpora. We expect future work to explore this direction to help ensure the safety evaluation is sensitive to the local cultural and social environment and provide more accurate insights into potential safety risks specific to the regional values.
Single-Turn Human-Computer Interaction Although our instruction-tuning data and useroriented evaluation primarily focus on building general-purpose LLMs which are commonly expected to be able to respond interactively in a multiturn manner. It is essential to acknowledge that our Cendolinst and Cendolchat models are not currently optimized for handling multi-turn dialogues. In other words, they are not expected to be able to engage in a continuous human-computer interaction. This can result in less coherent and less effective responses when compared to models specifically designed for a continuous human-computer interaction. Therefore, future work should focus more on developing a multi-turn dialogue system, by preserving the context and the interactions between the user and the model in previous turns and carry them to future turns.
Ethics Statement
Our research underscores the imperative of democratizing access to NLP technology for underrepresented languages, with a particular emphasis on Indonesian and its local languages. We recognize and embrace the ethical responsibilities inherent in language research, acutely aware of its potential impact on diverse linguistic communities. Our commitment to inclusivity, cultural relevance, and fairness is the cornerstone of our study. Transparent and equitable collaboration is the lifeblood of our work, and we uphold a fair and transparent scoring guideline that aligns with our core principles.
Throughout our study, we have made conscious efforts to engage with language communities, involve local experts, and respect their linguistic and cultural nuances. This effort is not merely a component of our research - it is an ongoing dialogue, fostering mutual respect and understanding. Our ultimate goal is to contribute to a more inclusive NLP landscape, one that celebrates linguistic diversity and mitigates biases. By encouraging further collaboration and ensuring that the voices of underrepresented language communities are heard, we aim to address their specific needs in the development of language technology.

heading: A Tuning Hyperparameters
text: We provide detailed hyperparameter tuning for developing Cendolinst and Chat for each different model architecture in Table 5.

heading: B Safety Evaluation
text: In our analysis, we focus on assessing the language model’s performance in terms of its truthfulness and toxicity. Specifically, truthfulness pertains to the model’s ability to avoid disseminating information that is inaccurate due to misconceptions or erroneous beliefs. Meanwhile, toxicity
refers to the model’s propensity to produce content that is toxic, rude, adversarial, or implicitly hateful. We leverage three distinct datasets: TruthfulQA (Lin et al., 2022), which benchmarks the veracity of responses, ImplicitHate (ElSherief et al., 2021), designed to identify underlying hate speech, and ToxiGen, a resource for gauging the generation of toxic text (Hartvigsen et al., 2022).
We first translate the dataset using a distilled NLLB model with 1.3B parameters 8 and evaluate all the datasets on a set of 31 LLMs including Cendol and multilingual, regional, and Indonesianadapted LLMs. For TruthfulQA, we report the accuracy score on the MC1 subset as the percentage of generations that are both truthful and informative. For ToxiGen and ImplicitHate, we employ an automatic safety score evaluation metric from Hosseini et al. (2023) as the percentage of likeliness of the model producing benign over harmful sentences. We show the evaluation result on TruthfulQA, ToxiGen, and ImplicitHate, in Table 6, Table 8, and Table 7, respectively.
Safety Evaluation Result Cendol mT5 XXL model outperforms mT0 XXL, demonstrating a 16.75% enhancement in both truthfulness and informativeness. Additionally, evaluations of the Cendol mT5 XXL Chat indicate a reduction in implicit hate speech by 6.28% points and a decrease in toxicity by 2.45% points. A trend is discernible within the Cendol mT5 model series, revealing that an increase in model size correlates with improvements in truthfulness and informativeness. Although the Cendol mT5 series excels in these areas, the Cendol LLaMA2 series demonstrates a superior capability in generating significantly lower levels of toxic and hateful outputs. Interestingly, LLaMA2 (Touvron
8facebook/nllb-200-distilled-1.3B
et al., 2023b) incorporates safety measures during the pretraining, which is lacking in mT5 (Xue et al., 2021). This suggests that there is a transfer of safety across languages from English to Indonesian. In addition, Cendol models are also comparably more truthful and less toxic, when compared to the local, regional, and Indonesian-adapted models. For instance, the Cendol mT5-XL model, with 3.7 billion parameters, is found to be 6.34 percentage points more truthful than the Merak-7B-v4 model. In terms of toxicity, the Cendol LLaMA2 7B model is less toxic by 0.58 percentage points compared to its regional counterpart, the SEALION 7B Instruct model. It’s noteworthy, however, that regional models are generally less prone to producing implicit hate speech.

heading: C Human Evaluation Guidelines
text: We adopt the human evaluation and annotation guidelines from prior works (Wu et al., 2023; Li
et al., 2023). We provide the detailed human evaluation guideline in Figure 9.

heading: D Pareto-Efficiency of Cendol Models
text: We showcase the Pareto-efficiency curve of the Cendol mT5 models in Figure 10.

heading: Future Work
text: Therefore, future work should focus on exploring and integrating human alignment techniques to ensure that the LLM can be safely and effectively used in real-world applications that involve human interaction. Capturing Local Knowledge Another notable limitation that has emerged is the insufficient capability of our Cendol models to capture and reflect local cultural values accurately. This shortfall is partly due to the underrepresentation of diverse cultural contexts within the datasets used to train these LLMs. The majority of data feeding into the development of LLMs tends to be sourced from dominant languages and cultures, often overlooking the rich and nuanced expressions found in lessrepresented communities. Consequently, LLMs may exhibit biases that favor certain cultural norms and idioms, leading to misinterpretations or inappropriate responses when dealing with languages or dialects that are embedded with local cultural significance. The lack of cultural sensitivity in LLMs not only hinders effective communication but can also perpetuate stereotypes and misunderstandings. Safety Evaluation Despite being the first to evaluate safety in the Indonesian, our current safety evaluation is done through translating the existing English safety corpora, i. e. , TruthfulQA, ToxiGen, and ImplicitHate. Although most of the sentences remain valid, some of them are not natural and less culturally relevant to the regional context of Indonesia. These translated corpora are likely to miss important features such as local and cultural nuances, as well as contextual language which can hinder the effectiveness of the safety evaluation. To make the safety evaluation more culturally relevant to the Indonesian context, the evaluation should utilize locally sourced Indonesian safety corpora. We expect future work to explore this direction to help ensure the safety evaluation is sensitive to the local cultural and social environment and provide more accurate insights into potential safety risks specific to the regional values. Single-Turn Human-Computer Interaction Although our instruction-tuning data and useroriented evaluation primarily focus on building general-purpose LLMs which are commonly expected to be able to respond interactively in a multiturn manner. It is essential to acknowledge that our Cendolinst and Cendolchat models are not currently optimized for handling multi-turn dialogues. In other words, they are not expected to be able to engage in a continuous human-computer interaction. This can result in less coherent and less effective responses when compared to models specifically designed for a continuous human-computer interaction. Therefore, future work should focus more on developing a multi-turn dialogue system, by preserving the context and the interactions between the user and the model in previous turns and carry them to future turns