Batayan: A Filipino NLP benchmark for evaluating Large Language Models
Authors: Jann Railey Montalan, Jimson Paulo Layacan, David Demitri Africa, Richell Isaiah Flores, Michael T. Lopez II, Theresa Denise Magsajo, Anjanette Cayabyab, William Chandra Tjhi
Accepted at: acl25

BATAYAN: A Filipino NLP benchmark for evaluating Large Language Models

Jann Railey Montalan1,2, Jimson Paulo Layacan3, David Demitri Africa4, Richell Isaiah Flores3, Michael T. Lopez II3, Theresa Denise Magsajo, Anjanette Cayabyab, William Chandra Tjhi1,2

1AI Singapore, 2National University of Singapore,
3Ateneo de Manila University, 4University of Cambridge
Correspondence: railey@aisingapore.org

Abstract

Recent advances in large language models (LLMs) have demonstrated remarkable capabilities on widely benchmarked high-resource languages. However, linguistic nuances of under-resourced languages remain unexplored. We introduce BATAYAN, a holistic Filipino benchmark that systematically evaluates LLMs across three key natural language processing (NLP) competencies: understanding, reasoning, and generation. BATAYAN consolidates eight tasks, three of which have not existed prior for Filipino corpora, covering both Tagalog and code-switched Taglish utterances. Our rigorous, native-speaker-driven adaptation and validation processes ensures fluency and authenticity to the complex morphological and syntactic structures of Filipino, alleviating the pervasive translationese bias in existing Filipino corpora. We report empirical results on a variety of open-source and commercial LLMs, highlighting significant performance gaps that signal the under-representation of Filipino in pre-training corpora, the unique hurdles in modeling Filipino’s rich morphology and construction, and the importance of explicit Filipino language support. Moreover, we discuss the practical challenges encountered in dataset construction and propose principled solutions for building culturally and linguistically-faithful resources in under-represented languages. We also provide a public evaluation suite as a clear foundation for iterative, community-driven progress in Filipino NLP.

1 Introduction

Spurred on by recent advances in computing power, big data, and machine learning, LLMs have come into widespread use due to the emergence of a variety of novel and useful capabilities at scale (Hadi et al., 2023). These capabilities have made user applications based on LLMs some of the fastest-growing consumer applications in human history, but have also rendered previous benchmarks insufficiently difficult and diverse (Wey, 2024; Yang et al., 2023).

Benchmarks are standard datasets used to measure and compare the performance of models against one another. In particular, the increasingly general capabilities of LLMs have necessitated holistic benchmarks which test a diversity of metrics like fairness, truthfulness, and robustness, as well as a variety of tasks like text summarization, casual reasoning, and translation (Yang et al., 2023; Guo et al., 2023; Liu et al., 2023). The vast majority of LLM benchmarks evaluate tasks in English, with non-English works being much fewer (Liu et al., 2021; Son et al., 2024).

Filipino, despite being the national language of the Philippines and being spoken by over 80 million people, remains an under-resourced language. It is under-represented in multilingual LLMs, and existing corpora are domain-specific, non-multilingual (Cajote et al., 2024), and created by non-native speakers (Quakenbush, 2005; Dita et al., 2009). Filipino is often excluded from multilingual LLM benchmarks, suffer from

<table>
  <tr>
    <th>8 Distinct Tasks</th>
    <th>Evaluation with native Filipino prompts</th>
  </tr>
  <tr>
    <td>NL Understanding</td>
    <td>Bibigyan ka ng dalawang pangungusap.<br>Sentence 1, at sentence 2. Tukuyin kung all isalin ang sumusunod na teksto sa language:</td>
  </tr>
  <tr>
    <td>NL Reasoning</td>
    <td>Sumagot gamit ang sumusunod na format:</td>
  </tr>
  <tr>
    <td>NL Generation</td>
    <td>Sumagot gamit ang sumusunod na format:<br>Superman ay isang superman.<br>Paliwan ang $OPTION ng napiling sagot.</td>
  </tr>
</table>

Comprehensive documentation of challenges and considerations in creating Filipino NLP resources

Figure 1: BATAYAN is a benchmark that holistically evaluates LLM capabilities on a wide range of Filipino language tasks. The rigorous curation and adaptation done by native Filipinos preserves the complexity and authenticity of Filipino language usage today.
limited task diversity, or have serious deficiencies in grammatical correctness, completeness, and diversity (Bandarkar et al., 2024).

Moreover, Filipino is a complex language which exhibits a highly complex linguistic structure, particularly in its rich morphological system (Ramos, 2021; Go and Nocon, 2017). Its agglutinative nature allows for extensive use of affixation and creating nuance through prefixes, infixes, suffixes, and circumfixes (Archibald and O’Grady, 2001; Jubilado, 2004). These affixes, combined with root words, allow intricate verb conjugations for marking tense, focus, and mood (Zamar, 2022). Filipino incorporates elements from a wide array of linguistic influences, such as Spanish (Bowen, 1971; Wolff, 2001), Chinese (Gonzales, 2022; Reid, 2018; Chan-Yap, 1980), and Malay (Wardana et al., 2022; Baklanova, 2017), with codeswitching between English and Filipino being common (Bautista, 1991, 2004).

Our threefold contributions attempt to address these challenges. First, we present BATAYAN,1 a holistic Filipino benchmark for evaluating LLMs across 8 distinct natural language processing (NLP) tasks spanning understanding, reasoning, and generation. We place a rigorous emphasis on authenticity to natural Filipino language use through native speaker translation and annotation, addressing translationese bias and other limitations of existing Filipino datasets. BATAYAN is released as part of SEA-HELM, a publicly available evaluation suite2 and leaderboard3 for assessing LLMs across linguistic and reasoning tasks for Southeast Asian languages. Second, we provide extensive evaluation results from a comprehensive set of open-source and commercial LLMs on BATAYAN, from differing sizes (7B–671B parameters) and Filipino-language representation, revealing significant disparities in model performance across different linguistic capabilities in Filipino. Third, we document systematic challenges and methodological considerations in creating high-quality Filipino NLP datasets, particularly highlighting issues in translation fluency, vocabulary adaptation, and the preservation of Filipino’s rich morphological features. These insights provide a practical framework for future development of Filipino language resources.

1The word “batayan” means “basis” or “ground truth”.
2https://github.com/aisingsapore/sea-helm
3https://leaderboard.sea-lion.ai

2 Considerations for Filipino Evaluations

2.1 Language of Evaluation

While Filipino is the official language of the Philippines, it has been argued in literature that the de facto lingua franca is Taglish, the practice of code-switching between English and Tagalog (Go and Gustilo, 2013). Naturally-occurring datasets in Filipino (e.g., mined from social media, recorded interviews) typically contain some code-switching (Bautista, 2004). We defer the problematization of the debated difference between Tagalog and Filipino, and use the two terms interchangeably.

Filipino, while text-rich, is considered under-resourced, lacking the linguistic data, tools, and resources for effective natural language processing (Miranda, 2023b). To address data scarcity, developers take advantage of high-resource languages such as English through translation (Goyal et al., 2022; Doddapaneni et al., 2023). Furthermore, leveraging multilingual datasets is potent not only because it enhances the performance of models trained on limited resources, but also because code-switching and bilingualism is a common phenomenon in the Philippines (Tupas and Martin, 2017).

2.2 Language Design Principles

The design of BATAYAN emphasized linguistic authenticity and representativeness in word choice, sentence structure, and grammar. In terms of language use and word choice, we employed common Filipino and Taglish vocabulary, balancing loanwords with native Filipino terms by prioritizing colloquial usage. This was also done when words had both English and Filipino variants, taking into account spelling and orthographic preferences prevalent among native speakers.

Regarding sentence structure, we prioritized sentences with attention to natural syntax and the choice of ayos (sentence arrangement), namely direct (karaniwang ayos, KA, lit. usual order) or inverted (di-karaniwang ayos, DKA, lit. unusual order) forms (Tanawan et al., 2008). In KA, Filipino constructions follow the typical predicate-initial word order (Malicsi, 2013). In contrast, DKA, which is also referred to linguistically as an ay-inversion, is a type of construction in Filipino where non-predicative constituents (such as simuno, lit. grammatical subject) are “fronted” or shifted to precede the predicate, marking them as the “topic” of the
<table>
  <tr>
    <th>Comp.</th>
    <th>Task</th>
    <th>Dataset</th>
    <th># test samples</th>
    <th>Target<sup>1</sup></th>
    <th>Language</th>
    <th>Source<sup>2</sup></th>
    <th>Adaptation</th>
    <th>Our contribution</th>
  </tr>
  <tr>
    <td rowspan="4">NLU</td>
    <td>PI</td>
    <td>PAWS (Zhang et al., 2019)</td>
    <td>2,000</td>
    <td>label (2)</td>
    <td>English</td>
    <td>English-sourced</td>
    <td>Native translation</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>Balbalele (Bandarkar et al., 2024)</td>
    <td>900</td>
    <td>span (4)</td>
    <td>Filipino</td>
    <td>English-adapted</td>
    <td>Native re-translation</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>SA</td>
    <td>PH Elections (Cabasag et al., 2019)</td>
    <td>5,160</td>
    <td>label (3)</td>
    <td>Taglish</td>
    <td>Natively-sourced</td>
    <td>No adaptation</td>
    <td></td>
  </tr>
  <tr>
    <td>TD</td>
    <td>PH Elections (Cabasag et al., 2019)</td>
    <td>5,160</td>
    <td>label (2)</td>
    <td>Taglish</td>
    <td>Natively-sourced</td>
    <td>No adaptation</td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2">NLR</td>
    <td>CR</td>
    <td>Balanced COPA (Kavumba et al., 2019)</td>
    <td>500</td>
    <td>label (2)</td>
    <td>English</td>
    <td>English-sourced</td>
    <td>Native translation</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>NLI</td>
    <td>XNLI (Conneau et al., 2018)</td>
    <td>5,010</td>
    <td>label (3)</td>
    <td>English</td>
    <td>English-sourced</td>
    <td>Native translation</td>
    <td>✓</td>
  </tr>
  <tr>
    <td rowspan="2">NLG</td>
    <td>AS</td>
    <td>XL-Sum (Hasan et al., 2021)</td>
    <td>11,535</td>
    <td>summary</td>
    <td>English</td>
    <td>English-sourced</td>
    <td>Native translation</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>MT</td>
    <td>FLORES 200 (NLLB Team et al., 2022)</td>
    <td>1,012</td>
    <td>translation</td>
    <td>Filipino</td>
    <td>English-adapted</td>
    <td>Native re-translation</td>
    <td>✓</td>
  </tr>
</table>

<sup>1</sup> Number of options are shown in parenthesis
<sup>2</sup> English-adapted: previously translated from English; English-sourced: originally in English; Natively-sourced: originally in Taglish

Table 1: Source datasets for each task in BATAYAN.

sentence (Kroeger, 1993). Inverted constructions are often used in formal settings, and could therefore be deemed unnatural in most situations (Pizarro-Guevara, 2010). This characteristic of Filipino is of interest due to our observation of machine translations preferring DKA (see Section 4.1); hence, the further need for native re-translations with a preference for KA.

2.3 Related Datasets and Benchmarks

Previous work on Filipino language model evaluation has been largely fragmented. Researchers have made efforts in developing datasets for Filipino language tasks such as fake news detection (Cruz et al., 2020), named entity recognition (Miranda, 2023a), natural language inference (Cruz et al., 2021), sentiment analysis (Villavicencio et al., 2021), and others. However, the field lacks a unified, comprehensive benchmark for systematic evaluation of LLMs across different linguistic dimensions in Filipino.

Unlike existing Southeast Asian benchmarks that either overlook Filipino or treat it through machine translation, BATAYAN is the first holistic, native-speaker-driven evaluation suite tailored specifically for Filipino NLP. For instance, BHASA provides a multilingual SEA benchmark covering Indonesian, Tamil, Thai, and Vietnamese, but it does not include Filipino (Leong et al., 2023). Similarly, IndoNLU focuses exclusively on Indonesian understanding tasks and lacks any generative or reasoning evaluations (Wilie et al., 2020). SeaExam and SeaBench introduce local exam and open-ended Vietnamese, Thai, and Indonesian queries but exclude Filipino entirely (Liu et al., 2025), and SailCompass evaluates robustness for SEA languages without any Filipino components (Guo et al., 2024). SeaEval provides a broad multilingual and multicultural benchmark across 29 datasets for Southeast Asian languages, including Filipino, but still relies primarily on machine translations and generic multilingual prompts (Wang et al., 2024).

In contrast, BATAYAN integrates eight tasks—including three novel Filipino-centric ones—grounded in native translations and rigorous acceptability judgments, ensuring fluency, authenticity, and sensitivity to Filipino’s complex morphological and code-switching patterns.

3 Task and Dataset Curation

3.1 Task Selection

Through BATAYAN, we significantly expand the scope of Filipino LLM evaluation by introducing tasks that assess a gamut of NLP competencies: understanding (NLU), generation (NLG), and reasoning (NLR). Our Filipino benchmark incorporates machine translation (MT), natural language inference (NLI), question answering (QA), sentiment analysis (SA), and toxicity detection (TD). We also release three novel tasks that have not been previously explored in the Filipino context, namely abstractive summarization (AS), causal reasoning (CR), and paraphrase identification (PI).

Our benchmark design and selection of tasks are informed by established multilingual evaluation frameworks, particularly BHASA (Leong et al., 2023) and XTREME (Hu et al., 2020).

3.2 Dataset Selection

To construct BATAYAN, we curated open-source corpora with clear provenance when possible, prioritizing datasets that exhibit authentic language use across various domains such as social media, news articles, and other publicly available texts. We identified existing Filipino-language datasets for MT, QA, SA, and TD. Meanwhile, the AS, CR, NLI, and PI tasks lacked an expert-annotated,
natively-sourced Filipino corpus. Given the scarcity of high-quality Filipino corpora for these tasks, we chose to natively translate and adapt existing English corpora into Filipino following the methods detailed in Section 3.3. In translation, we deliberately “naturalized” the language use by preferring KA, colloquial terms, and other adaptations to preserve native Filipino speech.

Table 1 outlines key attributes of each dataset, such as output type, domain, language, source, and any adaptations that we applied to ensure alignment with natural Filipino language construction. More detailed statistics can be found in Appendix A.

3.3 Dataset Adaptation and Annotation
For each task, entries were filtered by length (20–2000 characters), then randomly sampled \(5n\) entries for a target size of \(n\), maintaining balanced class distributions.

Native speaker acceptability judgments for quality filtering. For tasks that were natively-sourced (such as TD) or were previously translated from English to Filipino (such as MT, QA), we filtered entries by text quality. Our evaluation of language quality was driven by acceptability judgments from native Filipino speakers4 in groups of three so that context and natural language variation are appropriately considered. Relying on standardized automated metrics would be less effective for languages with rich morphological and syntactic traits (Rivera-Trigueros, 2022) such as Filipino.

Annotation for SA task. We chose to repurpose the natively-sourced TD dataset due to the authenticity and diversity of registers in the dataset. Three native speakers annotated the Philippine election-related tweets (Cabasag et al., 2019) dataset for the SA task with three sentiment polarities: positive, negative, and neutral. Inter-annotator agreement was calculated using Cohen’s kappa (Cohen, 1960), which is the difference between observed agreement and the probability of chance agreement over the probability that the raters did not agree by chance, and Krippendorff’s alpha (Krippendorff, 2018), which is 1 minus the ratio of observed disagreement by raters and expected chance disagreement). The resulting Cohen’s kappa of 0.8202 and Krippendorf’s alpha of 0.8268 indicate substantial agreement. For both TD and SA tasks, we chose to maintain the naturally-occurring language to capture the nuances of code-switched Taglish usage prevalent in social media in the Philippines.

Authoritative translation guidelines. For datasets that were adapted into Filipino, a critical aspect of dataset design was ensuring compliance with official guidelines set by the Komisyon sa Wikang Filipino (Commission on the Filipino Language) of the Philippines. In translation, we followed prescribed principles aimed at preserving the source material’s intent while adjusting for cultural and contextual relevance (Almario, 2016) and effective writing (Almario, 2014). Translations were performed by native Filipino speakers in the identified lingua franca, whenever appropriate, to maintain semantic equivalence without resorting to overly literal phrasing.

Native re-translation of previously translated datasets. We observed that the QA and MT datasets were of subpar quality and suffered from issues with inaccuracies and translationese (Gellerstam, 1986; Riley et al., 2020) such as the use of unnaturally formal language or inappropriate word choices. As such, these were re-translated to enhance contextual and linguistic accuracy.

Native translation of English datasets. For multilingual tasks that had no equivalent Filipino subsets (such as AS, CR, NLI, and PI), we generated initial translations using Helsinki NLP’s

![Dataset curation process for BATAYAN.](page_172_120_1097_627.png)

Figure 2: Dataset curation process for BATAYAN.

4Demographics: Working professionals, research faculty, and university students, aged between 21–30 years old. Native bilingual speakers and code-switchers of Filipino and English. Lived in the Philippines for a majority of their lives.
OPUS model. This model was selected over other off-the-shelf tools, such as Google Translate, as it demonstrated superior performance in terms of Filipino translation quality and accuracy as tested by the authors. These translations were then manually revised by native Filipino speakers to generate high-quality and fluent translations. Our native-speaker-driven evaluation process focused on cultural and linguistic relevance to ensure that the translated texts resonate with Filipino contexts.

3.4 Dataset Evaluation

Human evaluation for data quality. Each sample in each dataset underwent further quality assessment by teams of three raters against three binary criteria: completeness, fluency, and sensibility. Samples were assessed as complete only if they contained well-formed sentences, excluding standalone dependent clauses or fragmentary headers and titles lacking complete meaning. Fluency was evaluated on native-like constructions based on multiple factors including appropriate verb conjugations (e.g., avoiding constructions like “*Nagpapasalamatan ako sa iyo”, lit. “*I give thanks to you.”), natural word choices (avoiding awkward phrases like “*Si Pacquiao ang kamao ng bansa” instead of “Si Pacquiao ang pambansang kamao”, lit.: “Pacquiao is the nation’s fist”), and preference for KA sentence structure where contextually appropriate. Samples were assessed as sensible if relevant to the task, screening for confounding factors language capability (e.g., incorrect answers in QA mislabeled as correct or requiring re-translation).

Additional criteria for abstractive summarization. We employed additional criteria based on previous work (Leong et al., 2023) to assess the appropriateness of the summary in reference to the content of the provided passage. We scored samples on binary criteria (completeness, fluency, and faithfulness) and a Likert-scaled (0-3) criteria (relevance, fluency, and coherence of the summary), with three independent raters per entry. For faithfulness, raters evaluated the factual consistency between the summary and source article, with particular attention to fact preservation and penalization of hallucinated content. For relevance, raters evaluated both coverage of essential information and information filtering, penalizing summaries containing redundancies or excess details. For fluency, raters assessed for proper formatting, appropriate capitalization, and grammatical construction, with higher scores awarded to summaries exhibiting natural Filipino language patterns. For coherence, raters assessed how effectively information flowed between sentences, with higher scores given to summaries that built logically from sentence to sentence to create a cohesive narrative about the topic.

<table>
  <tr>
    <th>Competency</th>
    <th>Task</th>
    <th>Label</th>
    <th># test samples</th>
  </tr>
  <tr>
    <td rowspan="6">NLU</td>
    <td>PI</td>
    <td>True</td>
    <td>200</td>
  </tr>
  <tr>
    <td></td>
    <td>False</td>
    <td>200</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>span</td>
    <td>100</td>
  </tr>
  <tr>
    <td>SA</td>
    <td>Negative (Negativo)</td>
    <td>200</td>
  </tr>
  <tr>
    <td></td>
    <td>Neutral (Neutral)</td>
    <td>200</td>
  </tr>
  <tr>
    <td></td>
    <td>Positive (Positivo)</td>
    <td>200</td>
  </tr>
  <tr>
    <td rowspan="2">TD</td>
    <td>Clean (<i>Malinis</i>)</td>
    <td></td>
    <td>200</td>
  </tr>
  <tr>
    <td>Toxic (<i>Mapoot</i>)</td>
    <td></td>
    <td>200</td>
  </tr>
  <tr>
    <td rowspan="4">NLR</td>
    <td>CR</td>
    <td>Cause (<i>Sunuh</i>)</td>
    <td>200</td>
  </tr>
  <tr>
    <td></td>
    <td>Effect (<i>Bunga</i>)</td>
    <td>200</td>
  </tr>
  <tr>
    <td>NLI</td>
    <td>Contradiction</td>
    <td>200</td>
  </tr>
  <tr>
    <td></td>
    <td>Entailment</td>
    <td>200</td>
  </tr>
  <tr>
    <td></td>
    <td>Neutral</td>
    <td>200</td>
  </tr>
  <tr>
    <td rowspan="3">NLG</td>
    <td>AS</td>
    <td>summary</td>
    <td>100</td>
  </tr>
  <tr>
    <td>MT</td>
    <td>eng→tgl translation</td>
    <td>600</td>
  </tr>
  <tr>
    <td></td>
    <td>tgl→eng translation</td>
    <td>600</td>
  </tr>
  <tr>
    <td colspan="3">Total</td>
    <td>3,800</td>
  </tr>
</table>

Table 2: Class distribution per task in BATAYAN.

3.5 Dataset Statistics

For all datasets except AS, only entries that were unanimously rated as demonstrating all 3/3 binary criteria were sampled for the final test splits. For AS, only entries that received a unanimous 3/3 binary criteria assessment and at least and average score of 2 for the scaled criteria were considered for inclusion, ensuring strict maintenance of factual integrity. In cases where initial sampling yielded insufficient qualifying entries, deficient samples were carefully corrected by authors through targeted improvements to grammar, translation of English passages, or other identified issues while maintaining authentic Filipino language patterns.

Joint agreement on our evaluation criteria is presented in Appendix C. We present joint agreement over traditional metrics such as Cohen’s kappa and Krippendorff’s alpha because metrics that use chance correction are less suitable for binary categorical tasks (Powers, 2012) and classification tasks in general (Delgado and Tibau, 2019; Feng and Zhao, 2016).

Overall, BATAYAN provides 8 distinct tasks with 3,800 test instances. Table 2 provides the distribution of classes for each dataset, and more detailed statistics can be found in Appendix A. We also provided example splits composed of 5 entries that serve as exemplars for few-shot prompting.
4 Challenges with Developing a Filipino Benchmark

This section discusses issues in creating BATAYAN, shedding light on issues that future researchers may encounter in creating new Filipino datasets or adapting previous ones.

4.1 Issues with English-adapted and English-sourced Data

Creating more relevant summaries for XL-Sum. Previous work has noted that a significant portion of reference summaries in the XL-Sum dataset is highly abstractive, demonstrates factual errors, or contains information not mentioned in the provided articles (Guo et al., 2022), putting to question its factuality and validity. To address this, we developed new Filipino summaries for each article. We ensured relevance and fluency by including only the most important information that can be directly lifted from the article using natural-sounding and grammatically-correct constructions.

Limitations in adapting vocabulary. Adapting the English tasks to Filipino revealed issues in maintaining the intelligibility of the text. Technical terms that exist in English were difficult to translate into Filipino, with several of them having no direct translations. For example, an instance from Belebele included the term “rule of thirds”, which describes a specific photography technique. Originally, the translation of this phrase in the Tagalog subset of Belebele was “tuntunin ng mga sangkatlo” (lit. “rule of thirds”), which does not make sense in Filipino and does not convey the intended meaning. For jargon such as this, we chose to keep the original English phrasing.

English idiomatic expressions were also prevalent and required a more flexible approach in adaptation to ensure the integrity of meaning. For example, one premise from Balanced COPA was, “The criminal turned himself in”, using the idiomatic expression “turned himself in” to mean “surrendered”. We chose to translate this into Filipino as, “Isinuko ng kriminal ang kanyang sarili” (lit.: “The criminal surrendered himself”).

Moreover, the presence of English words that have homonyms added to the complexity of translating English sentences to Filipino. For example, some sentences from PAWS contained the adjective “right”, which can refer to either the direction (in Filipino: “kanan”) or an assertion of the correctness of the object it is modifying (in Filipino: “tama”). In such cases, we inferred the most probable meaning of the words from the context of the entire passage and then identified the corresponding Filipino word to be incorporated in our Filipino translation.

(Dis)fluency in expert and machine translations. Our rigorous review of existing English datasets with machine and expert-guided translations in Filipino revealed weaknesses in their adaptation. The authors found that the automatic translations were unnatural, disfluent, and showed characteristics of translationese despite being grammatically correct. Notably, we observed that these initial translations demonstrated an unusual preference with using ay-inversion, and used Filipino terms that were synonymous to their English counterparts but pragmatically-awkward. Hence, we applied re-translation to ensure that the samples sounded more native and natural.

For example, one passage from the English subset of Belebele was, “The CCTV would certainly send a strong signal...”. The original Tagalog (machine) translation was, “Ang CCTV ay tiyak na magpapadala ng malakas na hudyat...” The term “magpapadala” here means “to send” in the sense of transporting a thing from one place to another, while the term “hudyat” implies a “starting sign”. It also uses the unnatural DKA construction or ay-inversion. Within the context of the original passage, a more fluent (human) translation would be, “Tiyak na maghahatid ang CCTV ng malakas na pahiwatig...”, where “maghahatid” means “to bring about” and “pahiwatig” conveys a sense of “reminder” or “warning”, and the sentence follows the usual KA construction.

Inconsistencies in ay-inversions in translation construction. Styles in translation vary from person-to-person, depending on their valuations and reading of the original text (Castagnoli, 2020). Likewise, ay-inversion in the context of translation boils down to a stylistic choice of the translator.

While KA is argued to be the natural pattern of speech for Filipino speakers (Yapan, 2017), this valuation of natural and unnatural is debated. Magracia (2001) notes that while DKA is correct in “meaning, syntactic order, and grammaticality,” it is not the “natural” way of speaking for Filipinos, as it reflects the speaker’s language of thought—in this case, the influence of English. The same justification, in terms of source language for translated corpora, can be said for machine-translated text (Visweswariah et al., 2011),
though this remains an open question for Filipino.

On the other hand, other research argues that DKA is a necessary aspect for “discourse continuity” (Bolata, 2022). Hence, the more complex the sentence, the higher the chance of utilizing the ay-inversion (Fox, 1985). We observed this in some datasets such as NLI, which had longer, more context-rich sentences that required the use of DKA in contrast with the shorter, KA-ordered sentences found in the CR task.

These frameworks demonstrate the subjective nature of the selection of Filipino sentence structure. Hence, such inconsistencies may still be observed in BATAYAN despite being rigorously translated by native speakers.

4.2 Issues with Natively-Sourced Data

While the issues with most datasets in BATAYAN were related with fluency and naturalness, the Philippine election-related tweets (Cabasag et al., 2019) dataset used for the SA and TD tasks presented unique challenges.

Incomplete entries. Due to the contextual nature of Filipino and social media communication, many samples lacked sufficient contextual information. To maintain dataset authenticity, we prioritized entries with adequate information for sentiment or toxicity analysis rather than complete removal. This was accomplished by selecting high-agreement samples where native speakers demonstrated consistent task performance.

Non-standard orthography. The dataset exhibited significant orthographic variation (spelling, capitalization, word boundaries, etc.) due to its natural language origins. We preserved this variation as it reflects authentic Filipino language use (Ilao et al., 2012; Javier, 2018; Caroro et al., 2020) relevant to toxicity detection and sentiment analysis tasks, aligning with our methodological principles. We maintained quality control by verifying that orthographical variations remained comprehensible to native readers.

Class imbalance. The dataset showed a disproportionate distribution of negative sentiment for certain political entities. We rebalanced the class distribution and prioritized high-agreement samples to mitigate individual annotator bias.

5 Evaluation

BATAYAN serves as the Filipino component of the SEA-HELM leaderboard. The evaluation framework of SEA-HELM follows prior systems such as HELM (Liang et al., 2023).

5.1 Evaluation Design

Tasks. The selection of tasks in BATAYAN mentioned in Section 3 aims to comprehensively evaluate the Filipino language capabilities of LLMs. Each task is a collection of test and example instances composed of an input string, references, metadata, and a ground truth label.

Prompts. For BATAYAN, we developed prompt templates for each task written in Filipino. We ensured that the instructions for each task are consistent with the prompt template design already used in SEA-HELM. A comprehensive list of these prompt templates can be found in Appendix D.

During model evaluation, input prompts are constructed using evaluation instances and the corresponding prompt template. The default evaluation setting of BATAYAN for instruction-tuned models is zero-shot prompting, where model prompts do not include in-context input-label examples. Base pre-trained models without instruction-tuning are evaluated under a five-shot setting. Given the input prompts and decoding parameters (see Appendix F), a model then generates output completions.

Metrics. We adopt multiple metrics to quantify the models’ performance on each task. For each metric, the model completion is treated as the prediction, while the instance label is used as the reference. For the NLU and NLR tasks, we report the macro F1 score. For machine translation (English→Filipino and Filipino→English), we use ChrF++ (Popović, 2017) and MetricX-24 using the metricx-24-hybrid-xxl-v2p6-bf10at16 model (Juraska et al., 2024). For abstractive summarization, we report three metrics: BERTScore (Zhang et al., 2020), ChrF++, and ROUGE-L F1 from the multilingual implementation of ROUGE (Lin, 2004) used in XL-Sum (Hasan et al., 2021). Default package parameter settings are used, and performance scores are based on a single run of the benchmark.

5.2 Results

Our evaluation results include over 50 small (≤11B), medium (≤32B), and large (≥32B) LLMs, including several commercial AI systems on the BATAYAN benchmark. The results underscore two principal findings: the substantial benefit of explicit Filipino language support and the scaling effects
Figure 3: Model performance of selected models on NLU, NLR, and NLG tasks. NLU and NLR tasks are measured using macro F1 score, NLG tasks are measured using ChrF++, MetricX-24, BERTScore, and ROUGE-L F1. The unweighted average across tasks and scores is taken. Targeted Filipino instruction tuning allows Gemma-SEA-LION-v3-9B-IT to achieve competitive performance with significantly larger open-source and commercial models.

observed across model sizes. Figure 3 highlights several model performances, and the complete experiment results can be found in Appendix G.

Filipino language support enhances task-specific performance. Tables 12–15 in Appendix G reveal that models with Filipino instruction tuning (such as the SEA-LION family of models) consistently outperform counterparts on culturally nuanced tasks. In particular, the Filipino-adapted model Gemma-SEA-LION-v3-9B-IT achieves an average macro F1 of 79.23 among small models, with a striking CR score of 92.75. In contrast, non-Filipino-specific models, such as many variants in the Meta Llama 3 series, exhibit very low or zero performance in CR, which emphasizes the challenge of capturing Filipino’s complex causality and morphology without explicit tuning.

On the NLG competency, tables 16–19 in Appendix G show that while ROUGE-L F1 scores show only modest differences between Filipino-tuned and general models, complementary evaluation using MetricX-24 provides a more sensitive measure of semantic fidelity. For instance, in eng→tgl translation, Filipino-supported systems like Gemma-SEA-LION-v3-9B-IT and Sailor2-8B-Chat achieve MetricX-24 scores exceeding 87, indicating superior preservation of Filipino-specific discourse and semantics. This observation shows that the apparent parity in ROUGE-L is only in part: when examined alongside semantic metrics, the benefit of dedicated language tuning is clear.

Scaling effects have clear trends beyond parameter similarity. Although many small models lie within a narrow 7B–9B parameter range, our broader evaluation—including medium and large models—illustrates a clear scaling effect. For example, among large models, Gemma-SEA-LION-v3-9B-IT attains an average macro F1 of 78.41 for NLU, macro F1 of 80.69 for NLR, and aggregated score of 60.35 in NLG, significantly outperforming similarly-tuned small models. This trend is evident even when models are compared within the same family: larger variants consistently demonstrate enhanced capability in handling both the nuanced understanding and generation challenges posed by Filipino text. The results thus suggest that increased model capacity—when combined with region-specific instruction tuning—enables a better representation of Filipino linguistic phenomena, including code-switching and agglutinative morphology.

Insights from commercial systems. In addition to open-source models, our evaluation also includes several commercial LLMs (Tables 15 and 19). Commercial systems such as Gemini and GPT-4o variants exhibit competitive performance across both understanding and generation tasks. For instance, on NLU and NLR tasks, commercial
models achieve average macro F1 scores ranging from 75.14 to 86.23. On the NLG competency, commercial systems also deliver robust results: Gemini and GPT-4o models maintain strong MetricX-24 scores above 88, indicative of high-quality semantic translation in both \( \text{eng} \rightarrow \text{tgl} \) and \( \text{tgl} \rightarrow \text{eng} \) directions.

While these commercial models benefit from large-scale pre-training and domain-specific optimization, our findings show that open-source, Filipino-tuned models achieve comparable—and in certain tasks, superior—performance. In particular, when cultural and linguistic nuances are critical, explicit regional instruction tuning offers a tangible advantage. The comparable performance of commercial systems reinforces that our dual strategy of targeted language support combined with model scaling is effective in academic settings as well as real-world applications.

Overall, our results provide strong evidence for a dual strategy in multilingual modeling: integrating region-specific instruction tuning with scalable architectures. Filipino-tuned models excel in tasks that require nuanced cultural and morphological sensitivity, and their performance improves markedly with increased capacity. Notably, open-source models fine-tuned with Filipino instructions are competitive with and even surpass state-of-the-art commercial systems on both understanding and reasoning tasks. These findings highlight that explicit Filipino language support not only enhances semantic fidelity and discourse preservation but also positions such models ideally for real-world, production-level applications in the Philippines.

6 Conclusion

In this paper, we introduced BATAYAN for holistically evaluating LLMs on a gamut of Filipino language tasks covering natural language understanding, reasoning, and generation. Our findings show that LLMs with explicit Filipino language support and fine-tuned on Filipino instructions demonstrate better performance on BATAYAN compared to other models.

As one of the major challenges in developing BATAYAN is maintaining the naturalness and fluency of the language, we plan to develop metrics and tools that can help discriminate from translationese and natural texts, inspired by prior research (Lovenia et al., 2024; Riley et al., 2020).

Limitations

While our work provides a unique and comprehensive Filipino language benchmark, we also reported in previous sections the challenges and limitations in developing high-quality NLP resources for the under-represented language. Additionally, prosodic characteristics such as stress and sarcasm are not immediately obvious in the datasets utilized. We account for this by selecting only high-agreement samples. We also note that the domain of both SA and TD tasks (which were tweets surrounding political events), limits the distributions of these tasks, and as such we recommend utilizing other datasets that cover a wider set of domains and use cases.

While we recognize that datasets with larger amounts of samples are preferred to ensure generalizability, we maintain our position to prioritize consistent, high-quality, and representative samples. The thoroughness and rigor of our review, annotation, and translation processes were essential and maintaining the quality and reliability of our benchmark. We believe that our benchmark serves as a valuable starting point for future research and can certainly be expanded in subsequent iterations.

Further, given the breadth of models and tasks, a comprehensive evaluation incorporating error cases was not possible. We also recognize the importance of evaluation LLMs that specialize on specific tasks such as translation. We hope that a public release of the BATAYAN codebase and benchmark on a leaderboard can help researchers better characterize the weaknesses of models, conduct their own analysis on the Filipino language capabilities of LLMs, and experiment with specialized models on BATAYAN.

Ethical Considerations

The methodology used in this work was approved by an Institutional Review Board (NUS-IRB-2024-617). We have released the BATAYAN dataset as part of the SEA-HELM evaluation suite and leaderboard under the Creative Commons Attribution Share-Alike 4.0 (CC-BY-SA 4.0) license, respecting the licenses of the source datasets used in this study.

Due to the nature of the toxicity detection task, we note that the authors were exposed to offensive material. Nonetheless, they were encouraged to report inappropriate samples and were given the
option to stop work if desired.

For the annotation of the sentiment analysis task, we involved a quality assurance team consisting of native Filipino speakers. The team, comprised of students from local universities in Singapore, were recruited through public advertisements. These stated the estimated work load and remuneration, which were consistent with university research guidelines and regulatory requirements.

We do not foresee negative social impacts from this paper. Our work introduces Filipino language resources that were reviewed by native Filipino speakers, paying due respect to local cultural sensitivities. We thus do not believe that our research will contribute to over-generalizations regarding Filipino culture.

Acknowledgments

This research/project is supported by the National Research Foundation, Singapore under its National Large Language Models Funding Initiative. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not reflect the views of National Research Foundation, Singapore.

The authors would like to thank National University of Singapore and AI Singapore, especially the AI Products and SEA-LION team, for their unwavering support with this endeavor.

The authors likewise thank the University of Cambridge, Department of Computer Science and Technology, and Magdalene College. David Africa’s work is supported by the Cambridge Trust and the Jardine Foundation.

Lastly, the authors would like to thank all the Filipino natives involved in this study for their time and valuable contributions.

References

Virgilio S. Almario. 2014. KWF Manwal sa Masinop na Pagsulat. Komisyon sa Wikang Filipino.

Virgilio S. Almario. 2016. Batayang pagsasalin: Ilang patnubay at babasahin para sa baguhan. Komisyon sa Wikang Filipino.

John Archibald and William Delaney O’Grady. 2001. Contemporary linguistics: An introduction. St. Martin’s Press.

Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Jon Ander Campos, Yi Chern Tan, Kelly Marchisio, Max Bartolo, Sebastian Ruder, Acyr Locatelli, Julia Kreutzer, Nick Frosst, Aidan Gomez, Phil Blunsom, Marzieh Fadaee, Ahmet Üstün, and Sara Hooker. 2024. Aya 23: Open weight releases to further multilingual progress. Preprint, arXiv:2405.15032.

Ekaterina Baklanova. 2017. Types of Borrowings in Tagalog/Filipino. Kritika Kultura, 28.

Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2024. The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 749–775, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.

Maria Lourdes S Bautista. 1991. Code-switching studies in the Philippines. International Journal of the Sociology of Language, 8(1).

Maria Lourdes S. Bautista. 2004. Tagalog-English Code Switching as a Mode of Discourse. Asia Pacific Education Review), 5(2):226–233.

Emmanuel Jayson V Bolata. 2022. Ang Noumenal at ang Nominal sa Panulaan ni Allan Popa. Daluyan: Journal ng Wikang Filipino, 28(2).

J Donald Bowen. 1971. Hispanic languages and influence in Oceania. Current trends in linguistics, 8:938–52.

Neil Vicente Cabasag, Vicente Raphael Chan, Sean Christian Lim, Mark Edward Gonzales, and Charibeth Cheng. 2019. Hate speech in Philippine election-related tweets: Automatic detection and classification using natural language processing. Philippine Computing Journal XIV, 1.

Rhandley D Cajote, Rowena Cristina L Guevara, Michael Gringo Angelo R Bayona, and Crisron Rudolf G Lucas. 2024. Philippine Languages Database: A Multilingual Speech Corpora for Developing Systems for Philippine Spoken Languages. LREC-COLING 2024, page 264.

Roseclaremath A Caroro, Rolysent K Paredes, and Jerry M Lumasag. 2020. Rules for orthographic word parsing of the Philippines’ Cebuano-Visayan language using context-free grammars. International Journal of Software Science and Computational Intelligence (IJSSCI), 12(2):34–49.

Sara Castagnoli. 2020. Translation choices compared: Investigating variation in a learner translation corpus. Translating and Comparing Languages: Corpus-based Insights. Corpora and Language in Use Proceedings, 6:25–44.

Gloria Chan-Yap. 1980. Hokkien Chinese borrowings in Tagalog. Dept. of Linguistics, Research School of Pacific Studies, Australian National University.
Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46.

Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating Cross-lingual Sentence Representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics.

Jan Christian Blaise Cruz, Jose Kristian Resabal, James Lin, Dan John Velasco, and Charibeth Cheng. 2021. Exploiting news article structure for automatic corpus generation of entailment datasets. In PRICAI 2021: Trends in Artificial Intelligence, pages 86–99, Cham. Springer International Publishing.

Jan Christian Blaise Cruz, Julianne Agatha Tan, and Charibeth Cheng. 2020. Localization of fake news detection via multitask transfer learning. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 2596–2604, Marseille, France. European Language Resources Association.

John Dang, Shivalika Singh, Daniel D’souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kublik, Meor Amer, Virata Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Kocmi, Florian Strub, Nathan Grinsztajn, Yannis Flet-Berliac, Aycr Locatelli, Hangyu Lin, Dwarkan Talupuru, Bharat Venkitesh, David Cairuz, Bowen Yang, Tim Chung, Wei-Yin Ko, Sylvie Shang Shi, Amir Shukayev, Sammie Bae, Aleksandra Piktus, Roman Castagné, Felipe Cruz-Salinas, Eddie Kim, Lucas Crawhall-Stein, Adrien Morisot, Sudip Roy, Phil Blunsom, Ivan Zhang, Aidan Gomez, Nick Frosts, Marzieh Fadaee, Beyza Ermis, Ahmet Üstün, and Sara Hooker. 2024. Aya expanse: Combining research breakthroughs for a new multilingual frontier. Preprint, arXiv:2412.04261.

DeepSeek-AI. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948.

Rosario Delgado and Xavier-Andoni Tibau. 2019. Why Cohen’s Kappa should be avoided as performance measure in classification. PloS one, 14(9):e0222916.

Shirley N Dita, Rachel Edita O Roxas, and Paul Inventado. 2009. Building online corpora of Philippine languages. In Proceedings of the 23rd Pacific Asia Conference on Language, Information and Computation, pages 646–653. Waseda University.

Sumanth Doddapaneni, Rahul Aralikatte, Gowtham Ramesh, Shreya Goyal, Mitesh M. Khapra, Anoop Kunchukuttan, and Pratyush Kumar. 2023. Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12402–12426, Toronto, Canada. Association for Computational Linguistics.

Longxu Dou, Qian Liu, Fan Zhou, Changyu Chen, Zili Wang, Ziqi Jin, Zichen Liu, Tongyao Zhu, Cunxiao Du, Penghui Yang, Haonan Wang, Jiaheng Liu, Yongchi Zhao, Xiachong Feng, Xin Mao, Man Tsung Yeung, Kunat Pipatanakul, Fajri Koto, Min Si Thu, Hynek Kydlicek, Zeyi Liu, Qunshu Lin, Sitipong Sirapisararmongkol, Kridathpad Sae-Khow, Nirattisai Thongchim, Taechawat Konkaew, Narong Borijindargoorn, Anh Dao, Matichon Maneegard, Phakphum Artkaew, Zheng-Xin Yong, Quan Nguyen, Wannaphong Phatthiyaphaibun, Hoang H. Tran, Mike Zhang, Shiqi Chen, Tianyu Pang, Chao Du, Xinyi Wan, Wei Lu, and Min Lin. 2025. Sailor2: Sailing in south-east asia with inclusive multilingual llms. Preprint, arXiv:2502.12982.

Guangchao Charles Feng and Xinshu Zhao. 2016. Do not force agreement: A response to Krippendorff (2016). Methodology.

Barbara A Fox. 1985. Word-order inversion and discourse continuity in Tagalog. Text-Interdisciplinary Journal for the Study of Discourse, 5(1-2):39–54.

Martin Gellerstam. 1986. Translationese in Swedish Novels Translated from English. Translation studies in Scandinavia, 1:88–95.

Matthew Phillip V Go and Nicco Nocon. 2017. Using Stanford part-of-speech tagger for the morphologically-rich Filipino language. In Proceedings of the 31st Pacific Asia Conference on Language, Information and Computation, pages 81–88. Waseda University.

Mikhail Alic Go and Leah Gustilo. 2013. Tagalog or Taglish: The lingua franca of Filipino urban factory workers. Philippine ESL Journal, 10:57–87.

Wilkinson Daniel Wong Gonzales. 2022. Interactions of Sinic languages in the Philippines: Sinicization, Filipinization, and Sino-Philippine language creation. In The Palgrave handbook of Chinese language studies, pages 369–408. Springer.

Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022. The Flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computational Linguistics, 10:522–538.

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jahuiri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien
Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Alonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esioubo, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasudevan Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Giridhar, Rohit Patel, Romain Sauvestre, Ronnie Polidorio, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sien Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vitor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenying Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenber, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsian Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymann, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcueta, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippo Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegen, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelen, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark,
Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battye, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojiang Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoqiu Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.

Jia Guo, Longxu Dou, Guangtao Zeng, Stanley Kok, Wei Lu, and Qian Liu. 2024. Sailcompass: Towards reproducible and robust evaluation for southeast asian languages. arXiv preprint arXiv:2412.01186.

Yanzhu Guo, Chloé Clavel, Moussa Kamal Eddine, and Michalis Vazirgiannis. 2022. Questioning the Validity of Summarization Datasets and Improving Their Factual Consistency. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5716–5727, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Zishan Guo, Renren Jin, Chuang Liu, Yufei Huang, Dan Shi, Linhao Yu, Yan Liu, Jiaxuan Li, Bojian Xiong, Deyi Xiong, et al. 2023. Evaluating large language models: A comprehensive survey. arXiv preprint arXiv:2310.19736.

Muhammad Usman Hadi, Rizwan Qureshi, Abbas Shah, Muhammad Irfan, Anas Zafar, Muhammad Bilal Shaikh, Naveed Akhtar, Jia Wu, Seyedali Mirjalili, et al. 2023. A survey on large language models: Applications, challenges, limitations, and practical usage. Authorea Preprints.

Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubashshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021. XL-Sum: Large-scale Multilingual Abstractive Summarization for 44 Languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4693–4703, Online. Association for Computational Linguistics.

Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: a massively multilingual multi-task benchmark for evaluating cross-lingual generalization. In Proceedings of the 37th International Conference on Machine Learning, ICML’20. JMLR.org.

Xin Huang, Tarun Kumar Vangani, Minh Duc Pham, Xunlong Zou, Bin Wang, Zhengyuan Liu, and Ai Ti Aw. 2025. Meralion-textllm: Cross-lingual understanding of large language models in chinese, indonesian, malay, and singlish. Preprint, arXiv:2501.08335.

Joel P Ilao, Timothy Israel D Santos, and Rowena Cristina L Guevara. 2012. Comparative analysis of actual language usage and selected grammar and orthographical rules for Filipino, Cebuano-Visayan and Ilokano: a Corpus-based Approach. Electrical and Electronics Engineering Institute. University of the Philippines Diliman.

Jem R Javier. 2018. Pagsusuri sa ortograhiya ng kambal-katingin sa Filipino batay sa korpus (A corpus-based analysis of consonant clusters in Filipino orthography). Social Science Diliman, 14(1).

Rodney C Jubilado. 2004. Philippine linguistics, Filipino language, and the Filipino nation. JATI-Journal of Southeast Asian Studies, 9(1):43–53.

Juraj Juraska, Daniel Deutsch, Mara Finkelstein, and Markus Freitag. 2024. MetricX-24: The Google submission to the WMT 2024 metrics shared task. In Proceedings of the Ninth Conference on Machine Translation, pages 492–504, Miami, Florida, USA. Association for Computational Linguistics.

Pride Kavumba, Naoya Inoue, Benjamin Heinzerling, Keshav Singh, Paul Reisert, and Kentaro Inui. 2019. When Choosing Plausible Alternatives, Clever Hans can be Clever. In Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing, pages 33–42, Hong Kong, China. Association for Computational Linguistics.

Klaus Krippendorff. 2018. Content analysis: An introduction to its methodology. Sage publications.
Paul Kroeger. 1993. Phrase structure and grammatical relations in Tagalog. Center for the Study of Language (CSLI).

Wei Qi Leong, Jian Gang Ngui, Yosephine Susanto, Hamsawardhini Rengarajan, Kengatharaiyer Sarveswaran, and William Chandra Tjihi. 2023. Preprint, arXiv:2309.06085. [link].

Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. 2023. Holistic Evaluation of Language Models. Preprint, arXiv:2211.09110.

Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.

Chaoqun Liu, Wenxuan Zhang, Jiahou Ying, Mahani Aljunied, Anh Tuan Luu, and Lidong Bing. 2025. SeaExam and SeaBench: Benchmarking LLMs with local multilingual questions in Southeast Asia. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 6119–6136, Albuquerque, New Mexico. Association for Computational Linguistics.

Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott. 2021. Visually Grounded Reasoning across Languages and Cultures. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10467–10485, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo Hao Cheng, Yegor Klochkov, Muhammad Faauz Taufiq, and Hang Li. 2023. Trustworthy LLMs: A survey and guideline for evaluating large language models’ alignment. arXiv preprint arXiv:2308.05374.

Holy Lovenia, Rahmad Mahendra, Salsabil Maulana Akbar, Lester Jamesalidat Miranda, Jennifer Santoso, Elyanah Aco, Akhdan Fadhilah, Jonibek Mansurov, Joseph Marvin Imperial, Onno P. Kampman, Joel Ruben Antony Moniz, Muhammad Ravi Shulthan Habibi, Frederikus Hudi, Jann Railey Montalan, Ryan Ignatius Hadiwijaya, Joanito Agili Lopo, William Nixon, Börje F. Karlsson, James Jaya, Ryandito Diandaru, Yuze Gao, Patrick Amadeus Irawan, Bin Wang, Jan Christian Blaise Cruz, Chenxi Whitehouse, Ivan Halim Parmonangan, Maria Khelli, Wenyu Zhang, Lucky Susanto, Reynard Adha Ryanda, Sonny Lazuardi Hermawan, Dan John Velasco, Muhammad Dehan Al Kautsar, Willy Fitra Hendria, Yasmin Moslem, Noah Flynn, Muhammad Farid Adilazuarda, Haochen Li, Johannes Lee, R. Damanhuri, Shuo Sun, Muhammad Reza Qorib, Amirbek Djanibekov, Wei Qi Leong, Quyet V. Do, Niklas Muennighoff, Tanrada Pansuwan, Ilham Firdausi Putra, Yan Xu, Tai Ngee Chia, Ayu Purwarianti, Sebastian Ruder, William Chandra Tjihi, Peerat Limkonchotiwat, Alham Fikri Aji, Sedrick Keh, Genta Indra Winata, Ruochen Zhang, Fajri Koto, Zheng Xin Yong, and Samuel Cahyawijaya. 2024. SEACrowd: A multilingual multimodal data hub and benchmark suite for Southeast Asian languages. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5155–5203, Miami, Florida, USA. Association for Computational Linguistics.

Emma B Magracia. 2001. Panumbas sa Filipino ng Panghihiram: Pagtingging Semantikal at Sintaktikal. Asia Pacific Journal of Social Innovation (formerly The Mindanao Forum), 16(2):31–42.

Jonathan C. Malicsi. 2013. Gramar ng Filipino. Aklat Sanyata. Sentro ng Wikang Filipino, Unibersidad ng Pilipinas.

Lester James Miranda. 2023a. Developing a named entity recognition dataset for tagalog. arXiv preprint arXiv:2311.07161.

Lester James Miranda. 2023b. Towards a Tagalog NLP pipeline.

Raymond Ng, Thanh Ngan Nguyen, Yuli Huang, Ngee Chia Tai, Wai Yi Leong, Wei Qi Leong, Xianbin Yong, Jian Gang Ngui, Yosephine Susanto, Nicholas Cheng, Hamsawardhini Rengarajan, Peerat Limkonchotiwat, Adithya Venkatadri Hulagadri, Kok Wai Teng, Yeo Yeow Tong, Bryan Siow, Wei Yi Teo, Wayne Lau, Choon Meng Tan, Brandon Ong, Zhi Hao Ong, Janh Railey Montalan, Adwin Chan, Sajebar Antonyrex, Ren Lee, Esther Choa, David Ong Tat-Wee, Bing Jie Darius Liu, William Chandra Tjihi, Erik Cambria, and Leslie Teo. 2025. Sea-lion: Southeast asian languages in one network. Preprint, arXiv:2504.05747.

NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Anna Sun Jean Maillard, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loïc Barrault, Gabriel Mejía Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan,
Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No Language Left Behind: Scaling Human-Centered Machine Translation.

OpenAI, ; Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Madry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex Renzin, Alex Tachard Passos, Alexander Kirillov, Alexi Christakis, Alexis Conneau, Ali Kamali, Allian Jabri, Allison Moyer, Allison Tam, Amadou Crookes, Amin Tootoochian, Amin Tootoochian, Ananya Kumar, Andrea Vallone, Andrej Karpathy, Andrew Braunstein, Andrew Cann, Andrew Codispoti, Andrew Galu, Andrew Kondrich, Andrew Tulloch, Andrej Mishchenko, Angela Baek, Angela Jiang, Antoine Pelisse, Antonia Woodford, Anuj Gosalia, Arka Dhar, Ashley Pantuliano, Avi Nayak, Avital Oliver, Barret Zoph, Behrooz Ghorbani, Ben Leimberger, Ben Rossen, Ben Sokolovsky, Ben Wang, Benjamin Zweig, Beth Hoover, Blake Samic, Bob McGrew, Bobby Spero, Bogo Giertler, Bowen Cheng, Brad Lightcap, Brandon Walkin, Brendan Quinn, Brian Guarraci, Brian Hsu, Bright Kellogg, Brydon Eastman, Camillo Lugaresi, Carroll Wainwright, Cary Bassin, Cary Hudson, Casey Chu, Chad Nelson, Chak Li, Chan Jun Shern, Channing Conger, Charlotte Barette, Chelsea Voss, Chen Ding, Cheng Lu, Chong Zhang, Chris Beaumont, Chris Hallacy, Chris Koch, Christian Gibson, Christina Kim, Christine Choi, Christine McLeavey, Christopher Hesse, Claudia Fischer, Clemens Winter, Coley Czarnecki, Colin Jarvis, Colin Wei, Constantin Koumouzelis, Dane Sherburn, Daniel Kappler, Daniel Levin, Daniel Levy, David Carr, David Farhi, David Mely, David Robinson, David Sasaki, Denny Jin, Dev Valladares, Dimitris Tsipras, Doug Li, Duc Phong Nguyen, Duncan Findlay, Edede Oiwoh, Edmund Wong, Ehsan Asdar, Elizabeth Proehl, Elizabeth Yang, Eric Antonow, Eric Kramer, Eric Peterson, Eric Sigler, Eric Wallace, Eugene Brevido, Evan Mays, Farzad Khorasani, Felipe Petroski Such, Filippo Raso, Francis Zhang, Fred von Lohmann, Freddie Sulit, Gabriel Goh, Gene Oden, Geoff Salmon, Giulio Starace, Greg Brockman, Hadi Salman, Haiming Bao, Haitang Hu, Hannah Wong, Haoyu Wang, Heather Schmidt, Heather Whitney, Heewoo Jun, Hendrik Kirchner, Henrique Ponde de Oliveira Pinto, Hongyu Ren, Huiwen Chang, Hyung Won Chung, Ian Kivlichan, Ian O’Connell, Ian O’Connell, Ian Osband, Ian Silber, Ian Sohl, Ibrahim Okuyucu, Ika Lan, Ilya Kosztikov, Ilya Sutskever, Ingmar Kanitscheider, Ishaan Gulrajani, Jacob Coxon, Jacob Menick, Jakub Pachocki, James Aung, James Betker, James Crooks, James Lennon, Jamie Kiros, Jan Leike, Jane Park, Jason Kwon, Jason Phang, Jason Teplitz, Jason Wei, Jason Wolfe, Jay Chen, Jeff Harris, Jenia Varavva, Jessica Gan Lee, Jessica Shieh, Ji Lin, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joanne Jang, Joaquin Quionero Candela, Joe Beutler, Joe Landers, Joel Parish, Johannes Heidecke, John Schulman, Jonathan Lachman, Jonathan McKay, Jonathan Uesato, Jonathan Ward, Jong Wook Kim, Joost Huizinga, Jordan Sitkin, Jos Kraaijeveeld, Josh Gross, Josh Kaplan, Josh Snyder, Joshua Achiam, Joy Jiao, Joyce Lee, Juntang Zhuang, Justyn Harriman, Kai Fricke, Kai Hayashi, Karan Singhal, Katy Shi, Kavin Kartik, Kayla Wood, Kendra Rimbach, Kenny Hsu, Kenny Nguyen, Keren Gu-Lemberg, Kevin Button, Kevin Liu, Kiel Howe, Krithika Muthukumar, Kyle Luther, Lama Ahmad, Larry Kai, Lauren Itow, Lauren Workman, Leher Pathak, Leo Chen, Li Jing, Lia Guy, Liam Fedus, Liang Zhou, Lien Mamitsuka, Lilian Wang, Lindsay McCallum, Lindsey Held, Long Ouyang, Louis Fevrier, Lu Zhang, Lukas Kondracikuk, Lukasz Kaiser, Luke Hewitt, Luke Metz, Lyric Doshi, Mada Aflak, Maddie Simens, Madeleine Boyd, Madeleine Thompson, Marat Dukhan, Mark Chen, Mark Gray, Mark Hudnall, Marvin Zhang, Marwan Aljubeh, Mateusz Litwin, Matthew Zeng, Max Johnson, Maya Shetty, Mayank Gupta, Meghan Shah, Mehmet Yatbaz, Meng Jia Yang, Mengcheng Zhao, Mia Glaese, Mianna Chen, Michael Janner, Michael Lampe, Michael Petrov, Michael Wu, Michele Wang, Michelle Fradin, Michelle Pokrass, Miguel Castro, Miguel Oom Temudo de Castro, Mikhail Pavlov, Miles Brundage, Miles Wang, Minal Khan, Mira Murati, Mo Bavarian, Molly Lin, Murat Yesildal, Nacho Soto, Natalia Gimelshein, Natalie Cone, Natalie Staudacher, Natalie Summers, Natan LaFontaine, Neil Chowdhury, Nick Ryder, Nick Stathas, Nick Turley, Nik Tezak, Niko Felix, Nithanth Kudige, Nitish Keskar, Noah Deutsch, Noel Bundick, Nora Puckett, Ofir Nachum, Ola Okelola, Oleg Boiko, Oleg Murk, Oliver Jaffe, Olivia Watkins, Olivier Godevert, Owen Campbell-Moore, Patrick Chao, Paul McMillan, Pavel Belov, Peng Su, Peter Bak, Peter Bakkum, Peter Deng, Peter Dolan, Peter Hoeschele, Peter Welinder, Phil Tillet, Philip Pronin, Philippe Tillet, Prafulla Dhariwal, Qiming Yuan, Rachel Dias, Rachel Lim, Rahul Arora, Rajan Troll, Randall Lin, Rapha Gontijo Lopes, Raul Puri, Reah Miyara, Reimar Leike, Renaud Gaubert, Reza Zamani, Ricky Wang, Rob Donnelly, Rob Honsby, Rocky Smith, Rohan Sahai, Rohit Ramchandani, Romain Huet, Rory Carmichael, Rowan Zellers, Roy Chen, Ruby Chen, Ruslan Nigmatullin, Ryan Cheu, Saachi Jain, Sam Altman, Sam Schoenholtz, Sam Toizer, Samuel Miserendino, Sandhini Agarwal, Sara Culver, Scott Ethersmith, Scott Gray, Sean Grove, Sean Metzger, Shamez Hermani, Shantanu Jain, Shengjia Zhao, Sherwin Wu, Shino Jomoto, Shirong Wu, Shuaiqi Xia, Sonia Phene, Spencer Papay, Srinivas Narayanan, Steve Coffey, Steve Lee, Stewart Hall, Suchir Balaji, Tal Broda, Tal Stramer, Tao Xu, Tarun Gogineni, Taya Christianson, Ted Sanders, Tejal Patwardhan, Thomas Cunningham, Thomas Degry, Thomas Dimson, Thomas Raoux, Thomas Shadwell, Tianhao Zheng, Todd Underwood, Todor Markov, Toki Sherbakov, Tom Rubin, Tom Stasi, Tomer Kaftan, Tristan Heywood, Troy Peterson, Tyce Walters, Tyna Eloundou, Valerie Qi, Veit Moeller,
Vinnie Monaco, Vishal Kuo, Vlad Fomenko, Wayne Chang, Weiyi Zheng, Wenda Zhou, Wesam Manassra, Will Sheu, Wojciech Zaremba, Yash Patil, Yilei Qian, Yongjik Kim, Youlong Cheng, Yu Zhang, Yuchen He, Yuchen Zhang, Yujia Jin, Yunxing Dai, and Yury Malkov. 2024. Gpt-4o system card. Preprint, arXiv:2410.21276.

Jed Sam Pizarro-Guevara. 2010. Revisiting Person-Case constraint on ay-inversion in Tagalog.

Maja Popović. 2017. chrF++: words helping character n-grams. In Proceedings of the Second Conference on Machine Translation, pages 612–618. Copenhagen, Denmark. Association for Computational Linguistics.

David Martin Ward Powers. 2012. The Problem with Kappa. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 345–355, Avignon, France. Association for Computational Linguistics.

J Stephen Quakenbush. 2005. Philippine linguistics from an SIL perspective: Trends and prospects. Current issues in Philippine linguistics and anthropology: Parangal kay Lawrence A. Reid, pages 3–27.

Qwen, .; An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115.

Teresita V Ramos. 2021. Tagalog structures. University of Hawaii Press.

Lawrence A Reid. 2018. Modeling the linguistic situation in the Philippines. Senri ethnological studies, 98:91–105.

Parker Riley, Isaac Caswell, Markus Freitag, and David Grangier. 2020. Translationese as a Language in “Multilingual” NMT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7737–7746, Online. Association for Computational Linguistics.

Irene Rivera-Trigueros. 2022. Machine translation systems and quality assessment: a systematic review. Language Resources and Evaluation, 56(2):593–619.

Guijin Son, Dongkeun Yoon, Juyoung Suk, Javier Aula-Blasco, Mano Aslan, Vu Trong Kim, Shayekh Bin Islam, Jaume Prats-Cristià, Lucia Tormo-Bahuelos, and Seungone Kim. 2024. Mm-eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-judge and Reward Models. arXiv preprint arXiv:2410.17578.

D.S. Tanawan, A.A. Nacin, and Lartec J.K. 2008. Istruktura ng Wikang Filipino. Trinitas Publishing House.

Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, Sorooosh Mariooryad, Yifan Ding, Xinyang Geng, Fred Alcober, Roy Frostig, Mark Omernick, Lexi Walker, Cosmin Paduraru, Christina Sorokin, Andrea Tacchetti, Colin Gaffney, Samira Daruki, Olcen Sercinoglu, Zach Gleicher, Juliette Love, Paul Voightlaender, Rohan Jain, Gabriela Surita, Kareem Mohamed, Rory Blevins, Junwhan Ahn, Tao Zhu, Kornraphop Kawintiranon, Orhan Firat, Yiming Gu, Yujing Zhang, Matthew Rahtz, Maanal Faruqui, Natalie Clay, Justin Gilmer, JD Co-Reyes, Ivo Penchev, Rui Zhu, Nobuyuki Morioka, Kevin Hui, Krishna Haridasan, Victor Campos, Mahdis Mahdieh, Mandy Guo, Samer Hassan, Kevin Kilgour, Arpi Vezer, Heng-Tze Cheng, Raoul de Liedekerke, Siddharth Goyal, Paul Barham, DJ Strouse, Seb Noury, Jonas Adler, Mukund Sundararajan, Sharad Vikram, Dmitry Lipikhin, Michela Paganini, Xavier Garcia, Fan Yang, Dasha Valter, Maja Trebacz, Kiran Vodrahalli, Chulayuth Asawaroengchai, Roman Ring, Norbert Kalb, Livio Baldini Soares, Siddhartha Brahma, David Steiner, Tianhe Yu, Fabian Mentzer, Antoine He, Lucas Gonzalez, Bibo Xu, Raphael Lopez Kaufman, Laurent El Shafey, Junhyuk Oh, Tom Hennigan, George van den Driessche, Seth Odoom, Mario Lucic, Becca Roelofs, Sid Lall, Amit Marathe, Betty Chan, Santiago Ontanon, Luheng He, Denis Teplyashin, Jonathan Lai, Phil Crone, Bogdan Damoc, Lewis Ho, Sebastian Riedel, Karel Lenc, Chih-Kuan Yeh, Aakanksha Chowdhery, Yang Xu, Mehran Kazemi, Ehsan Amid, Anastasia Petrushina, Kevin Swersky, Ali Khodaei, Gowoon Chen, Chris Larkin, Mario Pinto, Geng Yan, Adria Puigdomenech Badia, Piyush Patil, Steven Hansen, Dave Orr, Sebastien M. R. Arnold, Jordan Grinstead, Andrew Dai, Sholto Douglas, Rishika Sinha, Vikas Yadav, Xi Chen, Elena Gribovskaya, Jacob Austin, Jeffrey Zhao, Kaushal Patel, Paul Komarek, Sophia Austin, Sebastian Borgeaud, Linda Friso, Abhimanyu Goyal, Ben Caine, Kris Cao, Da-Woon Chung, Matthew Lamm, Gabe Barth-Maron, Thais Kagohara, Kate Olszewski, Mia Chen, Kaushik Shivakumar, Rishabh Agarwal, Harshal Godhia, Ravi Rajwar, Javier Snaider, Xerxes Dotiwalla, Yuan Liu, Aditya Barua, Victor Ungureanu, Yuan Zhang, Bat-Orgil Batsaikhan, Mateo Wirth, James Qin, Ivo Danihelka, Tulsee Doshi, Martin Chadwick, Jinlin Chen, Sanil Jain, Quoc Le, Arjun Kar, Madhu Gurumurthy, Cheng Li, Ruoxin Sang, Fangyu Liu, Lampros Lamprou, Rich Munoz, Nathan Lintz, Harsh Mehta, Heidi Howard, Malcolm Reynolds, Lora Aroyo, Quan Wang, Lorenzo Blanco, Albin Cassirer, Jordan Griffith, Dipanjan Das, Stephan Lee, Jakub Sygnowski, Zach Fisher, James Besley, Richard Powell, Zafarali Ahmed, Dominik Paulus, David Reitter, Zalan Borsos, Rishabh Joshi, Aedan Pope, Steven Hand, Vittorio Selo, Vihan Jain, Nikhil Sethi,
Megha Goel, Takaki Makino, Rhys May, Zhen Yang, Johan Schalkwyk, Christina Butterfield, Anja Hauth, Alex Goldin, Will Hawkins, Evan Senter, Sergey Brin, Oliver Woodman, Marvin Ritter, Eric Noland, Minh Giang, Vijay Bolina, Lisa Lee, Tim Blyth, Ian Mackinnon, Machel Reid, Obaid Sarvana, David Silver, Alexander Chen, Lily Wang, Loren Maggiore, Oscar Chang, Nithya Attaluri, Gregory Thornton, Chung-Cheng Chiu, Oskar Bunyan, Nir Levine, Timothy Chung, Evgenii Eltyshev, Xiance Si, Timothy Lillicrap, Demetra Brady, Vaibhav Aggarwal, Boxi Wu, Yuanzhong Xu, Ross McIlroy, Kartikeya Badola, Paramjit Sandhu, Erica Moreira, Wojciech Stokowiec, Ross Hemmings, Dong Li, Alex Tudor, Pranav Shyam, Elahe Rahimtoroghi, Salem Haykal, Pablo Sprechmann, Xiang Zhou, Diana Minucu, Yujia Li, Ravi Addanki, Kalpesh Krishna, Xiao Wu, Alexandre Frechette, Matan Eyal, Allan Dafoe, Dave Lacey, Jay Whang, Thi Avrahami, Ye Zhang, Emanuel Tarop, Hanzhao Lin, Daniel Toyama, Eliza Rutherford, Motoki Sano, HyunJeong Choe, Alex Tomala, Chalence Safraneck-Shrader, Nora Kassner, Mantas Pajarskas, Matt Harvey, Sean Sechrist, Meire Fortunato, Christina Lyu, Gamaeldin Elsayed, Chenkai Kuang, James Lottes, Eric Chu, Chao Jia, Chih-Wei Chen, Peter Humphreys, Kate Baumli, Connie Tao, Rajkumar Samuel, Cicero Nogueira dos Santos, Anders Andreassen, Nemanja Rakicevic, Dominik Grewe, Aviral Kumar, Stephanie Winkler, Jonathan Caton, Andrew Brock, Sid Dalmia, Hannah Sheahan, Iain Barr, Yingjie Miao, Paul Natsve, Jacob Devlin, Feryal Behbahani, Flavien Prost, Yanhua Sun, Artiom Myaskovsky, Thanumalayan Sankaranarayana Pillai, Dan Hurt, Angeliki Lazarioudi, Xi Xiong, Ce Zheng, Fabio Pardo, Xiaowei Li, Dan Horgan, Joe Stanton, Moran Ambar, Fei Xia, Alejandro Lince, Mingjiu Wang, Basil Mustafa, Albert Webson, Hyo Lee, Rohan Anil, Martin Wicke, Timothy Doazat, Abhishek Sinha, Enrique Piqueiras, Elahe Dabir, Shyam Upadhyay, Anudhyan Boral, Lisa Anne Hendricks, Corey Fry, Josip Djolonga, Yu Su, Jake Walker, Jane Labanowski, Ronny Huang, Vedant Misra, Jeremy Chen, RJ Skerry-Ryan, Avi Singh, Shruti Rijhwani, Dian Yu, Alex Castro-Ros, Beer Changpinyo, Romina Datta, Sumit Bagri, Arnar Mar Hrafnkelsson, Marcello Maggioni, Daniel Zheng, Yury Sulsky, Shaobo Hou, Tom Le Paine, Antoine Yang, Jason Riesa, Dominika Rogozinska, Dror Marcus, Dalia El Badawy, Qiao Zhang, Luyu Wang, Helen Miller, Jeremy Greer, Lars Lowe Sjos, Azade Nova, Heiga Zen, Rahma Chaabouni, Mihaela Rosca, Jiepu Jiang, Charlie Chen, Ruibo Liu, Tara Sainath, Maxim Krikun, Alex Polozov, Jean-Baptiste Lespiau, Josh Newlan, Zeynep Cankara, Soo Kwak, Yunhan Xu, Phil Chen, Andy Coenen, Clemens Meyer, Katerina Tsihlas, Ada Ma, Juraj Gottweis, Jinwei Xing, Chenjie Gu, Jin Miao, Christian Frank, Zeynep Cankara, Sanjay Ganapathy, Ishita Dasgupta, Steph Hughes-Fitt, Heng Chen, David Reid, Keran Rong, Hongmin Fan, Joost van Amersfoort, Vincent Zhuang, Aaron Cohen, Shixiang Shane Gu, Anhad Mohananey, Anastasija Illic, Taylor Tobin, John Wieting, Anna Bortsova, Phoebe Thacker, Emma Wang, Emily Caveness, Justin Chiu, Eren Sezener, Alex Kaskasoli, Steven Baker, Katie Millican, Mohamed Elhawaty, Kostas Aisopos, Carl Lebsack, Nathan Byrd, Hanjun Dai, Wenhao Jia, Matthew Wiethoff, Elnaz Davoodi, Albert Weston, Lakshman Yagati, Arun Ahuja, Isabel Gao, Golan Pundak, Susan Zhang, Michael Azzam, Khe Chat Sim, Sergi Caelles, James Keeling, Abhanshu Sharma, Andy Swing, YaGuang Li, Chenxi Liu, Carrie Grimes Bostock, Yamini Bansal, Zachary Nado, Ankesh Anand, Josh Lipschultz, Abhijit Karmarkar, Lev Proleev, Abe Ittycheriah, Soheil Hassas Yeganeh, George Polovets, Aleksandra Faust, Jiao Sun, Alban Russtemi, Pen Li, Rakesh Shivanna, Jeremiah Liu, Chris Welty, Federico Lebron, Anirudh Baddepudi, Sebastian Krause, Emilio Parisotto, Radu Soricut, Zheng Xu, Dawn Bloxwich, Melvin Johnson, Behnam Neyshabur, Justin Mao-Jones, Renshen Wang, Vinay Ramasesh, Zaheer Abbas, Arthur Guez, Constant Segal, Duc Dung Nguyen, James Svensson, Le Hou, Sarah York, Kieran Milan, Sophie Bridgers, Wiktor Gworek, Marco Tagliasacchi, James Lee-Thorp, Michael Chang, Alexey Guseynov, Ale Jakse Hartman, Michael Kwong, Ruizhe Zhao, Sheleem Kashem, Elizabeth Cole, Antoine Miech, Richard Tanburn, Mary Phuong, Filip Pavetic, Sebastien Cevey, Ramona Comanescu, Richard Ives, Sherry Yang, Cosmo Du, Bo Li, Zizhao Zhang, Mariko Inuma, Clara Huiyi Hu, Aurko Roy, Shaan Bijwadia, Zhenkai Zhu, Danilo Martins, Rachel Saputro, Anita Gergely, Steven Zheng, Dawei Jia, Ioannis Antonoglou, Adam Sadovsky, Shane Gu, Yingying Bi, Alek Andreev, Sina Samangooei, Mina Khan, Tomas Kocisky, Angelo Filos, Chintu Kumar, Colton Bishop, Adams Yu, Sarah Hodkinson, Sid Mittal, Premal Shah, Alexandre Moufarek, Yong Cheng, Adam Bloniarz, Jaehoon Lee, Pedram Pejman, Paul Michel, Stephen Spencer, Vladimir Feinberg, Xuehan Xiong, Nikolay Savinov, Charlotte Smith, Siham Shakeri, Dustin Tran, Mary Chesus, Bernd Bohnet, George Tucker, Tamara von Glehn, Carrie Muir, Yiran Mao, Hideto Kazawa, Ambrose Slone, Kedar Soparkar, Disha Shrivastava, James Cobon-Kerr, Michael Sharman, Jay Pavagadhi, Carlos Araya, Karolis Misinas, Nimesh Ghelani, Michael Laskin, David Barker, Qiujia Li, Anton Briukhov, Neil Housby, Mia Glasea, Balaji Lakshminarayanan, Nathan Schucher, Yunhao Tang, Eli Collins, Hyeonjaek Lim, Fangxiaoyu Feng, Adria Recasens, Guangda Lai, Alberto Magni, Nicola De Cao, Aditya Siddhant, Zoe Ashwood, Jordi Orbay, Mostafa Dehghani, Jenny Brennan, Yifan He, Kelvin Xu, Yang Gao, Carl Saroufim, James Molloy, Xinyi Wu, Seb Arnold, Solomon Chang, Julian Schrittwieser, Elena Buchatskaya, Soroush Radpour, Martin Polacek, Skye Giordano, Ankur Bapna, Simon Tokumine, Vincent Hellendoorn, Thibault Sottiaux, Sarah Cogan, Aliaksiej Severyn, Mohammad Saleh, Shantanu Thakoor, Laurent Shefey, Siyuan Qiao, Meenu Gaba, Shuo yiin Chang, Craig Swanson, Biao Zhang, Benjamin Lee, Paul Kishan Rubenstein, Gan Song, Tom Kwiatkowski, Anna Koop, Ajay Kannan, David Kao, Parker Schuh, Axel Stjerngren, Golnaz Ghiasi,
Gena Gibson, Luke Vilnis, Ye Yuan, Felipe Tiengo Ferreira, Aishwarya Kamath, Ted Klimenko, Ken Franko, Kefan Xiao, Indro Bhattacharya, Miteyan Patel, Rui Wang, Alex Morris, Robin Strudel, Vivek Sharma, Peter Choy, Sayed Hadi Hashemi, Jessica Landon, Mara Finkelstein, Priya Jhakra, Justin Frye, Megan Barnes, Matthew Mauger, Dennis Daun, Khuslen Baatarsukh, Matthew Tung, Wael Farhan, Henryk Michalewski, Fabio Viola, Felix de Chaumont Quityr, Charline Le Lan, Tom Hudson, Qingze Wang, Felix Fischer, Ivy Zheng, Elspeth White, Anca Dragan, Jean baptiste Alayrac, Eric Ni, Alexander Pritzel, Adam Iwanicki, Michael Isard, Anna Bulanova, Lukas Zilka, Ethan Dyer, Devendra Sachan, Srivatsan Srinivasan, Hannah Muckenhirn, Honglong Cai, Amol Madhane, Mukarram Tariq, Jack W. Rae, Gary Wang, Kareem Ayoub, Nicholas FitzGerald, Yao Zhao, Woohyun Han, Chris Alberti, Dan Garrette, Kashyap Krishnakumar, Mai Gimenez, Anselm Levskaya, Daniel Sohn, Josip Matak, Inaki Iturrate, Michael B. Chang, Jackie Xiang, Yuan Cao, Nishant Ranka, Geoff Brown, Adrian Hutter, Vahab Mirrokni, Nanxin Chen, Kaisheng Yao, Zoltan Egyed, Francois Gallie, Tyler Liechty, Praveen Kallakuri, Evan Palmer, Sanjay Ghemawat, Jasmine Liu, David Tao, Chloe Thornton, Tim Green, Mimi Jasarevic, Sharon Lin, Victor Cotruta, Yi-Xuan Tan, Noah Fiedel, Hongkun Yu, Ed Chi, Alexander Neitz, Jens Heitkaemper, Anu Sinha, Denny Zhou, Yi Sun, Charbel Kaed, Brice Hulse, Swaroop Mishra, Maria Georgaki, Sneha Kudugunta, Clement Farabet, Izhak Shafran, Daniel Vlasic, Anton Tsitsulin, Rajagopal Ananthanarayanan, Alan Carin, Guolong Su, Pei Sun, Shashank V, Gabriel Carvalaj, Josef Broder, Iulia Comsa, Alena Repina, William Wong, Warren WeiIun Chen, Peter Hawkins, Egor Filonov, Lucia Loher, Christoph Hirschhall, Weiyi Wang, Jingchen Ye, Andrea Burns, Hardie Cate, Diana Gage Wright, Federico Piccinini, Lei Zhang, Chu-Cheng Lin, Ionel Gog, Yana Kulizhskaya, Ashwin Sreevatsa, Shuang Song, Luis C. Cobo, Anand Iyer, Chetan Tekur, Guillermo Garrido, Zhuyun Xiao, Rupert Kemp, Huaxiu Steven Zheng, Hui Li, Ananth Agarwal, Christel Ngani, Kati Goshvadi, Rebeca Santamaria-Fernandez, Wojciech Fica, Xinyun Chen, Chris Gorgolewski, Sean Sun, Roopal Garg, Xinyu Ye, S. M. Ali Esliami, Nan Hua, Jon Simon, Pratik Joshi, Yelin Kim, Ian Tenney, Sahitya Potluri, Lam Nguyen Thiet, Quan Yuan, Florian Luisier, Alexandra Chronopoulou, Salvatore Scellato, Praveen Srinivasan, Minmin Chen, Vinod Koverkathu, Valentin Dalibard, Yaming Xu, Brennan Saeta, Keith Anderson, Thibault Sellam, Nick Fernando, Fantine Huot, Junehyuk Jung, Mani Varadarajan, Michael Quinn, Amit Raul, Maigo Le, Ruslan Habalov, Jon Clark, Komal Jalan, Kalesha Ballard, Achintya Singhal, Thang Luong, Boyu Wang, Sujeewan Rajayogam, Julian Eisenschlos, Johnson Jia, Daniel Finchelstein, Alex Yakubovich, Daniel Balle, Michael Fink, Sameer Agarwal, Jing Li, Dj Dvijotham, Shalini Pal, Kai Kang, Jaclyn Konzelmann, Jennifer Beattie, Olivier Dousse, Diane Wu, Remi Crocker, Chen Elkind, Siddhartha Reddy Jonnalagadda, Jong Lee, Dan Holtmann-Rice, Krystal Kallarackal, Rosanne Liu, Denis Vnukov, Neera Vats, Luca Invernizzi, Mohsen Jafari, Huanjie Zhou, Lilly Taylor, Jennifer Prendki, Marcus Wu, Tom Eccles, Tianqi Liu, Kavya Kopparapu, Francoise Beaufays, Christof Angermueller, Andreea Marzocca, Shourya Sarcar, Hilal Dib, Jeff Stanway, Frank Perbet, Nejc Trdin, Rachel Sterneck, Andrey Khorlin, Dinghua Li, Xihui Wu, Sonam Goenka, David Madras, Sasha Goldshtein, Willi Gierke, Tong Zhou, Yaxin Liu, Yannie Liang, Anais White, Yunjie Li, Shreya Singh, Sanaz Bahargam, Mark Epstein, Sujoy Basu, Li Lao, Adnan Ozturel, Carl Crous, Alex Zhai, Han Lu, Zora Tung, Neeraj Gaur, Alanna Walton, Lucas Dixon, Ming Zhang, Amir Globerson, Grant Yu, Andrew Bolt, Olivia Wiles, Milad Nasr, Ilia Shumailov, Marco Selvi, Francesco Piccinno, Ricardo Aguilar, Sara McCarthy, Misha Khalman, Mrinal Shukla, Vlado Galic, John Carpenter, Kevin Villela, Haibin Zhang, Harry Richardson, James Martens, Matko Bosnjak, Shreyas Rammohan Belle, Jeff Seibert, Mahmoud Alnahlawi, Brian Williams, Sankalp Singh, Annie Louis, Wen Ding, Dan Popovici, Lenin Simicich, Laura Knight, Pulkit Mehta, Nishesh Gupta, Chongyang Shi, Saaber Fatehi, Jovana Mitrovic, Alex Grills, Joseph Pagodara, Tsendsuren Munkhdalai, Dessie Petrova, Danielle Eisenbud, Zhishuai Zhang, Damion Yates, Bhavishya Mittal, Niles Tripramaneni, Yannis Assael, Thomas Broverlli, Prateek Jain, Mihajlo Velimirovic, Canfer Akbulut, Jiaqi Mu, Wolfgang Macherey, Ravin Kumar, Jun Xu, Haroon Qureshi, Gheorghe Comanici, Jeremy Wiesner, Zhiato Gong, Anton Ruddock, Matthias Bauer, Nick Felt, Anirudh GP, Anurag Arnab, Dustin Zelle, Jonas Rothfuss, Bill Rosgen, Ashish Shenoy, Bryan Seybold, Xinjian Li, Jayaram Mudigonda, Goker Erdogan, Jiawei Xia, Jiri Simsa, Andrea Michi, Yi Yao, Christopher Yew, Steven Kan, Isaac Caswell, Carey Radebaugh, Andre Elisseff, Pedro Valenzuela, Kay McKinney, Kim Paterson, Albert Cui, Eri Latorre-Chimoto, Solomon Kim, William Zeng, Ken Durden, Priya Ponnapalli, Tiberiu Sosea, Christopher A. Choquette-Choo, James Manyika, Brona Robeneck, Harsha Vashishth, Sebastien Pereira, Hoi Lam, Marko Velic, Denese Owusu-Afriyie, Katherine Lee, Tolga Bolukbasi, Alicia Parrish, Shawn Lu, Jane Park, Balaji Venkatraman, Alice Talbert, Lambert Rosique, Yuchung Cheng, Andrei Sozanschi, Adam Paszke, Praveen Kumar, Jessica Austin, Lu Li, Khalid Salama, Bartek Perz, Wooyeol Kim, Nandita Dukkipati, Anthony Baryshnikov, Christos Kaplantis, XiangHai Sheng, Yuri Cheruvoni, Caglar Unlu, Diego de Las Casas, Harry Askham, Kathryn Tunyasuvunakool, Felix Gimeno, Siim Poder, Chester Kwak, Matt Miecnikowski, Vahab Mirrokni, Alek Dimitriev, Aaron Parisi, Dangyi Liu, Tomy Tsai, Toby Shevland, Christina Kouridi, Drew Garmon, Adrian Goedeckenmeyer, Adam R. Brown, Anitha Vijayakumar, Ali Elqursh, Sadegh Jazayeri, Jin Huang, Sara Mc Carthy, Jay Hoover, Lucy Kim, Sandeep Kumar, Wei Chen, Courtney Biles, Garrett Bingham, Evan Rosen, Lisa Wang, Qijun Tan, David Engel, Francesco Pongetti, Dario de Cesare, Dongseong Hwang, Lily Yu, Jennifer
Pullman, Srin Narayanan, Kyle Levin, Siddharth Gopal, Megan Li, Asaf Aharoni, Trieu Trinh, Jessica Lo, Norman Casagrande, Roopali Vij, Loic Matthey, Bramandia Ramadhana, Austin Matthews, CJ Carey, Matthew Johnson, Kremena Goranova, Rohin Shah, Shereen Ashraf, Kingshuk Dasgupta, Rasmus Larsen, Yicheng Wang, Manish Reddy Vuyyuru, Chong Jiang, Joana Ijazi, Kazuki Osawa, Celine Smith, Ramya Sree Boppana, Taylan Bilal, Yuma Koizumi, Ying Xu, Yasemin Altun, Nir Shabat, Ben Barach, Alex Korchemnyi, Kiam Choo, Olaf Ronneberger, Chimezie Iwuanyanwu, Shubin Zhao, David Soergel, Cho-Jui Hsieh, Irene Cai, Shariq Iqbal, Martin Sundermeyer, Zhe Chen, Elie Bursztein, Chaitanya Malaviya, Fadi Biadsy, Prakash Shroff, Inderjit Dhillon, Tejasji Latakar, Chris Dyer, Hannah Forbes, Massimo Nicosia, Vitaly Nikolaev, Somer Greene, Marin Georgiev, Pidong Wang, Nina Martin, Hanie Sedghi, John Zhang, Praseem Banzal, Doug Fritz, Vikram Rao, Xuezhi Wang, Jiageng Zhang, Viorica Patraucean, Dayou Du, Igor Mordatch, Ivan Jurin, Lewis Liu, Ayush Dubey, Abhi Mohan, Janek Nowakowski, Vlad-Doru Ion, Nan Wei, Reiko Tojo, Maria Abi Raad, Drew A. Hudson, Vaishakh Keshava, Shubham Agrawal, Kevin Ramirez, Zhichun Wu, Hoang Nguyen, Ji Liu, Madhavi Sewak, Bryce Petrin, DongHyun Choi, Ivan Philips, Ziye Wang, Ioana Bica, Ankush Garg, Jarek Wilkiewicz, Priyanka Agrawal, Xiaowei Li, Danhao Guo, Emily Xue, Naseer Shaik, Andrew Leach, Sadh MMN Khan, Julia Wiesinger, Sammy Jerome, Abhishek Chakladar, Alek Wenjia Wang, Tina Ornduff, Folake Abu, Alireza Ghaffarkhah, Marcus Wainwright, Mario Cortes, Frederick Liu, Joshua Maynez, Andreas Terzis, Pouya Samangouei, Riham Mansour, Tomasz Kepa, Francois-Xavier Aubet, Anton Algymr, Dan Banica, Agoston Weisz, Andras Orban, Alexandre Senges, Ewa Andrejczuk, Mark Geller, Niccolo Dal Santo, Valentin Ankin, Majd Al Merey, Martin Baeuml, Trevor Strohman, Junwen Bai, Slav Petrov, Yonghui Wu, Demis Hassabis, Koray Kavukcuoglu, Jeff Dean, and Oriol Vinyals. 2024a. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. Preprint, arXiv:2403.05530.

Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Riviere, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaël Liu, Francesco Visin, Kathleen Kenealy, Lucas Beyer, Xiaohai Zhai, Anton Tsitsulin, Robert Busa-Fekete, Alex Feng, Noveen Sachdeva, Benjamin Coleman, Yi Gao, Basil Mustafa, Iain Barr, Emilio Parisotto, David Tian, Matan Eyal, Colin Cherry, Jan-Thorsten Peter, Danila Sinopalnikov, Surya Bhupatiraju, Rishabh Agarwal, Mehran Kazemi, Dan Malkin, Ravin Kumar, David Vilar, Idan Brusilovsky, Jiaming Luo, Andreas Steiner, Abe Friesen, Abhanshu Sharma, Abheesht Sharma, Adi Mayrav Gilady, Adrian Goedeckemeyer, Alaa Saade, Alex Feng, Alexander Kolesnikov, Alexei Bendebury, Alvin Abadic, Amit Vadi, András György, André Susano Pinto, Anil Das, Ankur Bapna, Antoine Miech, Antoine Yang, Antonia Paterson, Ashish Shenoy, Ayan Chakrabarti, Bilal Piot, Bo Wu, Bobak Shahriari, Bryce Petrin, Charlie Chen, Charline Le Lan, Christopher A. Choquette-Choo, CJ Carey, Cormac Brick, Daniel Deutsch, Danielle Eisenbud, Dee Cattle, Derek Cheng, Dimitris Paparas, Divyashree Shivakumar Sreepathihalli, Doug Reid, Dustin Tran, Dustin Zelle, Eric Noland, Erwin Huizenga, Eugene Kharitonov, Frederick Liu, Gagik Amirkhanyan, Glenn Cameron, Hadi Hashemi, Hanna Klimczak-Plucińska, Harman Singh, Harsh Mehta, Harshal Tushar Lehri, Hussein Hazimeh, Ian Ballantyne, Idan Szpektor, Ivan Nardini, Jean Pouget-Abadie, Jetha Chan, Joe Stanton, John Wieting, Jonathan Lai, Jordi Orbay, Joseph Fernandez, Josh Newlan, Ju yeong Ji, Jyotinder Singh, Kat Black, Kathy Yu, Kevin Hui, Kiran Vodrahalli, Klaus Greff, Linhai Qiu, Marcela Valentine, Marina Coelho, Marvin Ritter, Matt Hoffman, Matthew Watson, Mayank Chaturvedi, Michael Moynihan, Min Ma, Nabila Babar, Natasha Noy, Nathan Byrd, Nick Roy, Nikola Momchev, Nilay Chauhan, Noveen Sachdeva, Oskar Bunyan, Pankil Botarda, Paul Caron, Paul Kishen Rubenstein, Phil Colliton, Philipp Schmid, Pier Giuseppe Sessa, Pingmei Xu, Piotr Stanczyk, Pouya Tafti, Rakesh Shivanna, Renjie Wu, Renke Pan, Reza Rotki, Rob Willoughby, Rohith Vallu, Ryan Mullins, Sammy Jerome, Sara Smoot, Sertan Girgin, Shariq Iqbal, Shashir Reddy, Shruti Sheth, Siim Põder, Sijal Bhatnagar, Sindhu Raghuram Panyam, Sivan Eiger, Susan Zhang, Tianqi Liu, Trevor Yogevone, Tyler Liechty, Uday Kalra, Utku Evci, Vedant Misra, Vincent Roseberry, Vlad Feinberg, Vlad Kolesnikov, Woohyun Han, Woosuk Kwon, Xi Chen, Yinlarn Chow, Yuvein Zhu, Zichuan Wei, Zoltan Egyed, Victor Crouta, Minh Giang, Phoebe Kirk, Anand Rao, Kat Black, Nabila Babar, Jessica Lo, Erica Moreira, Luiz Gustavo Martins, Omar Sanseverio, Lucas Gonzalez, Zach Gleicher, Tris Warwentin, Vahab Mirrokni, Evan Senter, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, Yossi Matias, D. Sculley, Slav Petrov, Noah Fiedel, Noam Shazeer, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Jean-Baptiste Alayrac, Rohan Anil, Dmitry Lepikhin, Sebastian Borgeaud, Olivier Bachem, Armand Joulin, Alek Andreev, Cassidy Hardin, Robert Dadashi, and Léonard Hussonot. 2025. Gemma 3 technical report. Preprint, arXiv:2503.19786.

Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussonot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton,
Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette-Choo, Danila Sinopalnikov, David Weinberger, Dimple Vijaykumar, Dominika Rogozinska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimczak-Plucińska, Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leticia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Görner, Mat Velloso, Mateo Wirth, Matt Davidson, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moynihan, Ming Zhang, Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khatwani, Natalie Dao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culliton, Pradeep Kuppala, Ramona Comanescu, Ramona Merhej, Reena Jana, Reza Ardeshir Rokni, Rishabh Agarwal, Ryan Mullins, Samaneh Saadat, Sara Mc Carthy, Sarah Cogan, Sarah Perrin, Sébastien M. R. Arnold, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomas Kocisky, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotruta, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev. 2024b. Gemma 2: Improving open language models at a practical size. Preprint, arXiv:2408.00118.

Ruanni Tupas and Isabel Pefianco Martin. 2017. Bilingual and mother tongue-based multilingual education in the Philippines. Bilingual and multilingual education, 10:247–258.

Charlyn Villavicencio, Julio Jerison Macrohon, X Alphonse Inbaraj, Jyh-Horng Jeng, and Jer-Guang Hsieh. 2021. Twitter sentiment analysis towards COVID-19 vaccines in the Philippines using Naïve Bayes. Information, 12(5):204.

Karthik Visweswariah, Rajakrishnan Rajkumar, Ankur Gandhe, Ananthakrishnan Ramanathan, and Jiří Navrátil. 2011. A word reordering model for improved machine translation. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 486–496.

Bin Wang, Zhengyuan Liu, Xin Huang, Fangkai Jiao, Yang Ding, AiTi Aw, and Nancy Chen. 2024. SeaEval for multilingual foundation models: From cross-lingual alignment to cultural reasoning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 370–390, Mexico City, Mexico. Association for Computational Linguistics.

Muhammad Kiki Wardana, Dwi Widayati, Rostina Taib, and Muhammad Iqbal. 2022. Lexicostatistics of Malay, Tagalog and Ilocano languages: A Comparisional Historical Linguistic Study. Jurnal Education and Development, 10(3):475–479.

Tristan Koh Ly Wey. 2024. Current LLM evaluations do not sufficiently measure all we need.

Bryan Wilie, Karissa Vincentio, Genta Indra Winata, Samuel Cahyawijaya, Xiaohong Li, Zhi Yuan Lim, Sidik Soleman, Rahmad Mahendra, Pascale Fung, Syafri Bahar, and Ayu Purwarianti. 2020. IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 843–857, Suzhou, China. Association for Computational Linguistics.

John U Wolff. 2001. The influence of Spanish on Tagalog. Propio y lo ajeno en las lenguas austronésicas y amerindias: procesos interculturales en el contacto de lenguas indígenas con el español en el Pacífico e Hispanoamérica, pages 233–252.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruizhe Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. 2024. Qwen2 technical report. Preprint, arXiv:2407.10671.
Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E Gonzalez, and Ion Stoica. 2023. Rethinking benchmark and contamination for language models with rephrased samples. arXiv preprint arXiv:2311.04850.

Alvin Yapan. 2017. Bagay: Gabay sa pagsulat sa wikang Filipino. BlueBooks.

Maria Sheila Zamar. 2022. Filipino: An Essential Grammar. Routledge.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating Text Generation with BERT. In Proceedings of the 8th International Conference on Learning Representations (ICLR 2020), Addis Ababa, Ethiopia.

Wenxuan Zhang, Hou Pong Chan, Yiran Zhao, Mahani Aljunied, Jianyu Wang, Chaoqun Liu, Yue Deng, Zhiqiang Hu, Weiwen Xu, Yew Ken Chia, Xin Li, and Lidong Bing. 2024. Sealms 3: Open foundation and chat multilingual large language models for southeast asian languages. Preprint, arXiv:2407.19672.

Yuan Zhang, Jason Baldridge, and Luheng He. 2019. PAWS: Paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1298–1308, Minneapolis, Minnesota. Association for Computational Linguistics.

Yiran Zhao, Chaoqun Liu, Yue Deng, Jiahao Ying, Mahani Aljunied, Zhaodonghui Li, Lidong Bing, Hou Pong Chan, Yu Rong, Deli Zhao, and Wenxuan Zhang. 2025. Babel: Open multilingual large language models serving over 90% of global speakers. Preprint, arXiv:2503.00865.
A Overview of Tasks and Datasets

As discussed in Section 3 of this paper, the selection of the eight tasks are presented below. We also provide Table 3 describe the source datasets used for these tasks. The modifications and adaptations that were applied and the usage of the datasets for evaluation comply with their original intended uses.

Abstractive Summarization (AS). An LLM performing this task is given a paragraph and is expected to summarize the content in a sentence. The model is tested not only for identifying the salient points of the text, but also paraphrasing the content into a concise and coherent text. For this task, we use XL-Sum (Hasan et al., 2021), a collection of annotated article-summary pairs.

Causal Reasoning (CR). This task requires the LLM to understand the relationship between events. In particular, the model is given a premise, a set of statements, and an instruction to determine which of the provided statements is the cause or effect of the premise. We employed Balanced COPA (Kavumba et al., 2019), a dataset designed to evaluate commonsense causal reasoning with paired alternatives.

Machine Translation (MT). For this task, an LLM is given a text in one language, and is expected to provide the equivalent text translated into another language. In this study, we test for both English→Filipino and Filipino→English translation. This task leveraged the Filipino subset of FLORES 200 (NLLB Team et al., 2022), which includes translations across numerous languages and domains.

Natural Language Inference (NLI). This is classification task where an LLM is provided two sentences (X and Y), and is expected to determine whether the sentences are related in one in the following ways: (a) X implies Y; (b) X contradicts Y; and (c) X neither implies nor contradicts Y. This task utilized XNLI (Conneau et al., 2018), a dataset containing human-annotated examples for evaluating cross-lingual inference.

Paraphrase Identification (PI). This task requires an LLM to determine if two provided texts are paraphrased versions of each other; that is, whether both pieces of text convey the same idea. For this task, we used PAWS (Zhang et al., 2019), which contains paraphrase and non-paraphrase pairs with high lexical overlap.

Question Answering (QA). Given a passage and a question, an LLM performing this task must provide a span from the passage that answers the question. In this study, QA utilized Belebele (Bandarkar et al., 2024), a multiple-choice reading comprehension dataset designed to assess understanding of passages.

Toxicity Detection (TD) and Sentiment Analysis (SA). These two NLU tasks both require an LLM to analyze a natural language text. The TD task requires the model to identify whether hate speech and abusive language is used in the text, while the SA task requires the model to classify the text as either positive, negative, or neutral in sentiment polarity. Both tasks are derived from the Philippine election-related tweets dataset (Cabasag et al., 2019), which provided a resource for toxicity in political discourse. These tweets were collected during the 2016 presidential campaign. This dataset was also relabeled for sentiment analysis by three native Filipino speakers. There is substantial agreement between the annotations (Cohen’s kappa of 0.8202, Krippendorf’s alpha of 0.8268).

<table>
  <tr>
    <th>Competency</th>
    <th>Task</th>
    <th>Dataset</th>
    <th>License</th>
  </tr>
  <tr>
    <td rowspan="4">NLU</td>
    <td>PI</td>
    <td>PAWS (Zhang et al., 2019)</td>
    <td>CC BY 4.0</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>Belebele (Bandarkar et al., 2024)</td>
    <td>CC BY-NC 4.0</td>
  </tr>
  <tr>
    <td>SA</td>
    <td>PH Elections (Cabasag et al., 2019)</td>
    <td>Unknown</td>
  </tr>
  <tr>
    <td>TD</td>
    <td>PH Elections (Cabasag et al., 2019)</td>
    <td>Unknown</td>
  </tr>
  <tr>
    <td rowspan="2">NLR</td>
    <td>CR</td>
    <td>Balanced COPA (Kavumba et al., 2019)</td>
    <td>CC BY 4.0</td>
  </tr>
  <tr>
    <td>NLI</td>
    <td>XNLI (Conneau et al., 2018)</td>
    <td>CC BY-NC 4.0</td>
  </tr>
  <tr>
    <td rowspan="2">NLG</td>
    <td>AS</td>
    <td>XL-Sum (Hasan et al., 2021)</td>
    <td>CC BY-NC-SA 4.0</td>
  </tr>
  <tr>
    <td>MT</td>
    <td>FLORES 200 (NLLB Team et al., 2022)</td>
    <td>CC BY-SA 4.0</td>
  </tr>
</table>

Table 3: License details for datasets used in BATAYAN.

Table 4 presents quantitative statistics on dataset size and average word counts. The variation in text length reflects task-specific requirements: AS involves longer passages, while classification tasks (SA, TD, and PI) typically contain more concise text samples. The QA dataset, in particular, has relatively
<table>
  <tr>
    <th>Competency</th>
    <th>Task</th>
    <th># test samples</th>
    <th>Avg. No. Words/Sample</th>
  </tr>
  <tr>
    <td rowspan="4">NLU</td>
    <td>PI</td>
    <td>400</td>
    <td>22.50 (Sentence 1), 22.47 (Sentence 2)</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>100</td>
    <td>14.75 (Question), 4.26 (Choices)</td>
  </tr>
  <tr>
    <td>SA</td>
    <td>600</td>
    <td>24.32</td>
  </tr>
  <tr>
    <td>TD</td>
    <td>400</td>
    <td>20.73</td>
  </tr>
  <tr>
    <td rowspan="2">NLR</td>
    <td>CR</td>
    <td>400</td>
    <td>7.24 (Premise), 6.05 (Choices)</td>
  </tr>
  <tr>
    <td>NLI</td>
    <td>600</td>
    <td>23.30 (Sentence 1), 12.20 (Sentence 2)</td>
  </tr>
  <tr>
    <td rowspan="3">NLG</td>
    <td>AS</td>
    <td>100</td>
    <td>120.8</td>
  </tr>
  <tr>
    <td>MT (English Text)</td>
    <td>600</td>
    <td>21.42</td>
  </tr>
  <tr>
    <td>MT (Tagalog Text)</td>
    <td>600</td>
    <td>25.05</td>
  </tr>
</table>

Table 4: Summary of dataset statistics, including total rows and average words per text.

short questions and answer choices, mirroring real-world multiple-choice assessments. Further granularity is provided in Table 5, which breaks down sentence, word, and character-level statistics. This statistics shows structural differences between tasks; for instance, MT has sentence-level parallelism, while CR and NLI involve distinct premise-hypothesis or question-choice relationships.

<table>
  <tr>
    <th>Competency</th>
    <th>Task</th>
    <th>Component</th>
    <th>Avg. No. Sentences</th>
    <th>Avg. No. Words</th>
    <th>Avg. No. Characters</th>
  </tr>
  <tr>
    <td rowspan="9">NLU</td>
    <td>PI</td>
    <td>Sentence 1</td>
    <td>1.0</td>
    <td>22.50</td>
    <td>123.01</td>
  </tr>
  <tr>
    <td>PI</td>
    <td>Sentence 2</td>
    <td>1.0</td>
    <td>22.47</td>
    <td>122.99</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>Question</td>
    <td>1.0</td>
    <td>14.75</td>
    <td>80.62</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>Choice 1</td>
    <td>1.0</td>
    <td>4.66</td>
    <td>29.10</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>Choice 2</td>
    <td>1.0</td>
    <td>4.61</td>
    <td>28.60</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>Choice 3</td>
    <td>1.0</td>
    <td>4.76</td>
    <td>29.74</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>Choice 4</td>
    <td>1.0</td>
    <td>3.0</td>
    <td>28.58</td>
  </tr>
  <tr>
    <td>SA</td>
    <td>Text</td>
    <td>2.21</td>
    <td>24.32</td>
    <td>120.44</td>
  </tr>
  <tr>
    <td>TD</td>
    <td>Text</td>
    <td>2.45</td>
    <td>20.73</td>
    <td>99.37</td>
  </tr>
  <tr>
    <td rowspan="4">NLR</td>
    <td>CR</td>
    <td>Premise</td>
    <td>1.0</td>
    <td>7.24</td>
    <td>37.55</td>
  </tr>
  <tr>
    <td>CR</td>
    <td>Choice 1</td>
    <td>1.0</td>
    <td>6.05</td>
    <td>30.91</td>
  </tr>
  <tr>
    <td>CR</td>
    <td>Choice 2</td>
    <td>1.0</td>
    <td>6.05</td>
    <td>30.62</td>
  </tr>
  <tr>
    <td>NLI</td>
    <td>Sentence 1</td>
    <td>1.0</td>
    <td>23.30</td>
    <td>127.88</td>
  </tr>
  <tr>
    <td>NLI</td>
    <td>Sentence 2</td>
    <td>1.0</td>
    <td>12.20</td>
    <td>67.04</td>
  </tr>
  <tr>
    <td rowspan="3">NLG</td>
    <td>AS</td>
    <td>Text</td>
    <td>5.43</td>
    <td>120.8</td>
    <td>666.18</td>
  </tr>
  <tr>
    <td>MT</td>
    <td>English Text</td>
    <td>1.12</td>
    <td>21.42</td>
    <td>113.07</td>
  </tr>
  <tr>
    <td>MT</td>
    <td>Filipino Text</td>
    <td>1.12</td>
    <td>25.05</td>
    <td>140.58</td>
  </tr>
</table>

Table 5: Detailed dataset statistics, including sentence, word, and character counts.
B  Dataset Examples

<table>
  <tr>
    <th>Competency</th>
    <th>Task</th>
    <th>Example</th>
  </tr>
  <tr>
    <td>NLU</td>
    <td>PI</td>
    <td><b>Sentence 1:</b> Wala pang isang taon matapos ang kaniyang nagdaaang kasal, pinakasalan din ni Charlemagne si Desiderata at tinanggihan ang 13-anyos na Swabian na nagngangalang Hildegard.<br><br><b>Sentence 2:</b> Pinakasalan ni Charlemagne si Desiderata wala pang isang taon pagkatapos ng kanyang kasal at isinawalang-bahala ang isang 13 taong gulang na Swabian na nagngangalang Hildegard.<br><br><b>Label:</b> Paraphrase (Paraphrase)</td>
  </tr>
  <tr>
    <td>QA</td>
    <td></td>
    <td><b>Text:</b> Lumipat ang mga hukbong Coalition at Afghan sa lugar na iyon upang tiyaking ligtas ang lokasyon, at nagpadala ng iba pang eroplano ng koalisyon upang tumulong. Naganap ang pagbagsak sa mataas na bahagi ng kabundukan, at pinanimiwalaang resulta ng pagbabaril ng kalaban. Isang malaking hamon ang masamang panahon at malubak na daan sa paghahanap.<br><br><b>Question:</b> Ano ang pinanimiwalaang dahilan ng pagbagsak?<br><br><b>Choices:</b> (1) Pangit na daan (2) Masamang sunog (3) Mabundok na lupain (4) Masamang lagay ng panahon<br><br><b>Label:</b> 1</td>
  </tr>
  <tr>
    <td>SA</td>
    <td></td>
    <td><b>Text:</b> Pacha.. PURO DILAWAN DITO SA REDDIT AH HAHAHA.. AKALA NYO NAMAN ANG LAKI NA NG ACCOMPLISHMENT NYO DAHIL SA VIDEO NA YAN MGA GUNGGONG!! HAHAHA #DU50 PA RIN!! MGA TAE KAYO!<br><br><b>Label:</b> Negativo (Negative)</td>
  </tr>
  <tr>
    <td>TD</td>
    <td></td>
    <td><b>Text:</b> Ayun, nag-Filipino rin si Poe, ang presidente ko! Kitang-kita ang kanyang platapormang maka-mahirap at maka-tao. #PiliPinasDebates2016<br><br><b>Label:</b> Malinis (Clean)</td>
  </tr>
  <tr>
    <td>NLR</td>
    <td>CR</td>
    <td><b>Text:</b> Hindi tinanggap ang tsekeng ginawa ko.<br><br><b>Question:</b> Sanhi (Cause)<br><br><b>Choices:</b> (0) Walang laman ang bank account ko. (1) Tumaas ang aking suweldo.<br><br><b>Label:</b> 0</td>
  </tr>
  <tr>
    <td>NLI</td>
    <td></td>
    <td><b>Sentence 1:</b> Di mo ba natatandaan? Pupunta tayo ngayong araw sa birthday ni Tita Basia.<br><br><b>Sentence 2:</b> Hindi kami pupunta sa birthday party ni Tita Basia ngayon.<br><br><b>Label:</b> Contradiction</td>
  </tr>
  <tr>
    <td>NLG</td>
    <td>AS</td>
    <td><b>Article:</b> Kinumpirma noong nakaraang buwan na humawa na ang virus sa isang tupa sa isla matapos isilang nang patay at wala sa hugs ang limang kordero sa isang bukid. Sinabi ng state veterinary officer na malamang na ang birus ay sanhi ng windborne midges.<br><br><b>Summary:</b> Kinumpirma ng mga pagsusuri noong nakaraang buwan na nahawahan ng Schmallenberg virus ang mga tupa sa isla.</td>
  </tr>
  <tr>
    <td>MT</td>
    <td></td>
    <td><b>Text:</b> Former U.S. Speaker of the House Newt Gingrich came in second with 32 percent.<br><br><b>Translation:</b> Pumangalawa ang dating Speaker of the House ng U.S. na si Newt Gingrich nang may 32 porsyento.</td>
  </tr>
</table>

Table 6: Example instances for each task in BATAYAN.
C  Inter-rater Agreement

<table>
  <tr>
    <th>Competency</th>
    <th>Task</th>
    <th>Dataset</th>
    <th>Criteria</th>
    <th>Joint agreement</th>
    <th>Rating</th>
  </tr>
  <tr>
    <td rowspan="3">NLU</td>
    <td rowspan="3">PI</td>
    <td rowspan="3">PAWS</td>
    <td>Completeness</td>
    <td>0.9550</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Fluency</td>
    <td>0.5875</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Sensibility</td>
    <td>0.8025</td>
    <td>-</td>
  </tr>
  <tr>
    <td rowspan="3">QA</td>
    <td rowspan="3">Belebele</td>
    <td></td>
    <td>Completeness</td>
    <td>0.9700</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td>Fluency</td>
    <td>0.8300</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td>Sensibility</td>
    <td>0.9500</td>
    <td>-</td>
  </tr>
  <tr>
    <td rowspan="3">SA</td>
    <td rowspan="3">PH Election Tweets</td>
    <td></td>
    <td>Completeness</td>
    <td>0.6217</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td>Fluency</td>
    <td>0.6767</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td>Sensibility</td>
    <td>0.7050</td>
    <td>-</td>
  </tr>
  <tr>
    <td rowspan="3">TD</td>
    <td rowspan="3">PH Election Tweets</td>
    <td></td>
    <td>Completeness</td>
    <td>0.8310</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td>Fluency</td>
    <td>0.9060</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td>Sensibility</td>
    <td>0.8680</td>
    <td>-</td>
  </tr>
  <tr>
    <td rowspan="3">NLR</td>
    <td rowspan="3">CR</td>
    <td rowspan="3">Balanced COPA</td>
    <td>Completeness</td>
    <td>0.8975</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Fluency</td>
    <td>0.7275</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Sensibility</td>
    <td>0.9575</td>
    <td>-</td>
  </tr>
  <tr>
    <td rowspan="3">NLI</td>
    <td rowspan="3">XNLI</td>
    <td></td>
    <td>Completeness</td>
    <td>0.9716</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td>Fluency</td>
    <td>0.8433</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td>Sensibility</td>
    <td>0.9933</td>
    <td>-</td>
  </tr>
  <tr>
    <td rowspan="4">NLG</td>
    <td rowspan="4">AS</td>
    <td rowspan="4">XL-Sum</td>
    <td>Completeness</td>
    <td>0.8800</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Fluency</td>
    <td>0.9600</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Faithfulness</td>
    <td>1.0000</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Relevance of summary</td>
    <td>-</td>
    <td>1.99</td>
  </tr>
  <tr>
    <td rowspan="3">MT</td>
    <td rowspan="3">FLORES 200</td>
    <td></td>
    <td>Fluency of summary</td>
    <td>-</td>
    <td>2.64</td>
  </tr>
  <tr>
    <td></td>
    <td>Coherence of summary</td>
    <td>-</td>
    <td>2.56</td>
  </tr>
  <tr>
    <td></td>
    <td>Completeness</td>
    <td>0.7100</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td>Fluency</td>
    <td>0.9900</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td>Sensibility</td>
    <td>0.7750</td>
    <td>-</td>
  </tr>
</table>

Table 7: Joint agreement is calculated as the percentage of times all the raters unanimously agree that the samples fulfill the criteria. Rating is calculated as the average score assigned by the raters to the samples under the given Likert-scaled (0-3) criteria.
<table>
  <tr>
    <th>Task</th>
    <th>Filipino Prompt Template</th>
    <th>SEA-HELM English Prompt Template</th>
  </tr>
  <tr>
    <td>PI</td>
    <td>Bibigyan ka ng dalawang pangungusap, SENTENCE_1 at SENTENCE_2. Tukuyin kung alin sa sumusunod na pahayag ang pinakangkop para sa SENTENCE_1 at SENTENCE_2.<br>A: Paraprarse ang SENTENCE_2 ng SENTENCE_1.<br>B: Hindi paraprarse ang SENTENCE_2 ng SENTENCE_1.<br><br>Sumagot gamit ang sumusunod na format.<br>Sagot: $OPTION<br>Palitan ang $OPTION ng napiling sagot.<br>Gumamit lang ng titik A o B sa sagot mo.{fewshot_examples}<br><br>SENTENCE_1:<br>...<br>{sentence1}<br>...<br><br>SENTENCE_2:<br>...<br>{sentence2}<br>...</td>
    <td>You will be given two sentences, SENTENCE_1 and SENTENCE_2. Determine which of the following statements applies to SENTENCE_1 and SENTENCE_2 the best.<br>A: SENTENCE_2 is a paraphrase of SENTENCE_1.<br>B: SENTENCE_2 is not a paraphrase of SENTENCE_1.<br><br>Answer only using the following format:<br>Answer: $OPTION<br>Replace $OPTION with the selected option.<br>Use the letters A or B only as the answer.{fewshot_examples}<br><br>SENTENCE_1:<br>...<br>{sentence1}<br>...<br><br>SENTENCE_2:<br>...<br>{sentence2}<br>...</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>Bibigyan ka ng isang talata, isang tanong, at apat na pagpipiliang sagot. Sumagot base sa talata sa pamamagitan ng pagpili ng isa sa mga opsiyong ibinigay.<br><br>Sumagot gamit ang sumusunod na format:<br>Sagot: $OPTION<br>Palitan ang $OPTION ng napiling sagot.<br>Gumamit lang ng titik A, B, C, o D sa sagot mo.{fewshot_examples}<br><br>Talata:<br>...<br>{text}<br>...<br><br>Tanong: {question}<br>A: {choice1}<br>B: {choice2}<br>C: {choice3}<br>D: {choice4}</td>
    <td>You will be given one paragraph, one question and four answer options. Based on the paragraph, answer by choosing one of the options given.<br><br>Answer only using the following format:<br>Answer: $OPTION<br>Replace $OPTION with the selected option.<br>Use the letter A, B, C, or D as the answer.{fewshot_examples}<br><br>Paragraph:<br>...<br>{text}<br>...<br><br>Question: {question}<br>A: {choice1}<br>B: {choice2}<br>C: {choice3}<br>D: {choice4}</td>
  </tr>
  <tr>
    <td>SA</td>
    <td>Ano ang sentimemento sa sumusunod na pangungusap? Sumagot gamit ng isa sa mga sumusunod na pagpipilian: Positibo, Negatibo, o Neutral.<br><br>Sumagot gamit ang sumusunod na format:<br>Sagot: $OPTION<br>Palitan ang $OPTION ng napiling sagot.{fewshot_examples}<br><br>Pangungusap:<br>...<br>{text}</td>
    <td>What is the sentiment of the following sentence? Use one of the choices below: Positive, Negative, or Neutral.<br><br>Answer only using the following format:<br>Answer: $OPTION<br>Replace $OPTION with the selected option.{fewshot_examples}<br><br>Sentence:<br>...<br>{text}</td>
  </tr>
</table>
TD  Isa kang toxicity detector. Nangangahulugan ang mga sumusunod na label:
Malinis: Walang toxicity.
Mapoot: May toxicity at direktang pag-atake sa isang tao.

Batay sa kahulugan ng mga label, iuri ang mga sumusunod na pangungusap gamit lang ng isang salita: Malinis o Mapoot.

Sumagot gamit ang sumusunod na format:
Sagot: $LABEL
Palitan ang $LABEL ng napiling label.
{fewshot_examples}

Pangungusap:
...
{text}
...

You are a toxicity detector. The following labels are defined as follows:
Clean: There is no toxicity.
Toxic: There is toxicity and direct attacks towards a person.

Based on the label definitions, classify the following sentence only using one word:
Clean or Toxic.

Answer only using the following format:
Answer: $LABEL
Replace $LABEL with the selected label.
{fewshot_examples}

Sentence:
...
{text}
...

CR  Sumagot gamit ang sumusunod na format:
Sagot: $OPTION
Palitan ang $OPTION ng napiling sagot.
Gumamit lang ng titik A or B sa sagot mo.
{fewshot_examples}

Batay sa ibibigay na sitwasyon, alin sa sumusunod na pagpipilian ang mas maaari na {sanhi/bunga}?

Sitwasyon:
...
{text}
...

Piliin ang pinaka-angkop na sagot mula sa sumusunod na pagpipilian:
A: {choice1}
B: {choice2}

Answer only using the following format:
Answer: $OPTION
Replace $OPTION with the selected option.
Use only the letters A or B as the answer.
{fewshot_examples}

Based on the given situation, which of the following options is more likely to be the {cause/effect}?

Situation:
...
{text}
...

Choose the best answer from the following options:
A: {choice1}
B: {choice2}

NLI  Bibigyan ka ng dalawang pangungusap,
SENTENCE_1 at SENTENCE_2. Tukuyin kung alin sa sumusunod na pahayag ang pinaka-angkop para sa SENTENCE_1 at SENTENCE_2.
A: Kung totoo ang SENTENCE_1, dapat totoo din ang SENTENCE_2.
B: Sumasalungat ang SENTENCE_1 sa SENTENCE_2.
C: Kapag totoo ang SENTENCE_1, pwedeng totoo o hindi totoo ang SENTENCE_2.

Sumagot gamit ang sumusunod na format.
Sagot: $OPTION
Palitan ang $OPTION ng napiling sagot.
Gumamit lang ng titik A, B, o C sa sagot mo.{fewshot_examples}

SENTENCE_1:
...
{sentence1}
...
SENTENCE_2:
...
{sentence2}
...

You will be given two sentences, SENTENCE_1 and SENTENCE_2. Determine which of the following statements applies to SENTENCE_1 and SENTENCE_2 the best.
A: If SENTENCE_1 is true, SENTENCE_2 must be true.
B: SENTENCE_1 contradicts SENTENCE_2.
C: When SENTENCE_1 is true, SENTENCE_2 may or may not be true.

Answer only using the following format:
Answer: $OPTION
Replace $OPTION with the selected option.
Use the letters A, B or C only as the answer.{fewshot_examples}

SENTENCE_1:
...
{sentence1}
...
SENTENCE_2:
...
{sentence2}
...
<table>
  <tr>
    <th>AS</th>
    <td>Ibued ang sumusunod na artikulong Filipino sa isang talata na may isa o dalawang pangungusap.<br><br>Sumagot gamit ang sumusunod na format:<br>Buod: $SUMMARY<br>Palitan ang $SUMMARY ng buod.<br>{fewshot_examples}<br><br>Artikulo:<br>```<br>{text}<br>```</td>
    <td>Summarize the following {language} article into a paragraph with 1 or 2 sentences.<br><br>Answer only using the following format:<br>Summary: $SUMMARY<br>Replace $SUMMARY with the summary.<br>{fewshot_examples}<br><br>Article:<br>```<br>{text}<br>```</td>
  </tr>
  <tr>
    <th>MT</th>
    <td>Isalin ang sumusunod na teksto sa {language}.<br><br>Sumagot gamit ang sumusunod na format:<br>Salin: $TRANSLATION<br>Palitan ang $TRANSLATION ng isinalin na teksto.{fewshot_examples}<br><br>Teksto:<br>```<br>{text}<br>```</td>
    <td>Translate the following text into {language}.<br><br>Answer only using the following format:<br>Translation: $TRANSLATION<br>Replace $TRANSLATION with the translated text.{fewshot_examples}<br><br>Text:<br>```<br>{text}<br>```</td>
  </tr>
</table>

Table 8: Prompt templates used in evaluating LLMs on BATAYAN. Their corresponding English-language prompt templates are also provided.
<table>
  <tr>
    <th>Models</th>
    <th>Source Link</th>
    <th>No. params</th>
  </tr>
  <tr>
    <th colspan="3">Base pre-trained models</th>
  </tr>
  <tr>
    <td>asingapore/Gemma-SEA-LION-v3-9B</td>
    <td>https://huggingface.co/asingapore/Gemma-SEA-LION-v3-9B</td>
    <td>9B</td>
  </tr>
  <tr>
    <td>asingapore/Llama-SEA-LION-v2-8B</td>
    <td>https://huggingface.co/asingapore/Llama-SEA-LION-v2-8B</td>
    <td>8B</td>
  </tr>
  <tr>
    <td>asingapore/Llama-SEA-LION-v3-8B</td>
    <td>https://huggingface.co/asingapore/Llama-SEA-LION-v3-8B</td>
    <td>8B</td>
  </tr>
  <tr>
    <td>asingapore/Llama-SEA-LION-v3-70B</td>
    <td>https://huggingface.co/asingapore/Llama-SEA-LION-v3-70B</td>
    <td>70B</td>
  </tr>
  <tr>
    <td>google/gemini-2-9b</td>
    <td>https://huggingface.co/google/gemini-2-9b</td>
    <td>9B</td>
  </tr>
  <tr>
    <td>google/gemini-3-12b</td>
    <td>https://huggingface.co/google/gemini-3-12b</td>
    <td>12B</td>
  </tr>
  <tr>
    <td>google/gemini-3-12b-pt</td>
    <td>https://huggingface.co/google/gemini-3-12b-pt</td>
    <td>12B</td>
  </tr>
  <tr>
    <td>google/gemini-3-27b-pt</td>
    <td>https://huggingface.co/google/gemini-3-27b-pt</td>
    <td>27B</td>
  </tr>
  <tr>
    <td>meta-llama/Llama-3-1.8B</td>
    <td>https://huggingface.co/meta-llama/Llama-3-1.8B</td>
    <td>8B</td>
  </tr>
  <tr>
    <td>meta-llama/Llama-3-1.70B</td>
    <td>https://huggingface.co/meta-llama/Llama-3-1.70B</td>
    <td>70B</td>
  </tr>
  <tr>
    <td>Qwen/Qwen2-72B</td>
    <td>https://huggingface.co/Qwen/Qwen2-72B</td>
    <td>72B</td>
  </tr>
  <tr>
    <td>Qwen/Qwen2-5.7B</td>
    <td>https://huggingface.co/Qwen/Qwen2-5.7B</td>
    <td>7B</td>
  </tr>
  <tr>
    <td>Qwen/Qwen2-5.14B</td>
    <td>https://huggingface.co/Qwen/Qwen2-5.14B</td>
    <td>14B</td>
  </tr>
  <tr>
    <td>Qwen/Qwen2-5.32B</td>
    <td>https://huggingface.co/Qwen/Qwen2-5.32B</td>
    <td>32B</td>
  </tr>
  <tr>
    <td>Qwen/Qwen2-5.72B</td>
    <td>https://huggingface.co/Qwen/Qwen2-5.72B</td>
    <td>72B</td>
  </tr>
  <tr>
    <td>sail/Sailor2-8B</td>
    <td>https://huggingface.co/sail/Sailor2-8B</td>
    <td>8B</td>
  </tr>
  <tr>
    <td>sail/Sailor2-20B</td>
    <td>https://huggingface.co/sail/Sailor2-20B</td>
    <td>20B</td>
  </tr>
  <tr>
    <td>Scal.LMs/Scal.LMs-v3-7B</td>
    <td>https://huggingface.co/Scal.LMs/Scal.LMs-v3-7B</td>
    <td>7B</td>
  </tr>
  <tr>
    <th colspan="3">Instruction-tuned models</th>
  </tr>
  <tr>
    <td>asingapore/Gemma-SEA-LION-v3-9B-IT</td>
    <td>https://huggingface.co/asingapore/Gemma-SEA-LION-v3-9B-IT</td>
    <td>9B</td>
  </tr>
  <tr>
    <td>asingapore/Llama-SEA-LION-v2-8B-IT</td>
    <td>https://huggingface.co/asingapore/Llama-SEA-LION-v2-8B-IT</td>
    <td>8B</td>
  </tr>
  <tr>
    <td>asingapore/Llama-SEA-LION-v3-8B-IT</td>
    <td>https://huggingface.co/asingapore/Llama-SEA-LION-v3-8B-IT</td>
    <td>8B</td>
  </tr>
  <tr>
    <td>asingapore/Llama-SEA-LION-v3-70B-IT</td>
    <td>https://huggingface.co/asingapore/Llama-SEA-LION-v3-70B-IT</td>
    <td>70B</td>
  </tr>
  <tr>
    <td>CohereForAI/ayas-23-35B</td>
    <td>https://huggingface.co/CohereForAI/ayas-23-35B</td>
    <td>35B</td>
  </tr>
  <tr>
    <td>CohereForAI/ayas-expanse-8B</td>
    <td>https://huggingface.co/CohereForAI/ayas-expanse-8B</td>
    <td>8B</td>
  </tr>
  <tr>
    <td>CohereForAI/ayas-expanse-32B</td>
    <td>https://huggingface.co/CohereForAI/ayas-expanse-32B</td>
    <td>32B</td>
  </tr>
  <tr>
    <td>deepseek.ai/DeepSeek-V3</td>
    <td>https://huggingface.co/deepseek.ai/DeepSeek-V3</td>
    <td>671B</td>
  </tr>
  <tr>
    <td>google/gemini-2-9b-it</td>
    <td>https://huggingface.co/google/gemini-2-9b-it</td>
    <td>9B</td>
  </tr>
  <tr>
    <td>google/gemini-3-12b-it</td>
    <td>https://huggingface.co/google/gemini-3-12b-it</td>
    <td>12B</td>
  </tr>
  <tr>
    <td>google/gemini-3-27b-it</td>
    <td>https://huggingface.co/google/gemini-3-27b-it</td>
    <td>27B</td>
  </tr>
  <tr>
    <td>MERAION/Llama-3-MERAION-8B-Instruct</td>
    <td>https://huggingface.co/MERAION/Llama-3-MERAION-8B-Instruct</td>
    <td>8B</td>
  </tr>
  <tr>
    <td>meta-llama/Llama-3-1.8B-Instruct</td>
    <td>https://huggingface.co/meta-llama/Llama-3-1.8B-Instruct</td>
    <td>8B</td>
  </tr>
  <tr>
    <td>meta-llama/Llama-3-1.70B-Instruct</td>
    <td>https://huggingface.co/meta-llama/Llama-3-1.70B-Instruct</td>
    <td>70B</td>
  </tr>
  <tr>
    <td>meta-llama/Llama-3-70B-Instruct</td>
    <td>https://huggingface.co/meta-llama/Llama-3-70B-Instruct</td>
    <td>70B</td>
  </tr>
  <tr>
    <td>Qwen/Qwen2-72B-Instruct</td>
    <td>https://huggingface.co/Qwen/Qwen2-72B-Instruct</td>
    <td>72B</td>
  </tr>
  <tr>
    <td>Qwen/Qwen2-5.7B-Instruct</td>
    <td>https://huggingface.co/Qwen/Qwen2-5.7B-Instruct</td>
    <td>7B</td>
  </tr>
  <tr>
    <td>Qwen/Qwen2-5.14B-Instruct</td>
    <td>https://huggingface.co/Qwen/Qwen2-5.14B-Instruct</td>
    <td>14B</td>
  </tr>
  <tr>
    <td>Qwen/Qwen2-5.32B-Instruct</td>
    <td>https://huggingface.co/Qwen/Qwen2-5.32B-Instruct</td>
    <td>32B</td>
  </tr>
  <tr>
    <td>Qwen/Qwen2-5.72B-Instruct</td>
    <td>https://huggingface.co/Qwen/Qwen2-5.72B-Instruct</td>
    <td>72B</td>
  </tr>
  <tr>
    <td>sail/Sailor2-20B-Chat</td>
    <td>https://huggingface.co/sail/Sailor2-20B-Chat</td>
    <td>20B</td>
  </tr>
  <tr>
    <td>Scal.LMs/Scal.LMs-v3-7B-Chat</td>
    <td>https://huggingface.co/Scal.LMs/Scal.LMs-v3-7B-Chat</td>
    <td>7B</td>
  </tr>
  <tr>
    <td>Tower-Babel/Babel-9B-Chat</td>
    <td>https://huggingface.co/Tower-Babel/Babel-9B-Chat</td>
    <td>9B</td>
  </tr>
  <tr>
    <td>Tower-Babel/Babel-83B-Chat</td>
    <td>https://huggingface.co/Tower-Babel/Babel-83B-Chat</td>
    <td>83B</td>
  </tr>
  <tr>
    <th colspan="3">Reasoning models</th>
  </tr>
  <tr>
    <td>deepseek.ai/DeepSeek-R1</td>
    <td>https://huggingface.co/deepseek.ai/DeepSeek-R1</td>
    <td>671B</td>
  </tr>
  <tr>
    <th colspan="3">Commercial models</th>
  </tr>
  <tr>
    <td>google/gemini-1.5-flash-002</td>
    <td>https://cloud.google.com/vertex-ai/generative-ai/docs/learn/model-versions</td>
    <td>-</td>
  </tr>
  <tr>
    <td>google/gemini-1.5-pro-002</td>
    <td>https://cloud.google.com/vertex-ai/generative-ai/docs/learn/model-versions</td>
    <td>-</td>
  </tr>
  <tr>
    <td>google/gemini-2.0-flash-001</td>
    <td>https://cloud.google.com/vertex-ai/generative-ai/docs/learn/model-versions</td>
    <td>-</td>
  </tr>
  <tr>
    <td>google/gemini-2.0-flash-lite-001</td>
    <td>https://cloud.google.com/vertex-ai/generative-ai/docs/learn/model-versions</td>
    <td>-</td>
  </tr>
  <tr>
    <td>openai/gpt-4o-mini-2024-07-18</td>
    <td>https://platform.openai.com/docs/models/gpt-4o-mini</td>
    <td>-</td>
  </tr>
  <tr>
    <td>openai/gpt-4o-2024-08-06</td>
    <td>https://platform.openai.com/docs/models/gpt-4o</td>
    <td>-</td>
  </tr>
</table>

Table 9: Links and sizes of models evaluated in our experiments.
<table>
  <tr>
    <th>Model</th>
    <th>Reference</th>
    <th>Filipino IT</th>
    <th>Rationale for selection</th>
  </tr>
  <tr>
    <td>Aya 23 and Expanse</td>
    <td>(Aryabumi et al., 2024; Dang et al., 2024)</td>
    <td></td>
    <td>LLMs with multilingual support. It is unknown if these models were explicitly fine-tuned on Filipino instructions.</td>
  </tr>
  <tr>
    <td>Babel</td>
    <td>(Zhao et al., 2025)</td>
    <td>✓</td>
    <td>LLMs with multilingual support including Filipino. This model was pre-trained and fine-tuned on Filipino tokens.</td>
  </tr>
  <tr>
    <td>DeepSeek V3 and R1</td>
    <td>(DeepSeek-AI, 2025)</td>
    <td></td>
    <td>LLMs with multilingual support. It is unknown if these models were explicitly fine-tuned on Filipino instructions.</td>
  </tr>
  <tr>
    <td>Gemma 2 and 3</td>
    <td>(Team et al., 2024b, 2025)</td>
    <td></td>
    <td>LLMs with multilingual support. It is unknown if these models were explicitly fine-tuned on Filipino instructions.</td>
  </tr>
  <tr>
    <td>Gemini 1.5 and 2</td>
    <td>(Team et al., 2024a)</td>
    <td></td>
    <td>Commercial AI systems with multilingual support. It is unknown if these models were explicitly fine-tuned on Filipino instructions.</td>
  </tr>
  <tr>
    <td>GPT-4o and 4o-mini</td>
    <td>(OpenAI et al., 2024)</td>
    <td></td>
    <td>Commercial AI systems with multilingual support. It is unknown if these models were explicitly fine-tuned on Filipino instructions.</td>
  </tr>
  <tr>
    <td>Llama 3</td>
    <td>(Grattafiori et al., 2024)</td>
    <td></td>
    <td>LLMs with multilingual support. It is unknown if these models were explicitly fine-tuned on Filipino instructions.</td>
  </tr>
  <tr>
    <td>MERaLiON</td>
    <td>(Huang et al., 2025)</td>
    <td></td>
    <td>LLM with support for English, Chinese, and Indonesian. Llama 3 model was continue pre-trained on 120B English, Chinese, and Indonesian tokens. It is unknown if these models were explicitly fine-tuned on Filipino instructions.</td>
  </tr>
  <tr>
    <td>Qwen 2 and 2.5</td>
    <td>(Yang et al., 2024; Qwen et al., 2025)</td>
    <td>✓</td>
    <td>LLMs with multilingual support, including Tagalog.</td>
  </tr>
  <tr>
    <td>Sailor 2</td>
    <td>(Dou et al., 2025)</td>
    <td>✓</td>
    <td>LLMs with support for Southeast Asian languages. Qwen 2.5 7B model was continue pre-trained on 400B SEA tokens, and was specifically fine-tuned on Filipino instructions.</td>
  </tr>
  <tr>
    <td>SEA-LION v2 and v3</td>
    <td>(Ng et al., 2025)</td>
    <td>✓</td>
    <td>LLMs with support for Southeast Asian languages. Llama 3 and Gemma 2 series models were continue pre-trained on 200B tokens across 11 Southeast Asian languages, and was specifically fine-tuned on Filipino instructions.</td>
  </tr>
  <tr>
    <td>SeaLLMs v3</td>
    <td>(Zhang et al., 2024)</td>
    <td>✓</td>
    <td>LLMs with support for Southeast Asian languages. These models were pre-trained and fine-tuned on Filipino tokens.</td>
  </tr>
</table>

Table 10: Descriptions of model series evaluated in our experiments.
F  Experimental Setup

The default evaluation setting for BATAYAN is zero-shot prompting, however base pre-trained models with no instruction-tuning are evaluated on a five-shot setting. Performance scores are calculated based on a single run. Table 11 details the decoding parameters used in the experiments.

<table>
  <tr>
    <th>Competency</th>
    <th>Task</th>
    <th>max_tokens</th>
    <th>temperature</th>
    <th>top_p</th>
    <th>top_k</th>
    <th>repetition_penalty</th>
  </tr>
  <tr>
    <td rowspan="4">NLU</td>
    <td>PI</td>
    <td>32</td>
    <td>0.0</td>
    <td>1.0</td>
    <td>1.0</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>32</td>
    <td>0.0</td>
    <td>1.0</td>
    <td>1.0</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td>SA</td>
    <td>32</td>
    <td>0.0</td>
    <td>1.0</td>
    <td>1.0</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td>TD</td>
    <td>32</td>
    <td>0.0</td>
    <td>1.0</td>
    <td>1.0</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td rowspan="2">NLR</td>
    <td>CR</td>
    <td>32</td>
    <td>0.0</td>
    <td>1.0</td>
    <td>1.0</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td>NLI</td>
    <td>32</td>
    <td>0.0</td>
    <td>1.0</td>
    <td>1.0</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td rowspan="2">NLG</td>
    <td>AS</td>
    <td>512</td>
    <td>0.3</td>
    <td>1.0</td>
    <td>1.0</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td>MT</td>
    <td>256</td>
    <td>0.0</td>
    <td>1.0</td>
    <td>1.0</td>
    <td>1.0</td>
  </tr>
</table>

Table 11: Inference Parameters per task.
G Complete Experimental Results

<table>
  <tr>
    <th>Model</th>
    <th>PI</th>
    <th>QA</th>
    <th>SA</th>
    <th>TD</th>
    <th>NLU Avg.</th>
    <th>CR</th>
    <th>NLI</th>
    <th>NLR Avg.</th>
  </tr>
  <tr><th colspan="9">Base pre-trained models (five-shot)</th></tr>
  <tr>
    <td>aisingapore/Gemma-SEA-LION-v3-9B</td>
    <td>82.40</td>
    <td><b>89.02</b></td>
    <td>67.60</td>
    <td>51.54</td>
    <td>72.64</td>
    <td>0.00<sup>a</sup></td>
    <td><b>63.93</b></td>
    <td><b>31.97</b></td>
  </tr>
  <tr>
    <td>aisingapore/Llama-SEA-LION-v2-8B</td>
    <td>71.21</td>
    <td>69.54</td>
    <td>48.86</td>
    <td>52.89</td>
    <td>60.62</td>
    <td>0.00<sup>a</sup></td>
    <td>28.99</td>
    <td>14.49</td>
  </tr>
  <tr>
    <td>aisingapore/Llama-SEA-LION-v3-8B</td>
    <td>71.96</td>
    <td>71.07</td>
    <td>38.58</td>
    <td>66.44</td>
    <td>62.01</td>
    <td>0.00<sup>a</sup></td>
    <td>48.59</td>
    <td>24.30</td>
  </tr>
  <tr>
    <td>google/gemma-2-9b</td>
    <td>70.64</td>
    <td>85.79</td>
    <td>51.09</td>
    <td><b>74.26</b></td>
    <td>70.44</td>
    <td>0.00<sup>a</sup></td>
    <td>46.04</td>
    <td>23.02</td>
  </tr>
  <tr>
    <td>meta-llama/Llama-3.1-8B</td>
    <td>60.31</td>
    <td>71.91</td>
    <td>50.16</td>
    <td>58.51</td>
    <td>60.22</td>
    <td>0.00<sup>a</sup></td>
    <td>26.82</td>
    <td>13.41</td>
  </tr>
  <tr>
    <td>Qwen/Qwen2-7B</td>
    <td>84.69</td>
    <td>74.12</td>
    <td>72.11</td>
    <td>65.48</td>
    <td><b>74.10</b></td>
    <td>0.00<sup>a</sup></td>
    <td>45.85</td>
    <td>22.92</td>
  </tr>
  <tr>
    <td>Qwen/Qwen2.5-7B</td>
    <td>84.48</td>
    <td>69.89</td>
    <td><b>72.22</b></td>
    <td>64.71</td>
    <td>72.82</td>
    <td>0.00<sup>a</sup></td>
    <td>41.11</td>
    <td>20.55</td>
  </tr>
  <tr>
    <td>sail/Sailor2-8B</td>
    <td>79.44</td>
    <td>75.93</td>
    <td>70.25</td>
    <td>67.64</td>
    <td>73.32</td>
    <td>0.00<sup>a</sup></td>
    <td>43.84</td>
    <td>21.92</td>
  </tr>
  <tr>
    <td>SeaLLMs/SeaLLMs-v3-7B</td>
    <td><b>84.95</b></td>
    <td>74.05</td>
    <td>71.12</td>
    <td>63.54</td>
    <td>73.41</td>
    <td>0.00<sup>a</sup></td>
    <td>41.46</td>
    <td>20.73</td>
  </tr>
  <tr><th colspan="9">Instruction-tuned models (zero-shot)</th></tr>
  <tr>
    <td>aisingapore/Gemma-SEA-LION-v3-9B-IT</td>
    <td>82.00</td>
    <td>82.80</td>
    <td><b>75.99</b></td>
    <td><b>72.85</b></td>
    <td><b>78.41</b></td>
    <td><b>92.75</b></td>
    <td><b>68.64</b></td>
    <td><b>80.69</b></td>
  </tr>
  <tr>
    <td>aisingapore/Llama-SEA-LION-v2-8B-IT</td>
    <td>68.50</td>
    <td>45.41</td>
    <td>49.81</td>
    <td>58.04</td>
    <td>55.44</td>
    <td>50.23</td>
    <td>37.63</td>
    <td>43.93</td>
  </tr>
  <tr>
    <td>aisingapore/Llama-SEA-LION-v3-8B-IT</td>
    <td>74.97</td>
    <td>80.69</td>
    <td>51.24</td>
    <td>60.57</td>
    <td>66.87</td>
    <td>79.44</td>
    <td>65.87</td>
    <td>72.65</td>
  </tr>
  <tr>
    <td>CohereForAI/ay-a-23-8B</td>
    <td>24.68</td>
    <td>33.50</td>
    <td>33.49</td>
    <td>32.70</td>
    <td>31.09</td>
    <td>28.12</td>
    <td>16.67</td>
    <td>22.39</td>
  </tr>
  <tr>
    <td>CohereForAI/ay-expanse-8b</td>
    <td>32.65</td>
    <td>2.97<sup>a</sup></td>
    <td>57.44</td>
    <td>34.39</td>
    <td>31.86</td>
    <td>1.96<sup>a</sup></td>
    <td>25.27</td>
    <td>13.61</td>
  </tr>
  <tr>
    <td>google/gemma-2-9b-it</td>
    <td>82.74</td>
    <td><b>83.96</b></td>
    <td>70.80</td>
    <td>64.48</td>
    <td>75.50</td>
    <td>91.25</td>
    <td>60.24</td>
    <td>75.74</td>
  </tr>
  <tr>
    <td>MERaLiON/Llama-3-MERaLiON-8B-Instruct</td>
    <td>23.47</td>
    <td>64.90</td>
    <td>35.16</td>
    <td>27.64</td>
    <td>37.79</td>
    <td>28.97</td>
    <td>23.35</td>
    <td>26.16</td>
  </tr>
  <tr>
    <td>meta-llama/Llama-3.1-8B-Instruct</td>
    <td>61.38</td>
    <td>74.90</td>
    <td>52.55</td>
    <td>50.00</td>
    <td>59.71</td>
    <td>72.66</td>
    <td>56.28</td>
    <td>64.47</td>
  </tr>
  <tr>
    <td>Qwen/Qwen2-7B-Instruct</td>
    <td>73.08</td>
    <td>56.01</td>
    <td>52.63</td>
    <td>26.10</td>
    <td>51.96</td>
    <td>43.99</td>
    <td>32.35</td>
    <td>38.17</td>
  </tr>
  <tr>
    <td>Qwen/Qwen2.5-7B-Instruct</td>
    <td><b>85.22</b></td>
    <td>69.07</td>
    <td>67.82</td>
    <td>60.84</td>
    <td>70.74</td>
    <td>46.15</td>
    <td>57.80</td>
    <td>51.98</td>
  </tr>
  <tr>
    <td>sail/Sailor2-8B-Chat</td>
    <td>73.60</td>
    <td>59.48</td>
    <td>48.67</td>
    <td>47.24</td>
    <td>57.25</td>
    <td>57.52</td>
    <td>61.21</td>
    <td>59.36</td>
  </tr>
  <tr>
    <td>SeaLLMs/SeaLLMs-v3-7B-Chat</td>
    <td>37.85</td>
    <td>49.72</td>
    <td>70.63</td>
    <td>61.48</td>
    <td>54.92</td>
    <td>71.82</td>
    <td>20.72</td>
    <td>46.27</td>
  </tr>
  <tr>
    <td>Tower-Babel/Babel-9B-Chat</td>
    <td>78.76</td>
    <td>61.68</td>
    <td>72.52</td>
    <td>56.74</td>
    <td>67.42</td>
    <td>83.00</td>
    <td>26.89</td>
    <td>54.95</td>
  </tr>
</table>

<sup>a</sup> Could not follow answer instructions.

Table 12: Performance of small LLMs (<11B parameters) on BATAYAN natural language understanding (NLU) and natural language reasoning (NLR) tasks. Macro F1 scores are reported.
<table>
  <tr>
    <th>Model</th>
    <th>PI</th>
    <th>QA</th>
    <th>SA</th>
    <th>TD</th>
    <th>NLU Avg.</th>
    <th>CR</th>
    <th>NLI</th>
    <th>NLR Avg.</th>
  </tr>
  <tr>
    <th colspan="9">Base pre-trained models (five-shot)</th>
  </tr>
  <tr>
    <td>google/gemma-2-27b</td>
    <td>72.63</td>
    <td><b>86.04</b></td>
    <td>69.60</td>
    <td><b>75.95</b></td>
    <td>76.05</td>
    <td>0.00<sup>a</sup></td>
    <td>56.21</td>
    <td>28.11</td>
  </tr>
  <tr>
    <td>google/gemma-3-12b-pt</td>
    <td>71.42</td>
    <td>86.02</td>
    <td><b>79.30</b></td>
    <td>51.60</td>
    <td>72.08</td>
    <td>0.00<sup>a</sup></td>
    <td>59.47</td>
    <td>29.74</td>
  </tr>
  <tr>
    <td>google/gemma-3-27b-pt</td>
    <td>59.29</td>
    <td>84.83</td>
    <td>69.84</td>
    <td>52.34</td>
    <td>66.58</td>
    <td>0.00<sup>a</sup></td>
    <td>55.13</td>
    <td>27.56</td>
  </tr>
  <tr>
    <td>Qwen/Qwen2.5-14B</td>
    <td><b>89.25</b></td>
    <td>82.03</td>
    <td>68.36</td>
    <td>62.89</td>
    <td>75.63</td>
    <td>0.00<sup>a</sup></td>
    <td>53.10</td>
    <td>26.55</td>
  </tr>
  <tr>
    <td>Qwen/Qwen2.5-32B</td>
    <td>87.69</td>
    <td>84.98</td>
    <td>67.71</td>
    <td>66.62</td>
    <td><b>76.75</b></td>
    <td>0.00<sup>a</sup></td>
    <td>55.09</td>
    <td>27.54</td>
  </tr>
  <tr>
    <td>sail/Sailor2-20B</td>
    <td>85.46</td>
    <td>84.94</td>
    <td>68.47</td>
    <td>48.83</td>
    <td>71.92</td>
    <td>0.00<sup>a</sup></td>
    <td><b>63.09</b></td>
    <td><b>31.55</b></td>
  </tr>
  <tr>
    <th colspan="9">Instruction-tuned models (zero-shot)</th>
  </tr>
  <tr>
    <td>CohereForAI/aya-expanse-32b</td>
    <td>76.39</td>
    <td>54.82</td>
    <td>63.25</td>
    <td>74.70</td>
    <td>67.29</td>
    <td>55.46</td>
    <td>66.57</td>
    <td>61.01</td>
  </tr>
  <tr>
    <td>google/gemma-3-12b-it</td>
    <td>82.27</td>
    <td>83.94</td>
    <td>62.10</td>
    <td>80.50</td>
    <td>77.20</td>
    <td>93.75</td>
    <td>70.64</td>
    <td><b>82.19</b></td>
  </tr>
  <tr>
    <td>google/gemma-2-27b-it</td>
    <td>70.68</td>
    <td>69.18</td>
    <td>70.39</td>
    <td>80.36</td>
    <td>72.65</td>
    <td>67.76</td>
    <td><b>80.25</b></td>
    <td>73.04</td>
  </tr>
  <tr>
    <td>google/gemma-3-27b-it</td>
    <td>70.20</td>
    <td><b>87.91</b></td>
    <td>67.77</td>
    <td><b>81.74</b></td>
    <td>76.91</td>
    <td><b>94.75</b></td>
    <td>67.73</td>
    <td>81.24</td>
  </tr>
  <tr>
    <td>Qwen/Qwen2.5-14B-Instruct</td>
    <td>63.75</td>
    <td>64.64</td>
    <td>72.34</td>
    <td>72.73</td>
    <td>68.37</td>
    <td>53.47</td>
    <td>43.83</td>
    <td>48.65</td>
  </tr>
  <tr>
    <td>Qwen/Qwen2.5-32B-Instruct</td>
    <td><b>83.70</b></td>
    <td>84.97</td>
    <td><b>74.60</b></td>
    <td>75.50</td>
    <td><b>79.69</b></td>
    <td>85.98</td>
    <td>60.58</td>
    <td>73.28</td>
  </tr>
  <tr>
    <td>sail/Sailor2-20B-Chat</td>
    <td>39.17</td>
    <td>13.81</td>
    <td>54.41</td>
    <td>74.03</td>
    <td>45.36</td>
    <td>22.98</td>
    <td>47.88</td>
    <td>35.43</td>
  </tr>
</table>

<sup>a</sup> Could not follow answer instructions.

Table 13: Performance of medium LLMs (<32B parameters) on BATAYAN NLU/NLR tasks. Macro F1 scores are reported.

<table>
  <tr>
    <th>Model</th>
    <th>PI</th>
    <th>QA</th>
    <th>SA</th>
    <th>TD</th>
    <th>NLU Avg.</th>
    <th>CR</th>
    <th>NLI</th>
    <th>NLR Avg.</th>
  </tr>
  <tr>
    <th colspan="9">Base pre-trained models (five-shot)</th>
  </tr>
  <tr>
    <td>asingapore/Llama-SEA-LION-v3-70B</td>
    <td>83.48</td>
    <td>87.02</td>
    <td>45.89</td>
    <td>77.49</td>
    <td>73.47</td>
    <td>0.00<sup>a</sup></td>
    <td><b>79.62</b></td>
    <td><b>39.81</b></td>
  </tr>
  <tr>
    <td>meta-llama/Llama-3.1-70B</td>
    <td>83.49</td>
    <td>86.05</td>
    <td>59.72</td>
    <td>51.70</td>
    <td>70.24</td>
    <td>0.00<sup>a</sup></td>
    <td>55.78</td>
    <td>27.89</td>
  </tr>
  <tr>
    <td>Qwen/Qwen2-72B</td>
    <td><b>90.25</b></td>
    <td><b>88.86</b></td>
    <td>67.88</td>
    <td>77.51</td>
    <td>81.13</td>
    <td>0.00<sup>a</sup></td>
    <td>69.63</td>
    <td>34.81</td>
  </tr>
  <tr>
    <td>Qwen/Qwen2.5-72B</td>
    <td>86.89</td>
    <td>86.79</td>
    <td><b>74.73</b></td>
    <td><b>79.61</b></td>
    <td><b>82.01</b></td>
    <td>0.00<sup>a</sup></td>
    <td>67.58</td>
    <td>33.79</td>
  </tr>
  <tr>
    <th colspan="9">Instruction-tuned models (zero-shot)</th>
  </tr>
  <tr>
    <td>asingapore/Llama-SEA-LION-v3-70B-IT</td>
    <td>84.07</td>
    <td>87.01</td>
    <td>68.79</td>
    <td>76.99</td>
    <td>79.21</td>
    <td><b>94.75</b></td>
    <td><b>75.34</b></td>
    <td><b>85.04</b></td>
  </tr>
  <tr>
    <td>CohereForAI/aya-23-35B</td>
    <td>81.83</td>
    <td>61.46</td>
    <td>61.68</td>
    <td>53.06</td>
    <td>64.51</td>
    <td>74.16</td>
    <td>40.54</td>
    <td>57.35</td>
  </tr>
  <tr>
    <td>deepseek-ai/DeepSeek-V3</td>
    <td>83.07</td>
    <td><b>90.00</b></td>
    <td><b>81.78</b></td>
    <td>76.21</td>
    <td><b>82.77</b></td>
    <td>94.25</td>
    <td>63.25</td>
    <td>78.75</td>
  </tr>
  <tr>
    <td>meta-llama/Llama-3.1-70B-Instruct</td>
    <td>83.57</td>
    <td>88.13</td>
    <td>67.29</td>
    <td>63.80</td>
    <td>75.70</td>
    <td>93.49</td>
    <td>72.62</td>
    <td>83.06</td>
  </tr>
  <tr>
    <td>meta-llama/Llama-3.3-70B-Instruct</td>
    <td>82.62</td>
    <td>86.96</td>
    <td>66.12</td>
    <td>77.40</td>
    <td>78.28</td>
    <td>94.25</td>
    <td>72.97</td>
    <td>83.61</td>
  </tr>
  <tr>
    <td>Qwen/Qwen2-72B-Instruct</td>
    <td><b>85.37</b></td>
    <td>80.16</td>
    <td>76.16</td>
    <td>76.38</td>
    <td>81.00</td>
    <td>58.54</td>
    <td>68.39</td>
    <td>63.47</td>
  </tr>
  <tr>
    <td>Qwen/Qwen2.5-72B-Instruct</td>
    <td>80.40</td>
    <td>84.00</td>
    <td>75.32</td>
    <td>73.44</td>
    <td>78.29</td>
    <td>91.75</td>
    <td>63.95</td>
    <td>77.85</td>
  </tr>
  <tr>
    <td>Tower-Babel/Babel-83B-Chat</td>
    <td>85.06</td>
    <td>33.12</td>
    <td>73.20</td>
    <td><b>81.24</b></td>
    <td>68.15</td>
    <td>49.62</td>
    <td>68.14</td>
    <td>58.88</td>
  </tr>
  <tr>
    <th colspan="9">Reasoning models (zero-shot)</th>
  </tr>
  <tr>
    <td>deepseek-ai/DeepSeek-R1</td>
    <td>79.84</td>
    <td>87.93</td>
    <td>71.47</td>
    <td>71.47</td>
    <td>77.68</td>
    <td>97.00</td>
    <td>71.06</td>
    <td>84.03</td>
  </tr>
</table>

<sup>a</sup> Could not follow answer instructions.

Table 14: Performance of large LLMs (>=32B parameters) on BATAYAN NLU/NLR tasks. Macro F1 scores are reported.

<table>
  <tr>
    <th>Model</th>
    <th>PI</th>
    <th>QA</th>
    <th>SA</th>
    <th>TD</th>
    <th>NLU Avg.</th>
    <th>CR</th>
    <th>NLI</th>
    <th>NLR Avg.</th>
  </tr>
  <tr>
    <td>google/gemini-1.5-flash-002</td>
    <td>85.71</td>
    <td>84.98</td>
    <td>63.81</td>
    <td>66.04</td>
    <td>75.14</td>
    <td>94.24</td>
    <td>69.19</td>
    <td>81.72</td>
  </tr>
  <tr>
    <td>google/gemini-1.5-pro-002</td>
    <td>85.62</td>
    <td><b>89.95</b></td>
    <td>69.48</td>
    <td>79.27</td>
    <td>81.08</td>
    <td>95.74</td>
    <td><b>76.71</b></td>
    <td><b>86.23</b></td>
  </tr>
  <tr>
    <td>google/gemini-2.0-flash-001</td>
    <td>83.98</td>
    <td>88.05</td>
    <td>73.93</td>
    <td><b>80.49</b></td>
    <td><b>81.61</b></td>
    <td><b>97.74</b></td>
    <td>70.47</td>
    <td>84.11</td>
  </tr>
  <tr>
    <td>google/gemini-2.0-flash-lite-001</td>
    <td>85.02</td>
    <td>85.97</td>
    <td>64.33</td>
    <td>78.94</td>
    <td>78.57</td>
    <td>96.49</td>
    <td>70.20</td>
    <td>83.35</td>
  </tr>
  <tr>
    <td>openai/gpt-4o-mini-2024-07-18</td>
    <td>86.11</td>
    <td>85.04</td>
    <td>70.77</td>
    <td>70.77</td>
    <td>78.17</td>
    <td>92.99</td>
    <td>68.18</td>
    <td>80.59</td>
  </tr>
  <tr>
    <td>openai/gpt-4o-2024-08-06</td>
    <td><b>88.21</b></td>
    <td>71.88</td>
    <td><b>74.78</b></td>
    <td>74.78</td>
    <td>77.41</td>
    <td>97.25</td>
    <td>73.33</td>
    <td>85.29</td>
  </tr>
</table>

Table 15: Performance of commercial LLMs on BATAYAN NLU/NLR tasks. Macro F1 scores are reported.
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="3">AS</th>
    <th colspan="3">MT (eng→tgl)</th>
    <th colspan="3">MT (tgl→eng)</th>
    <th rowspan="2">NLG Avg.</th>
  </tr>
  <tr>
    <th>BERTScore</th>
    <th>ChrF++</th>
    <th>ROUGE-L</th>
    <th>ChrF++</th>
    <th>MetricX-24</th>
    <th>ChrF++</th>
    <th>MetricX-24</th>
  </tr>
  <tr><th colspan="11"><i>Base pre-trained models (five-shot)</i></th></tr>
  <tr><td>aisingapore/Gemma-SEA-LION-v3-9B</td><td>77.01</td><td>34.19</td><td>33.71</td><td>58.90</td><td>82.75</td><td>56.63</td><td>91.15</td><td>62.05</td></tr>
  <tr><td>aisingapore/Llama-SEA-LION-v2-8B</td><td>75.86</td><td>27.17</td><td>30.88</td><td>48.45</td><td>62.74</td><td>58.17</td><td>87.10</td><td>55.77</td></tr>
  <tr><td>aisingapore/Llama-SEA-LION-v3-8B</td><td>75.73</td><td>32.41</td><td>30.56</td><td>53.10</td><td>72.41</td><td>57.70</td><td>88.91</td><td>58.69</td></tr>
  <tr><td>google/gemma-2-9b</td><td>76.60</td><td>31.72</td><td>31.69</td><td>59.06</td><td>82.01</td><td>63.27</td><td>91.16</td><td>62.22</td></tr>
  <tr><td>meta-llama/Llama-3.1-8B</td><td>75.80</td><td>29.93</td><td>30.13</td><td>51.94</td><td>71.10</td><td>58.44</td><td>88.77</td><td>58.01</td></tr>
  <tr><td>Qwen/Qwen2-7B</td><td>73.84</td><td>27.64</td><td>25.71</td><td>42.61</td><td>47.37</td><td>53.60</td><td>82.13</td><td>50.41</td></tr>
  <tr><td>Qwen/Qwen2.5-7B</td><td>73.26</td><td>26.71</td><td>25.04</td><td>38.89</td><td>40.28</td><td>42.59</td><td>81.58</td><td>46.91</td></tr>
  <tr><td>sail/Sailor2-8B</td><td>77.05</td><td>30.69</td><td>34.50</td><td>62.01</td><td>86.30</td><td>63.41</td><td>91.05</td><td>63.57</td></tr>
  <tr><td>SeaLLMs/SeaLLMs-v3-7B</td><td>73.78</td><td>28.37</td><td>26.57</td><td>43.45</td><td>47.79</td><td>44.54</td><td>83.74</td><td>49.75</td></tr>
  <tr><th colspan="11"><i>Instruction-tuned models (zero-shot)</i></th></tr>
  <tr><td>aisingapore/Gemma-SEA-LION-v3-9B-IT</td><td>73.00</td><td>34.16</td><td>20.99</td><td>57.79</td><td>87.24</td><td>58.63</td><td>90.56</td><td>60.35</td></tr>
  <tr><td>aisingapore/Llama-SEA-LION-v2-8B-IT</td><td>73.39</td><td>32.91</td><td>24.14</td><td>50.90</td><td>64.62</td><td>48.49</td><td>55.04</td><td>49.93</td></tr>
  <tr><td>aisingapore/Llama-SEA-LION-v3-8B-IT</td><td>73.34</td><td>33.80</td><td>23.43</td><td>57.14</td><td>82.67</td><td>58.33</td><td>87.69</td><td>59.49</td></tr>
  <tr><td>CohereForAI/ay-23-8B</td><td>67.37</td><td>24.01</td><td>11.76</td><td>15.61</td><td>9.68</td><td>46.20</td><td>58.59</td><td>33.32</td></tr>
  <tr><td>CohereForAI/ay-expans-8b</td><td>59.48</td><td>6.69</td><td>1.64</td><td>39.30</td><td>37.47</td><td>49.81</td><td>78.88</td><td>39.04</td></tr>
  <tr><td>google/gemma-2-9b-it</td><td>72.73</td><td>34.19</td><td>21.81</td><td>56.80</td><td>84.23</td><td>50.97</td><td>76.62</td><td>56.76</td></tr>
  <tr><td>MERaLiON/Llama-3-MERaLiON-8B-Instruct</td><td>74.08</td><td>31.79</td><td>25.68</td><td>49.77</td><td>64.85</td><td>41.66</td><td>60.14</td><td>49.71</td></tr>
  <tr><td>meta-llama/Llama-3.1-SB-Instruct</td><td>74.24</td><td>34.27</td><td>26.94</td><td>44.79</td><td>51.76</td><td>46.33</td><td>70.65</td><td>49.85</td></tr>
  <tr><td>Qwen/Qwen2-7B-Instruct</td><td>69.51</td><td>26.09</td><td>13.58</td><td>42.22</td><td>42.86</td><td>45.13</td><td>50.01</td><td>41.34</td></tr>
  <tr><td>Qwen/Qwen2.5-7B-Instruct</td><td>70.61</td><td>28.89</td><td>15.86</td><td>38.99</td><td>34.57</td><td>49.02</td><td>76.36</td><td>44.91</td></tr>
  <tr><td>sail/Sailor2-8B-Chat</td><td>71.82</td><td>31.83</td><td>19.75</td><td>54.04</td><td>80.93</td><td>35.41</td><td>19.66</td><td>44.78</td></tr>
  <tr><td>SeaLLMs/SeaLLMs-v3-7B-Chat</td><td>70.34</td><td>27.84</td><td>16.49</td><td>46.36</td><td>61.07</td><td>43.27</td><td>60.37</td><td>46.53</td></tr>
  <tr><td>Tower-Babel/Babel-9B-Chat</td><td>71.61</td><td>31.55</td><td>18.93</td><td>49.97</td><td>72.50</td><td>48.69</td><td>75.69</td><td>52.70</td></tr>
</table>

Table 16: Performance of small LLMs (<11B parameters) on BATAYAN natural language generation (NLG) tasks.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="3">AS</th>
    <th colspan="3">MT (eng→tgl)</th>
    <th colspan="3">MT (tgl→eng)</th>
    <th rowspan="2">NLG Avg.</th>
  </tr>
  <tr>
    <th>BERTScore</th>
    <th>ChrF++</th>
    <th>ROUGE-L</th>
    <th>ChrF++</th>
    <th>MetricX-24</th>
    <th>ChrF++</th>
    <th>MetricX-24</th>
  </tr>
  <tr><th colspan="11"><i>Base pre-trained models (five-shot)</i></th></tr>
  <tr><td>google/gemma-2-27b</td><td>77.27</td><td>33.71</td><td>33.50</td><td>61.16</td><td>85.56</td><td>64.77</td><td>91.82</td><td>63.97</td></tr>
  <tr><td>google/gemma-3-12b-pt</td><td>77.42</td><td>32.12</td><td>34.32</td><td>61.66</td><td>85.52</td><td>64.69</td><td>92.08</td><td>63.97</td></tr>
  <tr><td>google/gemma-3-27b-pt</td><td><b>77.70</b></td><td><b>35.53</b></td><td><b>35.36</b></td><td><b>62.89</b></td><td><b>87.06</b></td><td><b>64.98</b></td><td><b>92.27</b></td><td><b>65.11</b></td></tr>
  <tr><td>Qwen/Qwen2.5-14B</td><td>74.39</td><td>30.34</td><td>27.87</td><td>49.01</td><td>62.69</td><td>59.41</td><td>88.63</td><td>56.05</td></tr>
  <tr><td>Qwen/Qwen2.5-32B</td><td>75.13</td><td>29.73</td><td>29.31</td><td>50.94</td><td>65.06</td><td>60.18</td><td>89.34</td><td>57.10</td></tr>
  <tr><td>sail/Sailor2-20B</td><td>76.73</td><td>32.41</td><td>33.47</td><td>61.75</td><td>86.30</td><td>63.88</td><td>91.73</td><td>63.75</td></tr>
  <tr><th colspan="11"><i>Instruction-tuned models (zero-shot)</i></th></tr>
  <tr><td>CohereForAI/ay-expanse-32b</td><td>71.18</td><td>30.12</td><td>16.85</td><td>49.14</td><td>62.76</td><td>57.35</td><td>85.05</td><td>53.21</td></tr>
  <tr><td>google/gemma-2-27b-it</td><td><b>73.41</b></td><td><b>35.11</b></td><td><b>23.21</b></td><td><b>59.51</b></td><td><b>88.19</b></td><td><b>61.87</b></td><td><b>91.68</b></td><td><b>61.86</b></td></tr>
  <tr><td>google/gemma-3-12b-it</td><td>72.10</td><td>32.80</td><td>19.43</td><td>60.09</td><td>89.39</td><td>61.85</td><td>91.61</td><td>61.04</td></tr>
  <tr><td>google/gemma-3-27b-it</td><td>72.25</td><td>33.26</td><td>19.55</td><td><b>61.02</b></td><td><b>90.71</b></td><td><b>62.09</b></td><td><b>92.18</b></td><td><b>61.58</b></td></tr>
  <tr><td>Qwen/Qwen2.5-14B-Instruct</td><td>71.92</td><td>31.85</td><td>19.24</td><td>44.78</td><td>54.51</td><td>58.01</td><td>86.08</td><td>52.34</td></tr>
  <tr><td>Qwen/Qwen2.5-32B-Instruct</td><td>72.47</td><td>33.43</td><td>21.22</td><td>47.47</td><td>59.47</td><td>58.77</td><td>86.90</td><td>54.25</td></tr>
  <tr><td>sail/Sailor2-20B-Chat</td><td>71.47</td><td>31.20</td><td>18.41</td><td>49.45</td><td>72.51</td><td>49.52</td><td>72.79</td><td>52.19</td></tr>
</table>

Table 17: Performance of medium LLMs (<32B parameters) on BATAYAN NLG tasks.
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="3">AS</th>
    <th colspan="2">MT (<i>eng→tgl</i>)</th>
    <th colspan="2">MT (<i>tgl→eng</i>)</th>
    <th rowspan="2">NLG Avg.</th>
  </tr>
  <tr>
    <th>BERTScore</th>
    <th>ChrF++</th>
    <th>ROUGE-L</th>
    <th>ChrF++</th>
    <th>MetricX-24</th>
    <th>ChrF++</th>
    <th>MetricX-24</th>
  </tr>
  <tr>
    <td colspan="9"><i>Base pre-trained models (five-shot)</i></td>
  </tr>
  <tr>
    <td>asingapore/Llama-SEA-LION-v3-70B</td>
    <td><b>76.87</b></td>
    <td><b>35.49</b></td>
    <td><b>33.66</b></td>
    <td><b>60.70</b></td>
    <td><b>85.28</b></td>
    <td>63.39</td>
    <td>91.52</td>
    <td><b>63.85</b></td>
  </tr>
  <tr>
    <td>meta-llama/Llama-3.1-70B</td>
    <td>76.80</td>
    <td>33.98</td>
    <td>33.54</td>
    <td>60.20</td>
    <td>84.32</td>
    <td><b>65.80</b></td>
    <td><b>91.85</b></td>
    <td>63.79</td>
  </tr>
  <tr>
    <td>Qwen/Qwen2-72B</td>
    <td>76.81</td>
    <td>34.15</td>
    <td>33.24</td>
    <td>54.75</td>
    <td>74.84</td>
    <td>65.13</td>
    <td>90.87</td>
    <td>61.40</td>
  </tr>
  <tr>
    <td>Qwen/Qwen2.5-72B</td>
    <td>76.41</td>
    <td>34.12</td>
    <td>31.96</td>
    <td>54.69</td>
    <td>74.92</td>
    <td>64.50</td>
    <td>90.85</td>
    <td>61.06</td>
  </tr>
  <tr>
    <td colspan="9"><i>Instruction-tuned models (zero-shot)</i></td>
  </tr>
  <tr>
    <td>asingapore/Llama-SEA-LION-v3-70B-IT</td>
    <td>73.25</td>
    <td>34.82</td>
    <td>23.81</td>
    <td>61.02</td>
    <td>86.24</td>
    <td>63.11</td>
    <td>91.28</td>
    <td>61.93</td>
  </tr>
  <tr>
    <td>CohereForAI/aya-23-35B</td>
    <td>72.81</td>
    <td>29.94</td>
    <td>22.08</td>
    <td>46.00</td>
    <td>56.79</td>
    <td>49.54</td>
    <td>72.45</td>
    <td>49.94</td>
  </tr>
  <tr>
    <td>deepseek-ai/DeepSeek-V3</td>
    <td>73.19</td>
    <td>35.16</td>
    <td>22.99</td>
    <td><b>61.61</b></td>
    <td><b>89.21</b></td>
    <td>63.06</td>
    <td><b>92.28</b></td>
    <td><b>62.50</b></td>
  </tr>
  <tr>
    <td>meta-llama/Llama-3.1-70B-Instruct</td>
    <td><b>74.76</b></td>
    <td><b>35.56</b></td>
    <td><b>27.94</b></td>
    <td>60.32</td>
    <td>84.73</td>
    <td>62.78</td>
    <td>91.20</td>
    <td>62.47</td>
  </tr>
  <tr>
    <td>meta-llama/Llama-3.3-70B-Instruct</td>
    <td>72.90</td>
    <td>34.03</td>
    <td>22.63</td>
    <td>58.98</td>
    <td>83.05</td>
    <td>61.24</td>
    <td>90.72</td>
    <td>60.51</td>
  </tr>
  <tr>
    <td>Qwen/Qwen2-72B-Instruct</td>
    <td>67.05</td>
    <td>25.08</td>
    <td>11.38</td>
    <td>52.84</td>
    <td>70.63</td>
    <td>60.05</td>
    <td>88.73</td>
    <td>53.68</td>
  </tr>
  <tr>
    <td>Qwen/Qwen2.5-72B-Instruct</td>
    <td>71.62</td>
    <td>31.52</td>
    <td>18.02</td>
    <td>52.01</td>
    <td>72.30</td>
    <td><b>64.33</b></td>
    <td>90.02</td>
    <td>57.12</td>
  </tr>
  <tr>
    <td>Tower-Babel/Babel-83B-Chat</td>
    <td>67.72</td>
    <td>20.58</td>
    <td>13.35</td>
    <td>53.29</td>
    <td>76.61</td>
    <td>51.56</td>
    <td>85.42</td>
    <td>52.65</td>
  </tr>
  <tr>
    <td colspan="9"><i>Reasoning models (zero-shot)</i></td>
  </tr>
  <tr>
    <td>deepseek-ai/DeepSeek-R1</td>
    <td>71.97</td>
    <td>32.70</td>
    <td>19.59</td>
    <td>59.15</td>
    <td>90.91</td>
    <td>61.01</td>
    <td>92.30</td>
    <td>61.09</td>
  </tr>
</table>

Table 18: Performance of large LLMs (>=32B parameters) on BATAYAN NLG tasks.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="3">AS</th>
    <th colspan="2">MT (<i>eng→tgl</i>)</th>
    <th colspan="2">MT (<i>tgl→eng</i>)</th>
    <th rowspan="2">NLG Avg.</th>
  </tr>
  <tr>
    <th>BERTScore</th>
    <th>ChrF++</th>
    <th>ROUGE-L</th>
    <th>ChrF++</th>
    <th>MetricX-24</th>
    <th>ChrF++</th>
    <th>MetricX-24</th>
  </tr>
  <tr>
    <td>google/gemini-1.5-flash-002</td>
    <td>72.44</td>
    <td>34.00</td>
    <td>21.20</td>
    <td>61.01</td>
    <td>90.11</td>
    <td>65.23</td>
    <td>92.04</td>
    <td>62.29</td>
  </tr>
  <tr>
    <td>google/gemini-1.5-pro-002</td>
    <td><b>73.62</b></td>
    <td><b>36.16</b></td>
    <td><b>24.00</b></td>
    <td>61.07</td>
    <td><b>90.90</b></td>
    <td>64.14</td>
    <td>91.30</td>
    <td>63.03</td>
  </tr>
  <tr>
    <td>google/gemini-2.0-flash-001</td>
    <td>72.99</td>
    <td>34.85</td>
    <td>21.96</td>
    <td><b>63.88</b></td>
    <td>90.87</td>
    <td><b>65.64</b></td>
    <td><b>92.53</b></td>
    <td><b>63.25</b></td>
  </tr>
  <tr>
    <td>google/gemini-2.0-flash-lite-001</td>
    <td>72.83</td>
    <td>34.80</td>
    <td>21.87</td>
    <td>63.45</td>
    <td>90.06</td>
    <td>65.32</td>
    <td>91.51</td>
    <td>62.83</td>
  </tr>
  <tr>
    <td>openai/gpt-4o-mini-2024-07-18</td>
    <td>72.70</td>
    <td>33.97</td>
    <td>21.67</td>
    <td>62.46</td>
    <td>88.59</td>
    <td>63.12</td>
    <td>91.68</td>
    <td>62.03</td>
  </tr>
  <tr>
    <td>openai/gpt-4o-2024-08-06</td>
    <td>72.83</td>
    <td>34.36</td>
    <td>21.55</td>
    <td>63.86</td>
    <td>90.19</td>
    <td>64.89</td>
    <td>92.51</td>
    <td>62.88</td>
  </tr>
</table>

Table 19: Performance of commercial LLMs on BATAYAN NLG tasks.

Reviewer paper summaries: This paper introduces a benchmark for evaluation of large language models in Filipino. The benchmark includes 8 tasks: Paraphrase Identification (PI), Question Answering (QA), Sentiment Analysis (SA), Toxicity Detection (TD), Causal Reasoning (CR), Natural language Inference (NLI). Abstractive Summarization (AS), and Machine Translation (MT). For SA the authors annotated an existing dataset, for MT, AQ, and TD they used existing datasets and for the rest of the tasks the authors (who are native speakers of Filipino) first translated existing English datasets and then manually performed verification and correction. The datasets are checked for fluency, completeness, and sensibility by the authors. High inter annotator agreement measures shows a high quality dataset. Eventually the authors reported their experience in preparing the benchmark, especially the challenges they faced and evaluated a number of best performing language models using their benchmark.
Summary of strengths: See the paper summary.
As far as my expertise allows, the authors have considered all necessary aspect for creating a modern benchmark for a low resource language. 
The paper is very well structured and written.
Summary of weaknesses: Translating English datasets while useful, still would be distant from a naturally written text by local speaker. It would lack the diversity and language usage that would inherently come from various local speakers. Translation would result in lack of diversity and innovativeness. 

Use of professional translator is one way to mitigate the problem.


Some of the translation challenges mentioned in section 4 are standard translation problems that can occur in translation from one language to another. Professional translators are well aware of such problems and trained to resolve them. If one weakness of this benchmark is heavy translation, the other one is not using professional translators.
Comments, suggestions and typos: Would be great to report evaluation results under various hyper parameter settings e.g. few shot size, prompt style, effect of system prompt vs user prompt.

Would be interesting to include other leading LLM (open and closed) in the results section. To name a few, deepseek v3, chatgpt 4o, phi 4, and gemini and command R multilingual are missing.
Overall assessment scores: 4