Automatic Readability Assessment for Closely Related Languages
Year: 2023

Abstract:
In recent years, the main focus of research on automatic readability assessment (ARA) has shifted towards using expensive deep learningbased methods with the primary goal of increasing models’ accuracy. This, however, is rarely applicable for low-resource languages where traditional handcrafted features are still widely used due to the lack of existing NLP tools to extract deeper linguistic representations. In this work, we take a step back from the technical component and focus on how linguistic aspects such as mutual intelligibility or degree of language relatedness can improve ARA in a low-resource setting. We collect short stories written in three languages in the Philippines – Tagalog, Bikol, and Cebuano – to train readability assessment models and explore the interaction of data and features in various crosslingual setups. Our results show that the inclusion of CROSSNGO, a novel specialized feature exploiting n-gram overlap applied to languages with high mutual intelligibility, significantly improves the performance of ARA models compared to the use of off-the-shelf large multilingual language models alone. Consequently, when both linguistic representations are combined, we achieve state-of-the-art results for Tagalog and Cebuano, and baseline scores for ARA in Bikol. We release our data and code at github.com/imperialite/ara-close-lang

heading: 1 Introduction
text: Automatic readability assessment (ARA) is the task that aims to approximate the difficulty level of a piece of literary material using computer-aided tools. The need for such application arises from challenges related to the misalignment of difficulty labels when humans with various domain expertise provide annotations, as well as to the difficulty of manual extraction of complex text-based features (Deutsch et al., 2020). At the same time,
readability assessment tools often use different definitions of complexity levels based on (a) age level (Vajjala and Meurers, 2012; Xia et al., 2016), (b) grade level (Imperial and Ong, 2020, 2021a), or on established frameworks such as (c) the Common European Framework of Reference for Languages (CEFR)1 (François and Fairon, 2012; Pilán et al., 2016; Xia et al., 2016; Reynolds, 2016; Vajjala and Rama, 2018).
In recent years, deep learning methods and large language models (LLMs) have gained popularity in the research community. Often studies using these methodologies focus primarily on improving the performance across various metrics. This is particularly manifest in ARA research in languages with a high number of accessible and publicly-available readability corpora such as English (Heilman et al., 2008; Flor et al., 2013; Vajjala and Lučić, 2018) and German (Hancke et al., 2012; Weiss et al., 2021; Weiss and Meurers, 2022) to name a few. At the same time, existing studies focusing on low-resource languages such as Cebuano (Imperial et al., 2022) and Bengala (Islam et al., 2012; Islam and Rahman, 2014) are still at the stage of primarily using traditional features such as word and sentence lengths to train predictive models.
We identify two problems that are related to the use of complex neural-based approaches: the success of such models depends on (a) whether there is enough available data to train a model using a customized deep neural network, and (b) in the case of LLMs, whether there exists an available off-the-shelf pre-trained model for a low-resource language of interest. Imperial et al. (2022) have recently shown that merely integrating extracted embeddings from a multilingual BERT model as features for Cebuano, a low-resource Philippine language, does not outperform models trained with
1https://www.coe.int/en/web/ common-european-framework-reference-languages/ level-descriptions
5371
orthographic features such as syllable patterns customized for the language. These challenges provide motivation for researchers to further explore methods that do not rely on the availability of large amounts of data or complex pre-trained models and investigate simpler, more interpretable models instead of black box architectures.
In this paper, we take a step back and focus on the data available for low-resource Philippine languages and the features extracted from them rather than on the algorithmic aspects. Specifically, we explore a scenario where small readability corpora are available for languages that are closely related or belong to one major language family tree. To the best of our knowledge, incorporating the degree of language closeness or relatedness has not been explored before in any cross-lingual ARA setup. In this study, we make the following contributions:
1. We conduct an extensive pioneer study on readability assessment in a cross-lingual setting using three closely related Philippine languages: Tagalog, Bikolano, and Cebuano.
2. We extract various feature sets ranging from linguistically motivated to neural embeddings, and empirically evaluate how they affect the performance of readability models in a singular, pairwise, and full cross-lingual setup.
3. We introduce cross-lingual Character N-gram Overlap (CROSSNGO), a novel feature applicable to readability assessment in closely related languages.
4. We also introduce and release a new readability corpus for Bikolano, one of the major languages in the Philippines.
5. Finally, we set a baseline for ARA in Bikol and report state-of-the-art results for Tagalog and Cebuano.

heading: 2 Background

heading: 2.1 The Philippine Linguistic Profile
text: The Philippines is a linguistically diverse country in Southeast Asia (SEA) with over 180 languages spoken by over 100 million people. Languages in the Philippines can be best described as morphologically rich due to their free-word order structures and high number of possible inflections, full and partial duplications, and compound words (Go and
Nocon, 2017). In addition, following lexicostatistical studies, languages are divided into two subgroups, northern and central, wherein the major languages Ilokano, Pangasinan, and Kapampangan belong to the northern subgroup, and Tagalog, Bikol, Hiligaynon, and Cebuano are allocated to the central subgroup (Walton, 1979; Constantino, 1998). Figure 1 illustrates the central subgroup of the Philippine language family tree.
In this study, our readability experiments focus on three major Philippine languages, Tagalog, Cebuano, and Bikol, which we refer to further in the paper with their corresponding ISO-639-2 language codes as TGL, CEB, and BCL, respectively.

heading: 2.2 Mutual Intelligibility
text: Preliminary linguistic profiling studies of the main Philippine languages such as by McFarland (2004) show that Tagalog, Bikol, and Cebuano are more closely related to one another than any languages in the northern family tree. A language’s closeness or its degree of relatedness to another language from the same family (sub)tree is commonly referred to as mutual intelligibility (Bloomfield, 1926). Such similarities can be seen across multiple aspects, including, for example (a) syllable patterns where all three languages have similar three case-marking particles – ang (En: the), ng (En: of ), and sa (En: at) for Bikol and Tagalog, and ug instead of sa for Cebuano; and (b) shared words, e.g. mata (En: eye) and tubig (En: water).
For languages belonging to one greater subgroup in the case of Central Philippine for Tagalog, Bikol, and Cebuano, showing stronger quantitative evidence of mutual intelligibility may provide additional proof that these languages are indeed, at some level, closely related to each other. Thus,
to contribute towards further understanding of mutual intelligibility in the Philippines language space, we apply two linguistic similarity-based measures using character n-gram overlap and genetic distance which we discuss in the sections below.
Character N-Gram Overlap. For our first measure, we use the overlap in character bigrams and trigrams for every pair from the selected set of languages. To do this, we simply extract and rank the top occurring character bigrams and trigrams for a given language and calculate the Rank-Biased Overlap (RBO)2 (Webber et al., 2010). RBO provides a measure of similarity between two lists while preserving the ranking. We also add English (ENG) as an unrelated control language not belonging to the Philippine family tree for comparison. We use the CommonCore readability dataset (Flor et al., 2013) for English as it also has three readability levels, and the level distribution is the most similar to the dataset of the three Philippine languages. Further information on the datasets in Tagalog, Bikol, and Cebuano can be found in Section 3. For all languages, we extract the top 25% of the most frequently occurring bigrams and trigrams for analysis. The top 40 most frequent bigrams and trigrams can be found in the Appendix.
Table 1 presents character overlap for bigrams and trigrams in a pairwise manner. These results show that all three Philippine languages have character overlap greater than 75% for bigrams among themselves while overlap with English is below 27%. This pattern is observed again
2https://github.com/changyaochen/rbo
in trigrams with the overlap levels of 53.3% to 62.8% between Tagalog, Bikol, and Cebuano and those below 15% for English. These ranges of mutual intelligibility values for bigram and trigram overlap serve as an estimate of the degree of relatedness between the three Philippine languages, with the values for English serving as a baseline for an unrelated language.
Genetic Distance. As a secondary measure of mutual intelligibility, we calculate the genetic distance score (Beau and Crabbé, 2022) for each pair of languages studied in this work. Similar to the character n-gram overlap analysis, we add English for comparison purposes. Genetic distance (Beaufils and Tomin, 2020) is an automatic measure for quantifying the distance between two languages without the need for human judgments. This metric requires a list of words and their equivalent translations for any two languages of interest and calculates the number of exact consonant matches using the following formula:
GeneticDistance = 100− (match(l1, l2) n ) (1)
where l1 and l2 are a pair of languages, n is the total number of words for analysis (usually 100), and match(·) is a function for extracting the consonant patterns for each word from the list as described in Beaufils and Tomin (2020). The metric is measured as a distance; thus, the values closer to 100 denote higher dissimilarity or non-relatedness.
Table 2 shows the calculated genetic distance scores for each pair of languages including English. The mapping provided in the table is the prescribed guide from Beaufils and Tomin (2020).
Judging by these results, the Philippine languages have genetic distance scores within the related and highly related languages range with the Tagalog–Cebuano pair showing the closest language distance of 24.846. Meanwhile, genetic distance scores between all considered Philippine languages and English fall within the very remotely related to no recognizable relationship categories, with the Tagalog–English pair showing the highest distance from each other. Similar to the character n-gram overlap, these results strengthen our initial observation and provide empirical evidence for mutual intelligibility between Tagalog, Bikol, and Cebuano languages which, beyond this study, may also be used in future linguistic research.

heading: 3 Readability Corpora in Philippine Languages
text: We have compiled open source readability datasets for Tagalog, Cebuano, and Bikol from online library websites and repositories. Each data instance in this study is a fictional short story. Table 3 shows the statistical breakdown and additional information on the levels in each readability dataset across different languages.
Tagalog and Cebuano. Datasets in these languages have already been used in previous research, including Imperial et al. (2019); Imperial and Ong (2020); Imperial (2021); Imperial and Ong (2021a); Imperial et al. (2022). We use the same datasets as in previous research and incorporate them into this study for comparison. For Tagalog, we have assembled 265 instances of children’s fictional stories from Adarna House3 and the Department of Education (DepED)4. For Cebuano, we use the dataset collected by Imperial et al. (2022) from Let’s Read Asia5 and Bloom Library6, which were funded by the Summer Institute of Linguistics (SIL International) and BookLabs to make literary materials in multiple languages available to the public.
Bikol. There are no pre-compiled datasets available for readability assessment in Bikol yet. For this, we collected all available Bikol short stories from Let’s Read Asia and Bloom Library totaling 150 instances split into 68, 27, and 55 for
3https://adarna.com.ph/ 4https://lrmds.deped.gov.ph/ 5https://www.letsreadasia.org/ 6https://bloomlibrary.org/
levels 1 to 3 respectively.
All collected data for this study follows the standard leveling scheme for early-grade learners or the first three grades from the K-12 Basic Curriculum in the Philippines.7 Each instance has been annotated by experts with a level from 1 to 3 as seen in Table 3. We use these annotations as target labels in our experiments. Finally, all datasets used in this study can be manually downloaded from their respective websites (see footnotes for links) under the Creative Commons BY 4.0 license.

heading: 4 Experimental Setup

heading: 4.1 ML Setup
text: In this study, our primary focus is on the depth of analysis of the traditional and neural features used in a cross-lingual setting applied to closely related languages. Thus, we use a vanilla Random Forest model which has been previously shown to be the best-performing monolingual-trained model for ARA in Tagalog and Cebuano (Imperial and Ong, 2021a; Imperial et al., 2022). We leave the technical breadth of exploring other supervised algorithms to future work.
We use a stratified k-fold approach with k=5 to have well-represented samples per class for a small-dataset scenario used in this study. We report accuracy as the main evaluation metric across all experiments for the ease of performance comparison with previous work (see Section 5). We use WEKA 3.8 (Witten et al., 1999)8 for all our modeling and evaluation and set hyperparameters of the Random Forest algorithm to their default values as listed in the Appendix.

heading: 4.2 Linguistic Features
text: We extract and consider a wide variety of features inspired by: (a) handcrafted predictors from previous work, (b) representations from a multilingual Transformer-based model (mBERT), and (c) CROSSNGO, a novel feature applicable to readability assessment in closely related languages. We discuss each feature group below.
Traditional Handcrafted Features (TRAD). We integrate available traditional surface-based and
7https://www.deped.gov.ph/k-to-12/about/ k-to-12-basic-education-curriculum/
8https://www.cs.waikato.ac.nz/ml/weka/
syllable pattern-based features in this study as predictors of text complexity. These features have been widely used in previous research on ARA in Tagalog and Cebuano (Imperial and Ong, 2020; Imperial et al., 2022). For Bikol, this is the first-ever study to develop a readability assessment model. In the case of low resource languages similar to those used in this study, these predictors are still the go-to features in ARA and have been empirically proven effective for Tagalog and Cebuano (Imperial and Ong, 2021b). We have extracted a total of 18 traditional features for each language, including:
1. The total number of words, phrases, and sentences (3).
2. Average word length, sentence length, and the number of syllables per word (3).
3. The total number of polysyllable words of more than 5 syllables (1).
4. Density of consonant clusters or frequency of consonants without intervening vowels in a word (e.g. Tagalog: sastre, En: dressmaker) (1).
5. Densities of syllable patterns using the following templates {v, cv, vc, cvc, vcc, ccv, cvcc, ccvc, ccvcc, ccvccc}, where v and c are vowels and consonants respectively (10).
Multilingual Neural Embeddings (mBERT). In addition to the surface-based features, we explore contextual representations from a multilingual Transformer-based large language model via mBERT (Devlin et al., 2019). Previous research on probing BERT has shown convincing evidence
that various types of linguistic information (e.g. semantic and syntactic knowledge) are distributed within its twelve layers (Tenney et al., 2019; Rogers et al., 2020). Applying this to ARA, Imperial (2021) showed that BERT embeddings could act as a substitute feature set for lower-resource languages such as Filipino, for which NLP tools like POS taggers are lacking.
For this study, we specifically chose mBERT as this particular model has been trained using Wikipedia data in 104 different languages including Tagalog and Cebuano. Bikol is not included in any available off-the-shelf Transformer-based language models due to extremely limited online resources not large enough for training. Nonetheless, we still used the representations provided by mBERT noting its high intelligibility with Tagalog and Cebuano. Feature-wise, we use the meanpooled representations of the entire twelve layers of mBERT via the sentence-transformers library (Reimers and Gurevych, 2019). Each instance in our readability data has an mBERT embedding representation of 768 dimensions.
Cross-lingual Character N-Gram Overlap (CROSSNGO). N-gram overlap has been used previously in various NLP tasks applied to Philippine language data such as language identification (Oco et al., 2013a; Cruz et al., 2016), spell checking and correction (Cheng et al., 2007; Octaviano and Borra, 2017; Go et al., 2017), and clustering (Oco et al., 2013b). Drawing inspiration from this fact and from the quantitative evidence of mutual intelligibility between Philippine languages presented in Section 2, we posit that a new feature designed specifically for closely related language data might improve the performance of the readability assess-
ment models. Thus, we introduce CROSSNGO, which quantifies linguistic similarity using character overlap from a curated list of high-frequency n-grams within languages of high mutual intelligibility. We propose the following formula for calculating this metric:
CrossNGOL,n = m(L)
⋂ m(d)
count(m(d)) (2)
where n ∈ {2, 3} denotes bigrams and trigrams, and m(·) is a function that extracts unique n-grams from a document instance d and compares them to a list of top n-grams from a specific language L. For each instance in a dataset, a vector containing three new features will be added representing the overlap between the text and the top n-grams from each of the three languages. We apply this calculation to both bigrams and trigrams using the n-gram lists for Tagalog, Bikol, and Cebuano obtained from the preliminary experiments, which results in a total of 6 new features.
While we presented two quantitative methods of mutual intelligibility in Section 2, only CROSSNGO is applied as a metric and a feature for this study. Staying faithful to the work of Beaufils and Tomin (2020), we did not use Genetic Distance to generate another set of features as it was originally developed as a language-to-language metric. Thus, we use it only as additional secondary evidence of language similarity. At the same time, we note that the proposed CROSSNGO bears certain conceptual similarities to Genetic Distance as it measures the frequency of n-gram overlap with other languages. We perform an ablation study and demonstrate the contribution of individual feature sets in Section 5.

heading: 5 Results and Discussion
text: Table 4 shows the accuracy values obtained when training Random Forest models with various combinations of feature groups for each language of interest. The experiments were divided into three setups: (a) singular cross-lingual (l1→l2), (b) pairwise cross-lingual ([l1+ l2]→l3), and (c) full crosslingual ([l1 + l2 + l3]→l1), each corresponding to a separate subsection of Table 4. We use the term cross-lingual in this context when a model is trained with a readability corpus from a chosen language ln or a combination of languages and evaluated with a test set from another language lm as is demonstrated in Table 4. Similar to our preliminary experiments (Section 2), we include English using
the CommonCore dataset as counter-evidence for comparison with closely related languages.

heading: 5.1 Low-Resource Languages Benefit from Specialized Cross-lingual Features
text: For the singular cross-lingual experiments, the effectiveness of exploiting the bigram and trigram overlap via CROSSNGO is demonstrated by high scores for Bikol and Cebuano (75.862 and 78.270) and comparable performance for Tagalog (50.100). Moreover, only for this setup, there is an observed trend where traditional features combined with CROSSNGO outperform mBERT embeddings or the combination of all features for the respective language pair l1→l2. For Tagalog, this results in 50.100 vs. 26.921 and 23.077; for Bikol – 75.862 vs. 68.965 and 69.000; for Cebuano – 78.270 vs. 71.015 and 73.913. In terms of cross-linguality, in the case of Tagalog, using a model trained with Bikol data proves to be more effective than training with the original Tagalog data with approximately 5.8-point difference in accuracy. However, we still recommend the Tagalog model using all features with 50.000 accuracy since the 0.1 difference is not a significant improvement. Consequently, this trend is not observed in the Bikol and Cebuano experiments where the best-performing models of readability assessment are trained on the data from the same language l1→l1.
To further confirm if the addition of the CROSSNGO feature statistically improves models’ performance as compared to the representations from mBERT for low-resource languages, we aggregate the scores from the TRAD+CROSSNGO group and compare them with the scores obtained when we use mBERT embeddings only, conducting a t-test. We did not include the scores using the combination of all types of features as it would confound the significance test. We achieve statistical significance at α = 0.01 level (p = 0.006) which shows that using traditional handcrafted features extended with CROSSNGO significantly improves ARA models for low-resource languages, provided the availability of data in a closely related language in the case of non-availability of multilingual LLMs (e.g., lack of mBERT model in Bikol).

heading: 5.2 Inclusion of a Closely Related Language in Data Produces More Confident Predictions
text: For pairwise cross-lingual experiments, we investigate the effect of adding a closely related language
on a model’s performance using confusion matrices. As the middle section of Table 4 demonstrates, there are three possible pairwise combinations of Tagalog, Bikol, and Cebuano tested on each individual language. As there can be numerous ways to analyze the table, we highlight the results of the cross-lingual models with the top-performing pair and their utilized feature groups and compare them to their equivalent models in the singular crosslingual experiment. Figure 2 illustrates this method of comparison for each language.
In the case of the Tagalog–Tagalog pair, most misclassifications occur between grades 1 and 2 in both training and test data using all features. This, in turn, is alleviated by incorporating the Bikol dataset in the training data, which reduces the level of confusion by approximately 7%. The inclusion of Bikol also improves classification between grades 2 and 3 by three instances. In the case of the Bikol test data, the same finding is observed for the combined Bikol and Cebuano model using all features, where confusion in classifying grades 1 and 3 is reduced by two instances. Lastly, for Cebuano, the top-performing model in the pairwise cross-lingual setup includes Bikol data and uses all features. For this model, misclassifications in predicting grade 1 against the other two levels are reduced, and performance for predicting grade 3 is improved.
We further corroborate our observations that pairwise cross-lingual models outperform singular cross-lingual models by aggregating the scores from the two setups and running a t-test. Further to the results reported in the previous section, we observe statistically significant difference at the α = 0.01 level (p = 0.003) when pairwise crosslingual models are compared to singular cross-
lingual models. Overall, our findings provide solid empirical evidence that including a closely related language in the training data for a low-resource language significantly improves performance.

heading: 5.3 Combining Specialized Cross-Lingual Features with Multilingual Neural Embeddings Achieves SOTA Results
text: While the previous sections highlight the significant increase in performance when using traditional features with CROSSNGO as compared to mBERT embeddings only, we now discuss results and contributions when both linguistic representations are combined. As is demonstrated in Table 4, the scores obtained using the combined features applied to Tagalog and Cebuano achieve state-ofthe-art results for ARA in these languages. For Tagalog, our model’s accuracy of 57.692 outperforms the SVM with 57.10 accuracy and the Random Forest model with 46.70 presented in Imperial (2021). For Cebuano, our model achieves 79.710 beating the Random Forest model presented in Imperial et al. (2022) with a score of 57.485 with both models utilizing the same Cebuano dataset. Lastly, as there are no automated readability assessment models yet for Bikol, we report a baseline accuracy of 79.328, which is achieved using a model with a combination of traditional features (extended with CROSSNGO) and mBERT embeddings extracted from data in all three Philippine languages.

heading: 5.4 Conventional Fine-Tuning of mBERT Underperforms for Low Resource Cross-Lingual ARA
text: While the main focus of our work is on using traditional machine learning models with Random Forest, we explore if the standard approach for fine-
(a) TGL only trained model (left) against TGL+BCL model (right) using all features for ARA in TGL.
(b) BCL only trained model (left) against BCL+CEB model (right) using all features for ARA in BCL.
(c) BCL only trained model (left) against BCL+CEB model (right) using multilingual embeddings for ARA in CEB.
tuning LLMs such as mBERT can produce comparable performance. We use the same uncased mBERT model as presented in Section 4.
Table 5 shows the performance of singular, pairwise, and full cross-lingual setups formatted similarly to Table 4. These results confirm the findings of Ibañez et al. (2022), who have applied a similar setup to monolingual Tagalog ARA using a Tagalog BERT model. Judging by their results, the conventional fine-tuning approach proved to be inferior to the traditional way of extracting linguistic features from text and training a machine learning model like SVM or Random Forest. For this study, the highest-performing setups for Tagalog and Cebuano use Cebuano data only, and that for Bikol uses the combined Cebuano + Bikol datasets. None of the fine-tuned models outperform those presented in Table 4 using combinations of traditional features and CROSSNGO. While previous work in cross-lingual ARA by Lee and Vajjala (2022) and Madrazo Azpiazu and Pera (2020) achieved relatively high performance with non-closely related languages using LLMs, we obtain less promising results which we can attribute to: (a) the use of datasets of substantially smaller sizes (a total of 13, 786 documents used in Azpiazu and Pera (2019) and 17, 518 in Lee and Vajjala (2022) vs. only 764 in out study), and (b) lack of diverse data sources since only Wikipedia dumps were used for Tagalog and Cebuano for training the mBERT model.

heading: 6 Conclusion
text: In this work, we took a step back from the trend of exploring various technical components of the complex, deep learning models and, instead, focused on studying the potential effectiveness of linguistic characteristics such as mutual intelligibility for ARA in closely related Philippine languages – Tagalog, Bikol, and Cebuano. We implemented three cross-lingual setups to closely study the effects of interaction between the three languages and proposed a new feature utilizing n-gram overlap, CROSSNGO, which is specially developed for cross-lingual ARA using closely related languages. Our results show that: (a) using CROSSNGO combined with handcrafted features achieves significantly higher performance than using mBERT embeddings, (b) the inclusion of another closely related Philippine language reduces model confusion, and (c) using the conventional fine-tuning for LLMs like mBERT in this setup still does not
outperform models with traditional features. Consequently, we come to the conclusion that using languages with high intelligibility is more suited for cross-lingual ARA. This is demonstrated in experiments with English added as an example of a non-related language, in which we do not achieve a substantial increase in performances for Tagalog, Cebuano, and Bikol.
Our results agree with the findings of previous studies in cross-lingual ARA such as those of Madrazo Azpiazu and Pera (2020) using English, Spanish, Basque, Italian, French, Catalan, and Weiss et al. (2021) using English and German, that also showed that the inclusion of additional language data can improve ARA results on other languages. However, our work is primarily motivated by the degree of language relatedness: we show that better results can be achieved for ARA in low-resource languages if we use closely related languages rather than any language, including nonrelated ones like English. Our study also provides an encouragement for researchers to consider approaches grounded in linguistic theories which can potentially be used to improve the performance in NLP tasks rather than always resorting to models that are expensive to train and hard to interpret.

heading: 7 Limitations
text: We discuss some limitations of our current work which can be further explored in the future.
On Data Format. We specifically use fictional short stories as our primary data for the study since we require gold standard labels for this document classification task. Moreover, fictional short stories are easier to find as they often come with a specified grade level compared to other types of literary texts such as magazines or web articles written in any of the three Philippine languages. We do not claim that our models are able to generalize on these other types of literary materials or on other types of closely related language pairs unless a full study is conducted which is outside the scope of this work.
On Handcrafted Features. We were only able to use traditional handcrafted features covering countbased predictors such as sentence or word count and syllable pattern-based features for training the Random Forest models. We did not extract other feature sets one may find in the previous work on
English such as lexical density or discourse-based features since such features require NLP tools that are able to extract POS, named entities, relations, and discourse patterns that do not yet exist for all three Philippine languages used in this study. The work of Imperial and Ong (2021b) covered a small set of lexical features such as type–token ratio and compound word density for readability assessment in Tagalog. Still, we cannot use this approach since all languages would need to have the same number of features as is a standard practice in model training.
On Model Training. Our choice of the Random Forest algorithm for training the ARA models is based on the substantial amount of previous work supporting the application of this method to low-resource ARA, e.g., to Tagalog and Cebuano in a monolingual setup (Imperial and Ong, 2020, 2021a; Imperial, 2021; Imperial et al., 2022), where it achieved better results than other algorithms such as SVM or Logistic Regression. One can consider these algorithms for comparison but the analysis of each ARA model trained with various algorithms to the same level of depth and focus that we have given to the Random Forest classifier in the present study would require a considerable amount of time as well as a higher page limit.
On Current Measures of Mutual Intelligibility. The majority of existing literature in linguistics, specifically on the topic of mutual intelligibility in Philippine languages, discusses examples in the context of speech communication. As such, one might claim that Cebuano and Tagalog are not mutually intelligible by giving an example where a Tagalog speaker may not fully comprehend (or only recognize a few common words) another speaker if they are talking in Cebuano. While this is certainly true, in this study, we specifically focus on the mutual intelligibility of languages at a word and character level via written texts such as children’s fiction books. From this, we see a substantial degree of closeness between Tagalog, Cebuano, and Bikol compared to English. Thus, based on our results, we posit that mutual intelligibility may be used as an additional feature (see CROSSNGO in Section 4) for text-based tasks such as readability assessment. We leave the exploration of our proposed novel feature in the speech communication
area to future work.

heading: 8 Ethical Considerations
text: We foresee no ethical issues related to the study.

heading: Acknowledgements
text: We thank the anonymous reviewers and area chairs for their constructive and helpful feedback. We also thank the communities and organizations behind the creation of open-source datasets in Philippine languages used in this research: DepED, Adarna House, Bloom Library, Let’s Read Asia, SIL, and BookLabs. JMI is supported by the UKRI CDT in Accountable, Responsible, and Transparent AI of the University of Bath and by the Study Grant Program of the National University Philippines.

heading: A Appendix
text: ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 7
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? We used WEKA as discussed in Section 4.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Section 3 for the collected Philippine language data and Section 4 for WEKA
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The dataset is not scraped from social media sites.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 2-3
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4
C 3 Did you run computational experiments? Left blank.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: We discuss some limitations of our current work which can be further explored in the future. On Data Format. We specifically use fictional short stories as our primary data for the study since we require gold standard labels for this document classification task. Moreover, fictional short stories are easier to find as they often come with a specified grade level compared to other types of literary texts such as magazines or web articles written in any of the three Philippine languages. We do not claim that our models are able to generalize on these other types of literary materials or on other types of closely related language pairs unless a full study is conducted which is outside the scope of this work. On Handcrafted Features. We were only able to use traditional handcrafted features covering countbased predictors such as sentence or word count and syllable pattern-based features for training the Random Forest models. We did not extract other feature sets one may find in the previous work on
English such as lexical density or discourse-based features since such features require NLP tools that are able to extract POS, named entities, relations, and discourse patterns that do not yet exist for all three Philippine languages used in this study. The work of Imperial and Ong (2021b) covered a small set of lexical features such as type–token ratio and compound word density for readability assessment in Tagalog. Still, we cannot use this approach since all languages would need to have the same number of features as is a standard practice in model training. On Model Training. Our choice of the Random Forest algorithm for training the ARA models is based on the substantial amount of previous work supporting the application of this method to low-resource ARA, e. g. , to Tagalog and Cebuano in a monolingual setup (Imperial and Ong, 2020, 2021a; Imperial, 2021; Imperial et al. , 2022), where it achieved better results than other algorithms such as SVM or Logistic Regression. One can consider these algorithms for comparison but the analysis of each ARA model trained with various algorithms to the same level of depth and focus that we have given to the Random Forest classifier in the present study would require a considerable amount of time as well as a higher page limit. On Current Measures of Mutual Intelligibility. The majority of existing literature in linguistics, specifically on the topic of mutual intelligibility in Philippine languages, discusses examples in the context of speech communication. As such, one might claim that Cebuano and Tagalog are not mutually intelligible by giving an example where a Tagalog speaker may not fully comprehend (or only recognize a few common words) another speaker if they are talking in Cebuano. While this is certainly true, in this study, we specifically focus on the mutual intelligibility of languages at a word and character level via written texts such as children’s fiction books. From this, we see a substantial degree of closeness between Tagalog, Cebuano, and Bikol compared to English. Thus, based on our results, we posit that mutual intelligibility may be used as an additional feature (see CROSSNGO in Section 4) for text-based tasks such as readability assessment. We leave the exploration of our proposed novel feature in the speech communication
area to future work