BasahaCorpus: An Expanded Linguistic Resource for Readability Assessment in Central Philippine Languages
Authors: Joseph Marvin Imperial
Ekaterina Kochmar
Venue: EMNLP 2023 Main (2023)
Keywords: readability assessment
corpus
linguistic resource
cross-lingual
Philippine languages
low-resource NLP
TL;DR: We introduce a corpus of levelled narratives in four Central Philippine languages to train baseline readability assessment models in various cross-lingual setups.

Abstract: Current research on automatic readability assessment (ARA) has focused on improving the performance of models in high-resource languages such as English. In this work, we introduce and release BasahaCorpus as part of an initiative aimed at expanding available corpora and baseline models for readability assessment in lower resource languages in the Philippines. We compiled a corpus of short fictional narratives written in Hiligaynon, Minasbate, Karay-a, and Rinconada—languages belonging to the Central Philippine family tree subgroup—to train ARA models using surface-level, syllable-pattern, and n-gram overlap features. We also propose a new hierarchical cross-lingual modeling approach that takes advantage of a language's placement in the family tree to increase the amount of available training data. Our study yields encouraging results that support previous work showcasing the efficacy of cross-lingual models in low-resource settings, as well as similarities in highly informative linguistic features for mutually intelligible languages.

BASAHACORPUS:
An Expanded Linguistic Resource for Readability Assessment in Central Philippine Languages
Joseph Marvin ImperialA.0 Ekaterina Kochmar"
University of Bath, UK @National University, Philippines
MBZUAI, UAE
ac uk ekaterina benterrina.occcmarrreme.r..... kochmarmmbzuii..cc..
Abstract
Central Philippine
Current research automatic readability as- sessment (ARA) has focused on improving performance of models in high-resource lan- guages such as English. In this work, we intro- duce release BASAHACORPUS as part an initiative aimed at expanding available corpora and baseline models for readability assessment in lower resource languages in the Philippines. We compiled corpus of short fictional narra- tives written in Hiligaynon, Minasbate, Karay- a, and Rinconada- lanconada-llanuugess belonging the Central Philippine family tree subgroup-ty to train ARA models using surface-level, syllable- pattern, and n-gram overlap features. We also propose new hierarchical cross-lingual mod- eling approach that takes advantage guage's 's placement in the family tree to increase the amount of available training data. Our study yields encouraging results that support previous work showcasing the efficacy of cross-lingual models low-resource settings, as well as sim- ilarities in highly informative linguistic features for mutually intelligible languages.'
Tagalog
Togalog (Filipino)
Albay Bikol
Bikol
Inland
Rinconada*
Coastal
Virac
Bisayan
Central
Hiligaynon
Cebuano
Cebuan
Boholano
lan-
Karay-a
Aklanon
Figure 1: The central subgroup of the Philippine lan- guage family highlighting the origins of the target languages Hiligaynon, Minasbate, Karay-a, and Rin- conada (both underlined and marked with *). Tagalog, Bikol, and Cebuano are also underlined as they part of further experiments. The complete visualization of this language tree can be found Zore (1976).
1 Introduction
To ensure optimal reading comprehension, literary resources such as books need be assigned read- ers based on their reading level assessment (Carrell, 1987). Readability assessment can be tackled with a variety of methods ranging from the application of rule-based approaches using such widely ac- cepted formulas as Flesch-Kincaid (Kincaid et al.. 1975), to the use of software for linguistic fea- ture extraction such as Coh-Metrix (Graesser et al., 2004) or LFTK (Lee al., 2021), the applica- tion of extensive machine learning models (Vajjala and Meurers. 2012: Xia et al., 2016). Recently, the latter have been the focus of research on ARA due to the availability of increasingly complex models, including deep learning architectures, that yield
better text representations and outperform simpler (e.g., formula-based) approaches (Vajjala, 2022). These are. however, mostly practical if trained on high-resource data like English.
Beyond the limelight on high-resource lan- guages like English, languages that do require ex- tensive research efforts and initiatives are com- monly considered low-resource. One such group of languages comprises over 170 living languages spoken in the Philippines, and one of the three concluding recommendations in the final report of the five-year (2013-2018) USAID Basa Pilipinas ("Read Philippines") Program2 targeting the im-
hhttps://githin wwtp://..ooooommiiiaaaaaaa http://hww...co..pppppaaaaaaaaaa taaasaaarrtteeeaaaaarrrrrrr #
hhttps://www edc, www..eddorg/uaaiihhiippisssssese -basa- il ilipinas-proge projett-trttjtttrrpprrrr

---

provement of literacy in the Philippines was the "provision of surriculum-based teaching and learn- ing materials (TLMs) and quality supplementary reading materials". This entails the need for a va- riety of age-appropriate texts for young learners in the Philippines to be accessible at home and in school. Following this, the Department of Educa- tion further encouraged the development of reading materials for the Mother Tongue-Based Multilin- gual Education (MTB-MLE) scheme implemented at primary levels of school education in the Philip- pines. 3 Thus, the Philippine education sector needs access not only to sufficient age- age-appropriate learn- ing and reading materials but also to automated tools for assessing text difficulty that can work ef- fectively for the variety of languages covered by MTB-MLE
the Bikol languages, the Bisayan languages, and the East Mindanao languages (McFarland, 2004). Focusing on the languages of interest for this study, we show where Hiligaayon, Minasbate, Karay-a, and Rinconada are situated in this tree (see Fig- ure 1) and how they are classified within specific subgroups (Zorc, 1976). In the following sections, integrate the hierarchical relationships of these four languages with other languages in the family tree into our experiments.
2.1 Data from Let's Read Asia
Let's Read Asia is an online library of community- translated literary materials sponsored by The Asia Foundation. Using the data collected from the web- site, we built BASAHACORPUS by compiling short stories written in Philippine languages, which were only available in Hiligaynon, Minasbate, Karay-a, and Rinconada, thus defining our choice work on these languages. The compiled data are distributed across the first three years of elementary education (L1, L.2, and I.3). We provide information on the language family, origin subgroup, linguistic vital- ity, availability of language resources, and whether these languages are used as a medium of instruction in classes in Table 1.
This work contributes a number of resources to the Philippine language research landscape, cater- ing to the challenges identified above, and aligns with other recent research efforts aimed at low- resource languages such as the IndoNLP (Wilic et al., 2020; Winata et al., 2023). First, to in- crease accessibility and encourage more research endeavors in low-resource Philippine languages, we collect and release BASAHACORPUS, 4 com- pilation of children's ' narratives and short stories from Let's Read Asia online library four Philip- pine languages (Hiligaynon, Minasbate, Karay-a, and Rinconda). Second, we train a number of base- line readability assessment models using various cross-lingual setups leveraging the hierarchy in the language family tree. Lastly, we apply simple model interpretation technique identify how dif- ferent linguistic features act as predictors of text complexity for each language.
We also compute basic statistical information about the data per language and per level, includ- ing mean word and sentence count and vocabulary size, and report it in Table 2. All languages in B. BASAHACORPUS exclusively use Latin script as is the case for the majority of languages in the Philip- pines. In terms of the distribution of resources, we obtained explicit permission from Let's Read Asia to use this data in this research with the further goal of publicly sharing the resources, which are licensed under Creative Commons BY 4.0.
2
A Closer Look into Central Philippine Languages
3 Linguistic Features for Low- Reo-Res Languages
The Philippines is an archipelago with rich lin- guistic background, evidenced by over 170 lan- guages at the backbone of its culture.5 In particular, the central Philippine language subtree is consid- ered the largest among other subtrees as is com- posed of number of branches, including the na- tional language Tagalog (also known as Filipino),
Due to the lack of available NLP tools to extract advanced linguistic information from texts that would work for all four target languages as well as enough data to use deep learning approaches, derive the same linguistic features as used by the previous work ARA in other low-resource Philippine languages such Tagalog (Imperial et al., 2019; Imperial and Ong, 2020, 2021a) and Cebuano (Imperial et al., 2022). However, we note that this study is the first to explore baseline
hht //www/ http://wwwwwdddd.ee/////// gov. ph1/2016/11/224/mohhrrrr wew.ning--aeeseeeeeeee- ve- ee-and-ed -
"Basaha generally denotes verbal form of "read"
Bisayan languages.
Ptttps: / //www ethnologue, wwwpl/gguu..eeeccoooeeeeeeeeeeee country/PH/

---

Digital Language
Language
Family
Indigenous
Vitality
Instruction
Support
Hiligaynon (HII.)
Central
No
Insitutional Insitutional Insiutionna
Yes
Ascending
Minasbate (MSB)
Central
No
As subject
Still
Kinaray-a (KRJ)
West
Stable indigenous
Yes
Emerging
Rinconada (BTO)
Inland Bikol
Stable indigenous
Stable
As subject
Still
Table 1: Description the language data for Hiligaynon, Minasbate, Karay-a, and Rinconada. We also include important information from Ethnologue about each language's ' classification and status, including origin subgroup (identified as indigenous or not), linguistic vitality, whether a language is used as medium of instruction or as a subject, and the availability of online digital resources (for terminology, please refer the Ethnologue page).
Document Count
Mean Word Count
Mean Sent Count
Language
Total
Level
Vocabulary
L1
65
198.8
20.7
2043
Hiligaynon (HIL)
133
L2
22
296.5
39.1
1539
L3
46
610.0
57.3
4137
LI
124
240.4
28.7
3836
Minasbate (MSB)
271
L2
77
360.5
43.0
4097
L3
70
578.6
62.5
5520
191.0
21.4
1937
Karay-a (KRJ)
177
34
410.9
47.7
2309
L3
82
569.1
60.3
6264
117
261.0
30.0
4222
Rinconada (BTO)
195
36
521.7
59.5
3313
42
505.2
3958
Table 2: Overview and basic statistics of the data included the BASAHACORPUS per language and per grade
level with the total number of documents. The vocabulary denotes the total count of unique words per class and per language. An increasing linear relationship can be observed for both mean word count and mean sentence count per document as the level increases.
modeling of text complexity in the four selected languages Hiligaynon, Minasbate, Karay-a, and Rinconada. We list all extracted features below.
(c) and vowel (v) patterns include cN vec, cev, evec, ccvec, eeveсe. We also extract consonant clusters and measure the average length of consonant groups in word without an intervening vowel which have been proven influence reading difficulty (Chard and Osborn, 1999).
Traditional Features. We extract 7 frequency- based features using count-based text character- istics. Despite being often considered simplistic, these predictors have consistently proven be effective in text readability detection (Pitler and Nenkova, 2008; Imperial and Ong, 2021b). As in the previous work on a Cebuano dataset Imperial et al. (2022), our extracted text-level features include the total number of unique words, total number of words per document, average word length per document, average number of syllables (per word), total sentence count per document, average sentence length per document, and the total number of polysyllable words.
Cross-lingual N-gram Overlap. A new CROSS- NGO feature has recently been introduced by Im- perial and Kochmar (2023), which exploits mutual intelligibility or the degree of language related- ness via n-gram overlap between languages from the same family tree. Since the study reported a significant boost performance for readability assessment in low-resource languages using this feature, we see clear practical application for it in this work. We estimate the bigram and trigram overlap between the four target central Philippine languages (Hiligaynon, Minasbate, Karay-a, and Rinconada), as well the three parent languages (Tagalog, Bikol, and Ccbuano) for a total new features. As an additional reference, we add two
Syllable-pattern Orthographic Features. We also extract 11 orthographic predictors taking into account all possible syllable patterns in the Philippine language space. These consonant

---

tables quantifying the similarities via cross-lingual n-gram overlap of the Philippine languages used in this study Tables and of Appendix A.
All Langs (*L). This setup combines all seven lan- guages used in the previous iterations (Hiligaynon, Minasbate, Karay-a, Rinconada, Bikol, Cebuano, and Tagalog) in the training data.
4 Hierarchical Cross-lingual Modeling
We note that the Tagalog, Bikol, and Cebuano data from Imperial and Kochmar (2023) that we use for this study also follows the same category distribution (L1, L2, and L3) our four languages from Table 2. For all model training in this study, we use Random Forest with default hyperparame- ters from Scikit-Learn (Pedregosa al., 201 1), as the efficacy of this algorithm was reported the previous studies on ARA in the Philippine lan- guages (Imperial et al.. 2022: Imperial and Ong, 2021a). We perform stratified 5-fold sampling for all three classes to be properly represented. Hyper- parameter values of the Random Forest model can also be found in Table of Appendix A.
Cross-lingual modeling for automatic readability assessment typically involves extracting language- independent features or experimenting with a classifier trained on the data in one language and applied to comparable data in another language (Madrazo Azpiazu and Pera. 2020; Weiss et al., 2021; Lee and Vajjala, 2022; Imperial and Kochmar, 2023). In this study, we introduce new approach to training readability assessment models using close relationships between languages in the family tree, which we refer to as hierarchy-based cross-lingual modeling. For this setup, our cross-lingual training process is divided into iterations involving different combinations of language data in the training split with respect to the hierarchy the languages occupy in the Central Philippine language subtree, as illustrated in Figure We list all such feature combinations below:
Results and Discussion
Below, we summarize and discuss the most notable insights obtained from the conducted experiments.
Language
L
L.P
L+N
L+P+N
*I
0.697
0.666
0.629
0.592
0.814
MSB
0.641
0.555
0.611
0.574
0.648
Monolingual (L). This setup uses the standard train-test split involving one language only.
KRJ
0.618
0.628
0.657
0.629
0.685
BTO
0.682
0.846
0.743
0.714
0.717
Table 3: Accuracy across the experiments described
Lang + Parent Lang (L+P). This setup combines the parent language or lingua franca with respect to their subgroup. For Hiligaynon, Minasbate, and Karay-a, belonging to the Bisayan languages in the Visayas region, we add the Cebuano language, while for Rinconada, classified as member of the Bikol languages belonging to the Luzon region, we add the Central Bikol language both from Imperial and Kochmar (2023).
Section 4 using various combinations language datasets and leveraging hierarchical relations between languages.
5.1 Using extensive multilingual training data results generally better performance.
Table 3 shows the results of the ablation experi- ments using the hierarchical cross-lingual model- ing setup described in Section 4. We observe that using the all languages setup (*L) for the training data helps obtain the best accuracy performance for the three Bisayan languages (HIL, MSB and KRJ) but not for Rinconada (BTO) from the Bikol subgroup. However, the score for Rinconda with the *I setup is still higher than for the monolingual model and even the L+P+N setup. A f-test con- firms that the scores obtained with the *L setup are significantly higher than those obtained with the monolingual setup L for all languages level (p = 0.048). These results support the findings of Imperial and Kochmar (2023), who showed the importance combining data from closely related
Lang + National Lang (L+N), This setup com- bines the four target languages with Tagalog in the training data extracted from Imperial and Kochmar (2023), Imperial and Ong (2020) and Imperial et al. (2019). Tagalog is the recognized national language taught as a subject in all elementary and secondary schools in the Philippines.
Lang + Parent Lang + National Lang (L+P+N). This setup combines each target language with its corresponding parent language and Tagalog in the training data.
0.05

---

languages (or languages within the same family tree) for performance improvements in ARA mod- els.
HIL
MSB
word count
0.096
word count
0.111
sentence count
0.079
sentence count
0.077
polysyll count
0.053
polysyll count
0.049
5.2 Stronger mutual intelligibility improves model performance.
avg sent len
0.045
avg len
0.039
sag trigram sim
0.038
tag trigran sim
0.037
Following Imperial and Kochmar (2023), we com- pute the overlap of the top 25% of the trigrams and bigrams in order to estimate mutual intelligibil- ity between languages from the Bisayan and Bikol subgroups and their respective parent languages, Cebuano and Bikol. We find that Rinconada (BTO) has the highest overlap (0.696 for trigrams and 0.887 for bigrams) with its parent language (Bikol) fact that explains why the best results for this lan- guage are obtained with the L+P combination. For comparison, the other three languages (Hiligaynon, Minasbate, and Karay-a) show n-gram overlaps of 0.609, 0.579, and 0.540 for trigrams, and 0.863, 0.789, and 0.809 for bigrams with their parent lan- guage, respectively. See the trigram and bigram overlap results in Tables 6 and 7 in Appendix A.
KRJ
BTO
word count
0.102
word count
0.115
sentence count
0.074
sentence count
0.072
polysyll count
0.052
polysyll count
0.047
avg sent len
0.042
sent len
0.044
tag trigram sim
0.036
tag trigram sim
0.037
Table 4: Most informative features per language identi- fied using mean decrease in impurity scores calculated for the Random Forest models.
the development of low-resource and cross-lingual readability assessment systems often focus pus building and the development of baseline mod- els. Previous efforts of this kind have covered a wide array of languages, including Bangla (Islam et al., 2012; Islam and Rahman, 2014), Tibetan (Wang et al., 2019), Arabic (Saddiki et al., 2018; Al Khalil et al., 2020), Vietnamese (Doan et al., 2022), Bengali (Chakraborty et al., 2021), and Bikol (Imperial and Kochmar, 2023). As compared to the previous works, ours is the first study in readability assessment that investigates the effects of different language hierarchies in modeling text complexity for languages belonging to different subgroups of a greater family tree, with the focus on central Philippine languages.
5.3 Support for traditional features in approximating text complexity for Philippine languages.
Finally, we identify the most informative linguistic features by calculating the mean decrease in impu- rities when splitting by the Random Forest models trained with all languages (*L) and applied to the test splits for each of the four languages. Table shows that all models for all languages demon- strate the same order in top features which are all considered count-based predictors. We believe that this finding corroborates results from previous work showing that frequency-basee -base features such as word and sentence counts are still viable mea- sures of text complexity for Philippine languages (Macahilig, 2014; Gutierrez, 2015).
7 Summary
We introduce BASAHACORPUS, a compilation of language resources that includes a collected cor- pus of short stories and baseline readability as- sessment models for four central Philippine lan- guages. We show that, through a hierarchical cross- lingual modeling approach, a readability model trained with all the available Philippine language data generally performs better compared using single-language datasets. Through model interpre- tation, we also provide further support for the use of Frequency-basee features such as word and sen- tence counts as effective predictors of complexity in Philippine languages. This study serves as a response to the call for more research efforts, the- oretically grounded baselines, and accessible data for low-resource languages.
6 Related Work
the past, cross- lingual modeling was applied to classification-base ARA both for non-related ( (Va- jjala and Rama, 2018; Madrazo Azpiazu and Pera, 2020; Weiss et al., 2021; Lee and Vajjala, 2022; Mollanorozy et al., 2023) and highly-related lan- guages (Imperial and Kochmar, 2023), with the latter reporting favorable results under predefined language similarity constraints such high n- n-gram overlap. 7 Research efforts that greatly contribute to
'Here, we interpret language relatedness as a measure of similarity of linguistic characteristics such as n-gram overlap
(Imperial and Kochmar, 2023).

---

Limitations
[EP/SS23437/L] of the University of Bath and by the Study Grant Program of National University Philippines.
Limited feature sets for low-resource languages. Due severely limited existing NLP research efforts for the Philippine languages that this work addresses, we have to resort to a small number of feature extraction methods covering surface-level characteristics, syllable patterns, and n-gram overlap that have been previously applied to the related Philippine languages such as Tagalog and Cebuano (Imperial and Ong, 2020, 2021a; Imperial et al., 2022; Imperial and Kochmar, 2023). Nevertheless, we believe that our findings are valuable as this work provides the baseline for readability assessment in Hiligaynon, Minasbate, Karay-a, and Rinconada in addition to being the first one to address these languages. We hope that future research efforts will lead to substantial expansion in the data available for these languages, which in turn will help rescarchers develop and apply more advanced ARA models these languages and benchmark them against the results reported in our paper. We consider our work the first step in the direction of addressing such challenging tasks as ARA in low-resource Philippine languages.
References
Muhamed Al Khalil, Nizar Habash, and Zhengyang Jiang. 2020. A large-scale leveled readability lex- icon for Standard Arabic. In Proceedings of the Twelfth Language Resources and Evaluation Confer- ence, pages 3053-3062, Marseille, France. European Language Resources Association.
Patricia L Carrell. 1987. Readability in ESL. Univer- sity of Hawaii National Foreign Language Resource Center.
Susmoy Chakraborty, Mir Tafseer Nayeem, and Wasi Uddin Ahmad. 2021. Simple or Complex? Learning Predict Readability of Bengali Texts. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 12621-12629.
David J Chard and Jean Osborn. 1999. Phonics and word recognition instruction in early reading pro- grams: Guidelines for accessibility. Learning Dis- abilities Research & Practice, 14(2):107-117.
Cam-Thuan Doan, Thi-Anh-Thi Le, An-Vinh Luong, and Dien Dinh. 2022. Combining Latent Seman- tic Analysis and Pre-trained Model for Vietnamese Text Readability Assessment: Combining Statisti- Semantic Embeddings and Pre-trained Model for Vietnamese Long- Tig-Sequence Readability Assessment. In Proceedings of the 4th International Conference on Information Technology and Computer Communi- cations, pages 45-52.
Low variety of the data. This work uses only fictional short stories in specific grade levels (L1, L2, and L3) as training data for the readability assessment models, which may be considered a limitation in the application domain. While the same features can be extracted and applied to other text forms in various domains such news articles poems, we do not claim that the results will generalize or apply such other datasets.
Arthur C Graesser, Danielle S McNamara, Max M Louwerse, and Zhiqiang Cai. 2004. Coh-Metrix: Analysis Text on Cohesion and Language. Behav- ior Research Methods, Instruments, & Computers, 36(2):193-25 202.
Merry Ruth M Gutierrez. 2015. The Suitability of the Fry and SMOG Readability Formulae in Determining the Readability of Filipino Texts. The Normal Lights, 8(1).
Ethics Statement
We foresee no serious ethical implications from this study.
Joseph Marvin Imperial and Ekaterina Kochmar. 2023.
Automatic Readability Assessment for Closely Re- lated Languages. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5371- 5386, Toronto, Canada. Association for Computa- tional Linguisties.
Acknowledgements
We thank the anonymous reviewers for their con- structive feedback and the ACs, SACs, and PCs for their appreciation of this work. We also thank the community translators and maintainers of the online library of Let's Read Asia for keeping the digital resources in the Philippine languages freely available for everyone. JMT is supported by the UKRI Centre for Doctoral Training in Account- able, Responsible, and Transparent AI (ART-AI)
Joseph Marvin Imperial and Ethel Ong. 2020. Explor- ing Hybrid Linguistic Feature Sets to Measure Fil- ipino Text Readability. In 2020 International Confer- ence on Asian Language Processing (IALP), pages 175-180. IEEE.
Joseph Marvin Imperial and Ethel Ong. 2021a. Diverse Linguistic Features for Assessing Reading Difficulty

---

of Educational Filipino Texts. In 29th International Conference on Computers in Education Conference, ICCE 2021, pages 51-56. Asia-Pacific Society for Computers in Education.
Heidi B Macahilig. 2014. A content-base readability formula for Filipino texts. The Normal Lights, 8(1).
lon Madrazo Azpiazu and Maria Soledad Pera. 2020. Is Cross- Readability Assessment Possible? Journal of the Association for Information Science and Technology. 71(6):6446656
Joseph Marvin Imperial and Ethel Ong. 202 Under the microscope: Interpreting readability assessment models for Filipino. Proceedings of the 35th Pa- cific Asia Conference on Language, Information and Computation, pages 1-10, Shanghai, China. Associa- tion for Computational Lingustics.
Curtis D McParland. 2004, The Philippine language situation. World Englishes, 23(1):59 75.
Sepideh Mollanorozy, Mare Tamti, and Malvina Nissim. 2023. Cross-lingual transfer learning with Persian. In Proceedings of the 5th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, pages 89-95, Dubrovnik, Croatia. Association for Computational Linguistics.
Joseph Marvin Imperial, Lloyd Lois Antonie Reyes, Michael Antonio Ibancz, Ranz Sapinit, and Mo- hammed Hussien. 2022. A baseline readability model for Cehuano. In Proceedings of the 17th Workshop on Innovative Use of NI.P for Building Educational Applications (BEA 2022), pages 27-32, Seattle, Washington. Association for Computational Linguisties.
Fabian Pedregosa, Gaäl Varoquaux, Alexandre Gram- fort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, and Vincent Dubourg. 2011. Scikit- Machine leam- ing in Python. the Journal of machine Learning research, 12:2825-2830.
Joseph Marvin Imperial, Rachel Edita Roxas, Er- ica Mae Campos, Jemelee Oandasan, Reyniel Cara- ballo, Ferry Winsley Sabdami, and Ani Rosa Almario. 2019. Developing machine lcarning-basse based grade level classifier for Filipino children's 's literature. In 2019 International Conference on Asian Language Processing (IALP), pages 413-418. IEEE
Emily Pitler and Ani Nenkova. 2008. Revisiting read- ability: unified framework for predicting text qual- ity. In Proceedings of the 2008 Conference on Empir- ical Methods in Natural Language Processing, pages 186-195, Honolulu, Hawaii. Association for Compu- tational Linguistics.
Zahrul Islam and Rashedur Rahman. 2014. Readability of Bangla news articles for children. In Proceed- ings of the 28th Pacific Asia Conference on Lan- guage, Information and Computing, pages 309-317, longkorn University.
Hind Saddiki, Nizar Habash, Violetta -fozza and Muhamed Al Khalil. 2018. Feature optimization for predicting readability of Arabic L1 and L2. In Proceedings of the 5th Workshop on Natural Lan- guage Processing Techniques for Educutional Appli- cations, pages 20-29, Melbourne, Australia. Associ- ation for Computational Linguistics.
Department of Linguistics, Chula-
Zahurul Islam, Alexander Mehler, and Rashedur Rah- man. 2012. Text readability classification of text- books of a low- low-resource language. In Proceedings of the 26th Pacific Asia Conference on Language, In- formation, and Computation, pages 545-553, Bali, Indonesia. Faculty of Computer Science, Universitas Indonesia.
Sowmya Vajjala. 2022. Trends, limitations and open challenges in automatic readability assessment re- search. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 5366- 5377, Marseille, France. European Language Re- sources Association.
Peter Kincaid, Robert P Fishburne Jr. Richard Rogers, and Brad S Chissom. 1975. Derivation Of New Readability Formulas (Automated Readability Index, Fog Count And Flesch Reading Ease Formula) For Navy Enlisted Personnel. Naval Technical Train- ing Command Millington TN Research Branch.
Sowmya Vajjala and Detmar Meurers. 2012. On improv- ing the accuracy of readability classification using insights from second language acquisition. In Pro- ceedings of the Seventh Workshop on Building Ed- ucational Applications Using NLP, pages 163-173, Montréal, Canada. Association for Computational Linguistics.
Bruce W. Lee, Yoo Sung Jang, and Jason Lec. 2021. Pushing text readability assessment: A trans- former meets handcrafted linguistic features. Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 10669 10686, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Sowmya Vajjala and Taraka Rama. 2018. Experiments with universal CEFR classification. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 147-153, New Orleans, Louisiana, Association for Computational Linguistics.
Justin Lee and Sowmya Vajjala. 2022. neural pair-
wise ranking model for readability assessment. In Findings of the Association for Computational Lin- guisties: ACL 2022. pages 3802-3813, Dublin, Ire- land. Association for Computational Linguistics.
Zhijuan Wang, Xiaobin Zhao, Wei Song, and Antai Wang. 2019. Readability Assessment of Textbooks in Low Resource Languages. Computers, Materials & Continua, 61(1).

---

Zarah Weiss, Xiaobin Chen, and Detmar Meurers. 2021. Using Broad Linguistic Complexity Modeling for Cross-Lingual Readability Assessment. In Proceed- ings of the 10th Workshop on NLP for Computer Assisted Language Learning, pages 38-54, Online. LiU Electronic Press.
A Appendix
Hyperparameter
Value
n_ Lestimators
100
criterion
gini
Bryan Wilie, Karissa Vincentio, Genta Indra Winata, Samuel Cahyawijaya, Xiaohong Li, Zhi Yuan Lim, Sidik Soleman, Rahmad Mahendra, Pascale Fung, Syafri Bahar, and Ayu Purwarianti. 2020. IndoNLU: Benchmark and resources for evaluating Indonesian natural language understanding. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the A Association for Computational Linguistics and the 10th International Joint Conference on Natural Lan- guage Processing, pages 843 3577, Suzhou, China. Association for Computational Linguisties.
max_depth
None
mui__samples 1
max_ fea_feessees
sqit
random_state
none
Table 5: Hyperparameter settings for the Random Forest algorithm used for training the models with scikit-learn. These are also default values preset in WEKA 0.2.
Genta Indra Winata, Alham Fikri Aji, Samuel Cahyawi- jaya, Rahmad Mahendra, Fajri Koto, Ade Romad- hony, Kemal Kurniawan, David Moeljadi, Radi- tyo Eko Prasojo, Pascale Fung, Timothy Baldwin, Jey Han Lau, Rico Sennrich, and Sebastian Ruder. 2023. NusaX: Multilingual parallel sentiment dataset for 10 Indonesian local languages. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 815-834, Dubrovnik, Croatia. Association for Com- putational Linguistics.
HIL
MSB
KRJ
BTO
TGL
CEB
BCL
1.000
0.713
0.741 0.610
0.590 0.609 0.604
MSB 0.713
1.000
0.696
0.673
0,586
0.579 0.649
KRJ 0.741
0.696
1.000
0.622
0.552
0,540 0.593
BTO 0.610 0.673
0.622
1.000
0,615
0.528 0.696
TGL 0.590 0.586
0.552
0.615
1.000
0,628 0,588
CEB 0.609 0.579 0.540
0.528
0.628
1.000 0.533
BCL. 0.604 0.649 0.593 0.696
0,588 0.533 1.000
Table 6: Trigram overlap between all seven Central Philippine languages.
Menglin Xia, Ekaterina Kochmar, and Ted Briscoe. 2016. Text readability assessment for second lan- guage learners. Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications, pages 12-22, San Diego, CA. Associa- tion for Computational Linguistics.
for
HII.
MSB
KRJ
BTO
TGL.
CEB
BCL
HIL.
1.000
0.830
0.859
0.831
0.834
0.863
0.818
MSB
0.830
1.000
0.831
0.863
0.782
0.789
0.847
David Zore. 1976. The Bisayan dialects of the Philip- pines: Subgrouping and reconstruction. Pacific Lin- guistics, Research School of Pacific and Asian Stud- ies.
KRJ
0.859
0.831
1.000
0.811
0.796
0.809
0.816
BTO
0.831
0.863
0.811
1.000
0.786
0.785
0.887
TGL
0.834
0.782
0.796
0.786
1.000
0.812 0.810
CEB 0.863
0.789
0.809
0.785
0.812
1,000 0.789
BCL 0.818 0.847
0.816
0.887
0.810
0.789 1.000
Table 7: Bigram overlap between all seven Central Philippine languages.


Reviews:
[{"review_id": "iOHUflRVnt", "reviewer": "Reviewer_v7Sy", "summary": "", "questions": "It is mentioned in the paper: The compiled data are distributed across the first three years of elementary education (L1, L2 and L3) (l 109 - 111). : how are levels assigned to the texts? What is the difference between i.e.,  L2 and L3 levels? What is the confidence for the manual distribution of texts to levels?\n\nThe work presented here is focused on L1, L2 and L3 levels. Isn't that a limitation to be clearly stated in the respective section?\n(it is implied now).\n\nTable 7: What does Vocab stand for?\n\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are extremely grateful for the reviewer’s feedback on our work and very much appreciate the positive comments regarding our contributions to the corpora building for readability assessment in Philippine languages. Please find our responses to the individual questions below. We hope that our responses address both the questions and the weaknesses identified, and we also hope that this may be reflected in the updated scores.\n\n1. *\"The paper does not provide meaningful comparisons with previous work; the results obtained are mentioned to support the findings of previous work.\"* - Since this is a short paper, due to space limitations, we did not include a separate section on related works. However, we made sure to cite and refer to the results that previous research has reported throughout our paper — in particular, in Sections 3, 4, and 5. We cited previous works to support their findings as reflected by our results using our new proposed method of hierarchy-based cross-lingual ARA. If our paper is accepted, we will be happy to use extra space for a dedicated section on related work.\n\n\n2. *\"How are levels assigned to the texts? What is the difference between i.e., L2 and L3 levels? What is the confidence for the manual distribution of texts to levels?\"* - The levels are assigned by experts from the Let’s Read Asia group. As mentioned in Section 2.1, each level is used in the context of the Philippine educational system. For example, L1 is used for first graders, L2 is for second graders, and so on. We extracted all available data for each level and language from Let’s Read Asia, and this distribution is reflected in Table 7.\n\n\n3. *\"The research is limited to L1, L2, and L3 levels.\"* - We acknowledge that this is a limitation of our study, however, we contend that similar limitations apply to all low-resource language studies. To clarify, there are seven grade levels in the Philippine educational system in total, but our multilingual corpus only covers the first three levels due to limited data availability from the source itself (Let’s Read Asia). However, as explained in the Introduction, machine-readable data in these languages are not yet available to the NLP community. Thus, the release of this newly compiled baseline corpus and the baseline readability models trained on it may serve as an important stepping stone to encourage more development and research on these low-resource Philippine languages in the future. Taking your feedback into account, we will include a clear definition of the three levels and add a separate paragraph in the Limitations section discussing these points in the revised version of the paper.\n\n\n4. *\"What does Vocab stand for?\"* - Vocab (or vocabulary) is the number of unique terms per class in a corpus. We will include this definition in the revised version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The main strengths of the paper can be summarised as follows:\n- the task is interesting and timely;\n- the dataset presented is intended for a group of less-resourced languages that have not been treated before;\n- the paper introduces the notion of hierarchical cross-lingual modeling;\n- the results obtained support the findings of previous work applied to new languages with mutual intelligibility.\n", "reasons_to_reject": "The main weaknesses of the paper can be summarised as follows:\n- the research is limited to a L1, L2, and L3 levels (also reported in the limitations section);\n- the paper includes a long introduction to the task and background information, but L1, L2, and L3 levels are not explained;\n- the paper does not provide meaningful comparisons with previous work;\n- the results obtained are mentioned to support the findings of previous work - yet the \n", "questions_to_authors": "It is mentioned in the paper: The compiled data are distributed across the first three years of elementary education (L1, L2 and L3) (l 109 - 111). : how are levels assigned to the texts? What is the difference between i.e.,  L2 and L3 levels? What is the confidence for the manual distribution of texts to levels?\n\nThe work presented here is focused on L1, L2 and L3 levels. Isn't that a limitation to be clearly stated in the respective section?\n(it is implied now).\n\nTable 7: What does Vocab stand for?\n\n", "paper_topic_and_main_contributions": "The paper presents a dataset for the assessment of readability for languages that belong to the central Philippine subgroup. The languages covered have not been treated before. The paper briefly presents the features taken into account, the experimentation setting and baseline developed leading to promising results. The author(s) make use of traditional widely used linguistic features plus the previously introduced cross-lingual N-gram overlap feature and introduce the hierarchical cross-lingual modeling for ARA.\n\nThe main contributions of the paper can be defined as follows:\n- a new corpus for automatic readability assessment for low-resourced languages;\n- the baseline developed;\n- the results support previous work for other languages;\n- the dataset will be freely available upon acceptance.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The paper is generally well structured and written."}, "extra_text": {}}, {"review_id": "1dXEZvv0FJ", "reviewer": "Reviewer_KDgi", "summary": "", "questions": "A) Did the authors specifically choose Random Forest as a model due to small data size? What about other models?\nB) If the data is in Latin script, then to what extent does it capture the sound(phonemic) differences between the languages? This has a bearing on how effectively the typological information is really being used", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are extremely grateful for the reviewer’s feedback on our work and very much appreciate the positive comments regarding our contributions to the corpora building for readability assessment in Philippine languages. Please find our responses to the individual questions below. We hope that our responses address both the questions and the weaknesses identified, and we also hope that this may be reflected in the updated scores.\n\n1. *\"The numbers under L1 L2 and L3 are not explained - are these the number of articles/words/sentences?\"* - We apologize for not being clear enough with the details on the data presented in the tables. These values denote the number of documents categorized as belonging to each level.\n\n2. *\"The cross-lingual modeling approach showed promising results, but again, a little more in-depth characterization of these results would have helped. To what extent is the data size affecting the results?\"*  - We thank the reviewer for raising this point. We believe that the question on the data size effects deserves further investigation, however, we are limited in our ability to fully investigate this aspect: given that we are working with low-resource languages, we can only report the results obtained with the available amount of data. We believe that if more data were available for these languages, we would be able to show further improvements on the current results (e.g., although not directly comparable, the results obtained on such high-resource languages as English using similar feature-based systems are higher than what we show in our work). \n\n3. *\"Did the authors specifically choose Random Forest as a model due to the small data size? What about other models?\"* - The choice of the Random Forest model is motivated by the extensive previous work on the Philippine languages by Imperial and Ong (2019-2021), which explored other algorithms such as Logistic Regression, SVM, and Random Forest for readability assessment and showed that the Random Forest model achieved the best results and demonstrated higher robustness to misclassifications. Thus, we used this algorithm to train the baseline models in our work.\n\n4. *\"Is the data in Latin script?\"* - Yes, all major languages in the Philippines, including the four used in the study, use the Latin alphabet. \n\n5. *\"Can this affect feature extraction in terms of syllables for these languages?\"* - Our scripts for syllable pattern features are only applicable to languages using the Latin alphabet and will not be applicable to languages using a different writing system. We thank the reviewer for pointing this out, and we will specifically include a discussion on this in our Limitations section.\n\n6. *\"To what extent does it capture the sound(phonemic) differences between the languages?\"* - Currently, our study is only limited to text-based features and does not cover phonemic or sound-based patterns due to the limited availability of NLP tools to extract more advanced features. We thank the reviewer for suggesting this possible future research direction and will be adding this to the Limitations section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The work describes a resource for a  group of languages that are not well represented in NLP. It also situates the need for this work within the context of education in a multi-lingual country.", "reasons_to_reject": "As the authors are primarily releasing the Basaha Corpus as a resource, the paper contains very few details about the corpus itself. E.g. Table 1 in the corpus describes the 4 languages and their status as per Ethnologue. But the numbers under L1 L2 and L3 are not explained- are these the number of articles/words/sentences? Some clear indication of the size of the corpus would have been more insightful here. Further, what is the script used for these languages - is it Latin? Can this affect feature extraction in terms of syllables for these languages? These points are not made clear in the paper. \nThe paper extensively cites previous work on Tagalog and Cebuano. A brief summary of this previous work would have been helpful to evaluate against the contribution made by this paper. \nThe cross-lingual modeling approach showed promising results, but again a little more in depth characterization of these results would have helped. To what extent is the data size affecting the results? It's hard to say given some missing details in the paper.", "questions_to_authors": "A) Did the authors specifically choose Random Forest as a model due to small data size? What about other models?\nB) If the data is in Latin script, then to what extent does it capture the sound(phonemic) differences between the languages? This has a bearing on how effectively the typological information is really being used", "paper_topic_and_main_contributions": "This paper describes a new Readability assessment corpus for 4 Central Phillipine languages. These are low-resource languages.\nFurther, it demonstrates baseline results for ARA (automatic readability assessment) using typological information. The authors mention that code and data will be released upon publication using a Creative Commons License", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "The paper does cite some relevant references on Readability Assessment- however it does not summarize this past work in relation to the current contributions. Further, there are few references on ARA on low resource languages in general (apart from the Central Phillipine languages)"}}, {"review_id": "9yMSPJ30Qb", "reviewer": "Reviewer_PKp6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are extremely grateful for the reviewer’s feedback on our work and very much appreciate the positive comments regarding our contributions to the corpora building for readability assessment in Philippine languages. Please find our responses to the individual questions below. We hope that our responses address both the questions and the weaknesses identified, and we also hope that this may be reflected in the updated scores.\n\n1. *\"The main weakness of this paper is that it did not mention why it introduced a new method. The paper did not compare the proposed method with previous works or show their differences.\"* - Our paper is inspired by the previous work of Imperial and Kochmar (2023), which describes the state-of-the-art approach in cross-lingual ARA in the Philippine languages. As a short paper, it focuses on an extension of this approach, and our original contribution to this field is the exploration of the hierarchical relations between low-resource languages as is demonstrated in Figure 1–thus, we call our approach “hierarchy-based cross-lingual ARA”. Table 2 shows the direct comparison and advantages of using our approach for ARA in low-resource languages. We thank the reviewer for making this point, and we will revise our paper accordingly (specifically, the discussion in Section 2) to emphasize further the motivation and theoretical background of hierarchical cross-lingual ARA in application to low-resource languages – Philippine languages in this case.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The introduction of low-resource Philippine languages and education shows strong motivation for this research, and hence the development of the corpus is an excellent contribution to the academic area.", "reasons_to_reject": "The main weakness of this paper is that it did not mention why it introduced a new method. According to Lines 171-186, we can see that previous works have already used different languages to enhance the models. However, the paper did not compare the proposed method with previous works or show their differences.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposed a new dataset for readability assessment consisting of four low-resource Philippine languages. It trained random forest models on the different combinations of these languages and showed the importance of combining data from closely related languages.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Summary (from reviewer v7Sy): The paper presents a dataset for the assessment of readability for languages that belong to the central Philippine subgroup. The languages covered have not been treated before. The paper briefly presents the features taken into account, the experimentation setting and baseline developed leading to promising results. The author(s) make use of traditional widely used linguistic features plus the previously introduced cross-lingual N-gram overlap feature and introduce the hierarchical cross-lingual modeling for automatic readability assessment (ARA).

The reviews acknowledged the value of the resource that was developed for this paper, and its unique position in both the languages it covers and the domain (ARA). Many of the indigenous Philippine languages lack significant text resources altogether, let alone resources in ARA.

The discussion period was very productive. While the reviewers pointed out a number of missing or not understood statistics and other pieces of information regarding the corpus, there was also some confusion about why the specific modeling approach was used. The authors’ responses helped clear all of this up to the point where any revisions required to the paper represent missing or not fully explained information, but not methodological flaws or soundness issues that should lead to rejection. However, the authors must carefully address all questions raised in revisions if the paper is accepted.

It is worth noting that this is a paper that primarily provides a resource, and does so for a set of less-resourced languages. Papers of this type often receive medium or low enthusiasm scores, despite the significant positive impact that publishing them in popular venues can have.
Decision: Accept-Main