Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
Year: 2023

Abstract:
A challenge towards developing NLP systems for the world’s languages is understanding how they generalize to typological differences relevant for real-world applications. To this end, we propose M2C, a morphologicallyaware framework for behavioral testing of NLP models. We use M2C to generate tests that probe models’ behavior in light of specific linguistic features in 12 typologically diverse languages. We evaluate state-of-the-art language models on the generated tests. While models excel at most tests in English, we highlight generalization failures to specific typological characteristics such as temporal expressions in Swahili and compounding possessives in Finish. Our findings motivate the development of models that address these blind spots.1

heading: 1 Introduction
text: In natural language processing (NLP), there is a need to build systems that serve more of the world’s approximately 6,900 languages. As one measure of linguistic diversity, the World Atlas of Language Structures (WALS; Haspelmath et al., 2005) records 192 linguistic features along which languages differ. These range from the order of subject, object, and verb (Dryer, 2013) to the number of basic color categories (Kay and Maffi, 2013). Languages present in existing NLP datasets mostly lie in low-density regions of the space of possible typological features (Ponti et al., 2021). In other words, many linguistic features that are common across the world’s languages are not observed in languages that are the focus of NLP research.2
It is thus important to investigate to which linguistic features models can generalize and where they face challenges. However, existing datasets
1We make all code publicly available at https://github. com/google-research/multi-morph-checklist.
2For instance, while tone is present in around 80% of African languages (Adebara and Abdul-Mageed, 2022), few Indo-European languages can be considered tonal.
do not allow for a fine-grained cross-lingual evaluation and mainly permit comparisons on a language level (Hu et al., 2020). Prior studies focused on syntax and grammar through the lens of acceptability judgements (Ravfogel et al., 2018; Ahmad et al., 2019; Mueller et al., 2020; Papadimitriou et al., 2022). While these enable the evaluation of what a model deems ‘natural’ in a given language, it is often unclear how such biases relate to real-world applications of NLP technology.
We propose Multilingual Morphological Checklist (M2C) to enable the investigation of a broader set of cross-lingual differences in practical scenarios. Specifically, we create a morphologicallyaware behavioral testing framework (Ribeiro et al., 2020) that allows for the specification of tests in a diverse set of languages. Using this framework, we design tests that probe model’s behavior in light of specific capabilities and typological features in 12 typologically diverse languages. We focus on a question answering setting as it represents one of
7181
the most general and widely useful NLP applications (McCann et al., 2018) and enables zero-shot evaluation of models. We create tests that cover a diverse set of reasoning capabilities involving general linguistic features that are expressed differently across languages—negation, numerals, spatial and temporal expressions, and comparatives—as well as features unique to certain languages such as time in Swahili, measure words in Chinese, compounding possessives in Finnish, and motion verbs in Russian. We evaluate state-of-the-art language models on the generated tests in zero-shot and oneshot settings. Our findings shed light on generalization failures to specific typological features. For instance, all models struggle with time expressions in Swahili and measure words in Chinese. We show the workflow of using M2C, from template creation to model evaluation, in Figure 2.
Our contributions are: (1) We create a new morphologically-aware multilingual behavioral testing framework. (2) We highlight linguistic features that are challenging in different languages. (3) We design tests that probe model capabilities in light of practically relevant typological differences. (4) We evaluate state-of-the-art language models on the generated tests. (5) We shed light on the challenges posed by typological differences in multilingual scenarios.

heading: 2 Related Work
text: Perplexity Perplexity is a standard measure of evaluating language model performance, which has also been used in multilingual settings (Gerz et al., 2018). Besides being difficult to compare across segmentations, perplexity does not provide more fine-grained insights regarding model behavior (Meister and Cotterell, 2021). Acceptability evaluations compare perplexity between minimal pairs of grammatical and ungrammatical sentences (Linzen et al., 2016; Warstadt et al., 2020). Such evaluations have been extended to other languages (Ravfogel et al., 2018; Ahmad et al., 2019; Mueller et al., 2020; Xiang et al., 2021; Papadimitriou et al., 2022), which requires writing extensive languagespecific grammars while the relevance of syntax biases in real-world applications remains unclear.
Evaluation of large models Most benchmarks designed for evaluating large models focus on assessing their performance on a collection of complex tasks (Wang et al., 2019; Hu et al., 2020; Hendrycks et al., 2021; Gehrmann et al., 2021; Srivastava et al., 2022). However, such benchmarks are unable to highlight more fine-grained model limitations (Ethayarajh and Jurafsky, 2020) and are outpaced by the development of new models.
Behavioral testing Behavioral testing sheds light on model capabilities via the design of simple targeted tasks. Early work such as bAbI (Weston et al., 2016) focused on toy tasks requiring simple reasoning capabilities while oLMpics (Talmor et al., 2020) consisted of 8 short classification tasks for masked language models. Recently, LMentry (Efrat et al., 2022) provides simple tests assessing fundamental generation capabilities. A common test bed is natural language inference (Naik et al., 2018; McCoy et al., 2019) where analyses of reasoning types have been extended to other languages (K et al., 2021; Joshi et al., 2020; Hartmann et al., 2021) but require existing data.
The CheckList framework (Ribeiro et al., 2020) enables the generation of behavioral tests for NLP models but its templates are English-centric. English Checklist tests have been extended to other languages via translation (Ruder et al., 2021; K et al., 2022). Such approaches, however, struggle with comprehensively covering linguistic features specific to a language and are not able to easily represent morphological variation. Relatedly, Jiang et al. (2020) create templates that integrate morphology for simple knowledge retrieval queries while Kassner et al. (2021) automatically translate knowledge retrieval queries into other languages. Compared to their approach, our framework allows for integrating morphology into a broader range of tests and is more scalable and flexible.

heading: 3 CheckList
text: CheckList (Ribeiro et al., 2020) relies on templates to generate a large amount of samples in order to evaluate models’ behavior regarding different tasks and capabilities in a controlled manner. A template consists of a string with placeholders such as {first_name} delimited by curly brackets, e.g., “{first_name} is {adj}”. The user provides a set of values for each placeholder, for instance, {first_name} = {Michael, John, ... } and {adj} = {busy, friendly, ... }, which are used to populate the templates with their Cartesian product. The generated samples can then be applied to systematically test a model’s performance in a specific setting.
Multilingual tests CheckList has been designed for English and provides mainly English-specific functionality. For example, it matches indefinite articles with nouns based on their starting letter, i.e., the placeholder {a:job} generates “a lawyer” and “an engineer”. As a consequence, CheckList is not
capable of effectively generating tests in languages with richer morphology, which require maintaining agreement between multiple parts of the template— a feature that is beyond the scope of CheckList.
While multilingual tests can be generated by translating English tests (Ruder et al., 2021; K et al., 2022), optionally including template extraction and human verification, such generated templates struggle with handling rich morphology. In addition, in order to systematically probe linguistic features specific to a language, it is crucial to be able to efficiently generate in-language tests from scratch.

heading: 4 M2C Framework
text: We propose the M2C (Multilingual Morphological Checklist) framework in order to enable the generation of tests in a broad set of languages, including languages with rich morphology. A user provides a template as a string, a list of values for each placeholder, and an optional configuration dictionary in case of duplicate placeholders. The placeholder values can either be passed without inflections (for example, names in English) as a list of strings, or as a list of dictionaries with their corresponding inflected values. Each key of the dictionary is a feature combination (e.g., MASC.PL) and the value is the corresponding string (e.g. “apples”). As such, each entity can have multiple inflections, for instance, in English “apple” and “apples”. We show the general M2C workflow in Figure 2.
Morphological categories Our library follows the UniMorph Schema representation (SylakGlassman, 2016), which decomposes morphology into 23 dimensions and over 212 features. For example, Gender is one dimension, which contains features such as Feminine (FEM), Masculine (MASC), and Neuter (NEUT).
The ability to indicate these dimensions using a clear codification allows us to describe both the value attributes given to placeholders and their dependence on one another. As an example, in order to differentiate between “Juliette est grande” and “Julien est grand” in French, it is necessary to ensure gender agreement between noun and adjective by including the Gender attribute in the template. To cover such functionality, we introduce a syntax describing the morphological dependence between placeholders: {X.<Y.D>} signifies that X should have the same feature for dimension D as Y. In the above example, this is realized by “{first_name} est {adj.<first_name.GENDER>}”.
Language-specific dimensions While initially relying on the UniMorph schema, we found cases where the existing dimensions are not sufficient to describe morphology of placeholders within the templates, which is especially necessary for dealing with exceptions. For instance, the trifold article distinction in Italian masculine gender—il treno, l’hotel, lo studente—depends on whether the noun starts with a consonant, vowel or h, or a specific consonant combination3 respectively. In order to lexically encode such exceptions, we provide the ability to add dimensions, in this case STARTSWITH, which includes features VOW, CONS, and CONS2. While the goal of M2C is not to be exhaustive, it should enable encoding a sufficient number of dimensions to allow the user to write templates for diverse use cases.4
Advanced templating system To cover the variety of morphological phenomena, we designed a templating system with a rich syntax. When describing dependence rules, features can be added sequentially and are commutative, e.g., <first_name.GENDER.NUMBER> is equivalent to <first_name.NUMBER.GENDER> where NUMBER = {singular, plural}. Often, only two or three output values are necessary, which directly depend on a placeholder’s feature. We allow a simple expression to be passed directly in the template to make this rule explicit:
{val_1:placeholder.feature_1 | ... | val_n:placeholder.feature_n},
e.g., {is:first_name.SG|are:first_name.PL}, which produces “is” for a singular {first_name} and “are” for a plural one. Finally, we allow multiple placeholders with the same type, e.g., {first_name1} and {first_name2}, to be populated by values of a common type, i.e., first_name. In the case of multiple placeholders, we can provide a configuration for each placeholder type that specifies boolean repetition and order fields to, for instance, avoid having examples like “John and John” (repetition) or “John and Mary” and “Mary and John” (order).
Manual enumeration of features and their corresponding values is a barrier to scaling. To circumvent this, we integrate UnimorphInflect (Anastasopoulos and Neubig, 2019), which uses mod-
3gn, pn, ps, x, y, z, s followed by another consonant or i followed by a vowel.
4UniMorph defines a generic dimension ‘Language Specific features’ with attributes LGSPEC1, .., LGSPECN, which does not provide the clarity and flexibility of our setup.
els trained on Unimorph data using the Unimorph Schema to generate inflections in 55 languages. As Unimorph models are imperfect—test accuracies range from 90%+ in many languages to 23% in Arabic—we envision a workflow where inflections are generated at scale using UnimorphInflect and then manually inspected by annotators for correctness. We expect the increase in productivity, and thus reduction in cost, to be significant by leveraging semi-automated as opposed to manual generation for languages with good performance.5
Answer validation Most prior benchmarks for behavioral testing of language models have focused on classification tasks (Talmor et al., 2020; Ribeiro et al., 2020). As M2C aims to support the evaluation of generative models using arbitrary templates, we implement functionality to match a range of outputs for each template, based on morphology, string matching and regex.6
Summary Overall, the M2C framework enables the systematic and controlled generation of highquality tests at scale in a broad set of languages. As such, it occupies a middle ground between libraries such as SimpleNLG (Gatt and Reiter, 2009) that generate high-quality data but require encoding each language-specific rule, and template expansion via generative language models (Honovich et al., 2022), which are highly scalable but less reliable and underperform on languages with limited data (Hu et al., 2020). M2C enables modular design by allowing the addition of user-specified dimensions and features for specific templates and languages without requiring to encode all possible rules of a language. Furthermore, an advanced templating syntax and the semi-automatic generation of inflections may improve user productivity.

heading: 5 Capabilities and Typological Features
text: Languages We generate tests targeting capabilities and typological features in 12 typologically diverse languages: English (EN), Spanish (ES), Italian (IT), French (FR), German (DE), Swedish (SV), Finnish (FI), Slovak (SK), Russian (RU), Swahili (SW), Mandarin Chinese (ZH), and Arabic (AR).
Recent models have excelled at a wide range of tasks in English requiring a diverse set of reasoning
5In order to ensure high-quality tests for the experiments in §6, we manually enumerate all relevant inflections.
6For each of the templates in §6, we curate possible outputs and implement regex and functions capturing them.
and understanding capabilities (Wang et al., 2019; Hendrycks et al., 2021). As most languages are morphologically richer than English, they encode the linguistic features representing such capabilities in more complex ways. The features we investigate are relevant in a variety of real-world applications including sentiment analysis (Wiegand et al., 2010), question answering (Dua et al., 2019), grounding (Kordjamshidi et al., 2020), reasoning with temporal change (Lazaridou et al., 2021) and quantitative attributes (Elazar et al., 2019).
We investigate capabilities and linguistic features present in all our investigated languages as well as linguistic features unique to certain languages. For each feature, we highlight differences in its cross-lingual instantiation and challenges for natural language understanding and generation. We create templates using the M2C framework to test a model’s understanding of each capability and feature. We show a subset in Table 1.

heading: 5.1 Language-agnostic features
text: Negation In Indo-European languages, negation is often expressed via a separate particle such as not (English), inte (Swedish), etc. In contrast, in Swahili, for instance, negation morphemes are fused with the verb root and thus harder to identify. For other negation terms such as kein (German) models need to produce the correct agreement when generating text. In addition to gender and number agreement with the subject, Ara-
bic negation takes up to five forms in singular, three forms in dual, and five forms in plural, e.g., Ë (SG.MASC) and I Ë (SG.FEM).
Numerals Models must be able to recognize and reason with numbers in their spelled-out and numerical forms across different writing and numeral systems, e.g., seventeen (English) and 17 (Western Arabic numerals) and Qå « éªJ. and 17 (Eastern Arabic numerals). For generation in Russian and Slovak, models must inflect the noun depending on the quantity of the object. Slovak, for instance, has separate inflections for quantities of one, two/three/four, and five and more, which also vary based on the object’s animacy.
Spatial expressions In Russian, prepositions are associated with different cases, for example the instrumental case for за (behind) and the prepositional case for on. Such case agreement needs to be taken into account when generating text in Russian. Finnish, in addition to prepositions, follows a system of postpositions, which relate the location of one thing to another and require objects to be inflected in either partitive or genitive case.
Temporal expressions Some languages with rich morphology such as Finnish and Swahili encode temporal expressions in less complex ways than their inflection-sparser counterparts. In Swahili, verbal structure follows a simple compounding schema of subject marker + tense marker
+ verb, e.g. a-na-soma (he reads) or u-ta-soma (you will read).
Comparatives Commonly, comparatives are expressed by a suffix or using a quantifier, e.g., more/less. Spanish and French follow the latter approach by placing más/menos and plus/moins before the adjective with only a few standard exceptions. On the other hand, in Finnish, for example, the formation of comparatives follows a complex system of rules for compounding that includes categories depending on the endings of adjectives and a suffix mpi.

heading: 5.2 Language-specific features
text: Time in Swahili In many languages, the day is divided into two periods: a.m. and p.m., with the daily cycle starting at midnight (0:00) and running through noon (12:00). In Swahili, time is based on sunset and sunrise, defined to be 6 pm and 6 am respectively in standard time. For example, 11.30 am in standard time is 5.30 in the morning in Swahili time. Understanding different time systems is key not only for in-language reasoning but also for cross-lingual applications.
Possessives in Finnish Compounding in Finnish along with its system of 15 cases is one of the most challenging aspects of the language. One relevant feature are the possessive suffixes, which attach to the stem of nouns, e.g., koulu (school) becomes kouluni (my school) and koulumme (our school). Possession is expressed via a suffix -lla, which compounds with other suffixes, e.g., siskollani (my sister has), which must be correctly inflected by models in order to achieve the intended meaning.
Particles in Mandarin Chinese Another language specific-feature are measure words in Mandarin Chinese, which include over 150 cases and are used for different types of objects depending on their characteristics, e.g., “本” for books, “双” for pairs, or “辆” for vehicles.
Motion verbs in Russian In most Slavic languages, motion verbs are a challenging concept as they behave differently than other verb categories. While most verbs have two forms (imperfective and perfective), motion verbs have three forms: one perfective form and two imperfective forms. Of the imperfective forms, the definite form indicates unidirectional or current one-time motion while the indefinite form represents multi-directional or habitual motion.

heading: 6 Experiments
text: Experimental setting We evaluate models on the generated tests in a question answering setting as can be seen in Figure 2. Each test consists of a context, a question, and an answer that needs to be predicted by the model. For each template, we generate 2,000 test examples on which the model is evaluated. A model’s performance on a template is its accuracy of predicting a valid answer for a test averaged across all tests of the template.
We evaluate models in both zero-shot and oneshot settings for each capability and language. In the one-shot setting, a test randomly generated using the same template is used as the exemplar. This simplifies the task in two ways: i) it provides the model with a clear format for generating the answer and may enable the model to infer the answer’s relationship to the rest of the template. While we conduct one-shot experiments to show the impact of additional instructions, zero-shot evaluation is the only setting that fully tests the model’s understanding and generative capabilities independent of confounders such as the exemplar choice (Zhao et al., 2021), in line with prior work on behavioral testing (Ribeiro et al., 2020; Efrat et al., 2022). We provide an example of both settings in Table 2.
Models We evaluate five state-of-the-art pretrained language models of different sizes: an LMadapted version (Vu et al., 2022) of mT5-XXL (13B parameters; Xue et al., 2021); PaLM-S (8B parameters), PaLM-M (62B parameters), and PaLML (540B parameters; Chowdhery et al., 2022); and PaLM 2 (Google et al., 2023). All models have
been trained on large amounts of web text but have not been otherwise fine-tuned for instructionfollowing or few-shot learning.
Generation Predictions are generated using greedy decoding with a temperature of 0 and a maximum of 20 decoding steps.

heading: 7 Results

heading: 7.1 Performance across Languages
text: We show the average results across tests covering language-agnostic features across languages and models in Table 3. We present the detailed results across test types for mT5-XXL and PaLM 2 in Table 4 and for PaLM-S, PaLM-M, and PaLM-L in Appendix A. We show results on language-specific features for all models in Table 5.
M2C tests are challenging, particularly for smaller models and for certain languages.
mT5-XXL and PaLM-S achieve comparatively poor performance on average across languages. While performance is highest for English, across the other languages both models only pass at most 50% of tests—and less than a third for Slovak (SK), Swahili (SW), and Arabic (AR) for PaLM-S. These results highlight that the tests generated with M2C are challenging for the majority of state-of-the-art models and demonstrate that a clear gap between performance on English and performance in other languages remains for most models.
Competence with language-agnostic features emerges at scale. We observe a 20 point improvement in average performance from PaLM-S to PaLM-M to PaLM-L, highlighting that model robustness to linguistic features improves with scale. The strongest model, PaLM 2, reaches almost perfect performance on English and on the Indo-European languages. Compared to PaLM-L,
PaLM 2 achieves the largest improvements on Slovak, Russian, Swahili, and Arabic. On Finnish, Slovak, Chinese, and Swahili average performance of PaLm 2 is still below 90%, however, indicating that there is headroom left in terms of competence with regard to language-agnostic features for even the strongest current models.

heading: 7.2 Performance across Linguistic Features
text: Language-agnostic features The most challenging test types for mT5-XXL and PaLM 2 in Table 4 are numerals and comparatives. mT5 performs poorly on addition and only slightly better on subtraction while PaLM 2 achieves around 90% performance on most languages. On comparatives, both models have more difficulty in the conditional case. While PaLM 2 passes negation tests with almost perfect accuracy across different languages, mT5 displays reduced performance, particularly when the question is negated and for non-Indo-European languages. This highlights that robust reasoning with negation only emerges at scale. On spatial and temporal tests, mT5 achieves reasonable performance in most languages, while PaLM 2 achieves perfect performance in most cases and only underperforms in Swahili.
Language-specific features We show the results on the language-specific feature tests in Table 5. All models have acquired a reasonable ability to distinguish between different forms of motion verbs in Russian. Small and medium-sized models generally fail to reason with compounding possessives in Finnish and time expressions in Swahili while all models are unable to perfectly employ the correct measure words in Chinese, despite it being a high-resource language. Similarly, even PaLM 2 is unable to correctly reason with time expressions in Swahili. We show examples of errors in model predictions for each test type together with English glosses in Appendix B.

heading: 7.3 Evaluating Morphological Correctness
text: The generated tests focus on evaluating a model’s understanding capabilities with regard to specific capabilities and linguistic features. As the linguistic features are often expressed via morphology, we additionally calculate the fraction of errors due to morphology in the models’ output for the tests with morphological variation in the answer. This enables us to assess a model’s ability to generate morphologically correct forms. For instance, in
highlighted cells are in Appendix C.
Slovak, a model must generate the correct accents and suffixes, e.g., it is an error if the model predicts the Trináste (13th) instead of Trinást’ (13). We automatically identify and manually curate these errors for PaLM-L and report the proportion of morphology-related errors for a subset of tests and languages in Table 6. We show examples of errors in model predictions that are due to morphology in Appendix C.
For certain tests with morphological variation in the answer, a non-negligible fraction of errors are due to producing incorrect morphological forms. For negation in Slovak, around half of PaLM-L’s errors are due to morphology such as an incorrect use of diacritics or suffixes, highlighting a weakness of subword-based models. For numerical reasoning, models frequently produce incorrectly inflected numerals. Similarly, models generate outputs with an incorrect case or number for tests related to spatial and temporal expressions and comparatives.

heading: 7.4 One-shot Evaluation
text: We show one-shot results for all models in Appendix D. The one-shot setting generally improves results as it allows the model to infer the format of the answer and potentially its relationship to the rest of the template. Improvements are larger for smaller models, which benefit more from information about the template. Nevertheless, even in this setting models are unable to achieve perfect accuracy across all languages. Reasoning with numerals and comparatives are still challenging for most models while improvements on numerals are also relatively smaller than on other test types. Models struggle particularly in Swahili across different test types. Overall, these results demonstrate that even in one-shot settings, large language models are not able to systematically generalize to certain typological features in multilingual settings.

heading: 8 Conclusion
text: In this paper, we have introduced M2C, a multilingual morphological framework for targeted behavioral evaluation of language-specific capabilities. As world languages present different challenges, M2C aims to provide flexibility in defining a suitable templating system with its individual dimensions and features. We have conducted experiments on state-of-the-art large language models, highlighted typological features that models struggle with, and quantified errors occurring due to morphology. We hope M2C inspires further research focused on tackling typological and morphological challenges with large language models.

heading: Acknowledgements
text: We thank Jialu Liu, Jiaming Shen, and Jonas Pfeiffer for helpful feedback on a draft of this paper.
Broader Impact Statement
Accessibility Our new behavioral testing framework enables the generation of tests that incorporate morphology, which makes the systematic and fine-grained evaluation of NLP models more accessible across a diverse set of languages. For many such languages, it was previously not feasible to gain a fine-grained understanding of a model’s capabilities.
Risks Risks are limited and mainly relate to obtaining a biased view of a capability due to the use of limited templates.
Limitations The creation of templates still requires native speaker expertise and an understanding of a language’s grammar. Morphological inflection models are imperfect so morphological forms may need to be enumerated to ensure highquality tests. We leave model-in-the-loop template creation and improving morphological inflection models for future work. While we design representative templates with thousands of permutations for each capability, a larger set of templates and arguments may be necessary to ensure a comprehensive coverage.

heading: A Zero-shot Results
text: We show zero-shot results for PaLM-S, PaLM-M, and PaLM-L across different tests and languages in Table 7.

heading: B Examples of Errors on Language-specific Feature Tests
text: We show examples of errors on language-specific feature tests with PaLM-L together with English glosses in Table 8.

heading: C Examples of Morphological Errors
text: We show example errors in predictions of PaLM-L that are due to morphology in Table 9.

heading: D One-shot Results
text: We show one-shot results for all models in Table 10. We show summary statistics of the average relative change in performance of the one-shot setting compared to the zero-shot setting for each language and model in Table 11.
Language Test and prediction English gloss
Russian C: Иногда он ходит в университет. Редко он ездит в театр. Q: Что он делает иногда? A: Ходит в университет.
P: Идёт в университет.
C: Sometimes he goes (by foot) to the university. Rarely does he go (by transportation) to the theatre. Q: What does he do sometimes? A: Goes to the university (multiple times).
P: Going to the university (one time).
Finnish
C: Äitini antoi isoäidilleni mukin. Isäni antoi sedälleni kameran. Q: Kenellä on uusi muki? A: Isoäidilläni.
P: Isoäidilleni.
C: My mother gave my grandmother a mug. My father gave my uncle a camera. Q: Who has a new mug? A: My grandmother has.
P: To my grandmother.
Chinese C: 桌子旁边放着六样东西，都是狗。 Q: 多少狗在桌子旁边? A: 六只。
P: 六个。
C: Next to the table are six things, all are dogs. Q: How many dogs are next to the table? A: Six (measure word for animals).
P: Six (generic measure word).
Swahili C: Sadiki anakula saa nne usiku na anaendesha masaa matatu baadaye. Q: Anaendesha saa ngapi? A: Saa saba usiku
P: Saa moja usiku.
C: Sadiki eats at 10 PM and then drives three hours after. Q: What time does he run? A: At 1 AM.
P: At 7 PM.
Table 8: Examples of errors in PaLM-L predictions and English glosses for language-specific feature tests. Each example includes a context (C), question (Q), answer (A), and the model prediction (P). Tests probe motion verbs in Russian, possessives in Finnish, measure words in Chinese, and time expressions in Swahili.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: In the Broader Impact Statement.

heading: 3 A2. Did you discuss any potential risks of your work?
text: In the Broader Impact Statement.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4. A new behavioral testing library.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4.
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? The library will be released under a Creative Commons license.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We describe how the library should be used in Section 4 and Figure 1.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No created data contains personally identifying information.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Sections 4 and 5.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Yes, in Section 6.
C 3 Did you run computational experiments? Section 6.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 6.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 6.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Yes.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Yes.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? All templates were created by the authors of the paper.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We leave model-in-the-loop template creation and improving morphological inflection models for future work. While we design representative templates with thousands of permutations for each capability, a larger set of templates and arguments may be necessary to ensure a comprehensive coverage