Exploring Linguistic Probes for Morphological Inflection
Authors: Jordan Kodner
Salam Khalifa
Sarah Ruth Brogden Payne
Venue: EMNLP 2023 Main (2023)
Keywords: morphology
inflection
linguistic probes
English
Spanish
Swahili
TL;DR: We apply language-specific probes to evaluate morphological inflection model's abilities and styles of generalization

Abstract: Modern work on the cross-linguistic computational modeling of morphological inflection has typically employed language-independent data splitting algorithms. In this paper, we supplement that approach with language-specific probes designed to test aspects of morphological generalization. Testing these probes on three morphologically distinct languages, English, Spanish, and Swahili, we find evidence that three leading morphological inflection systems  employ distinct generalization strategies over conjugational classes and feature sets on both orthographic and phonologically transcribed inputs.

Exploring Linguistic Probes for Morphological Generalization
Jordan Kodner and Salam Khalifa* and Sarah Payne* Department of Linguistics & Institute for Advanced Computational Science Stony Brook University, Stony Brook, NY, USA [ lerttstrkkkttttt edu
Abstract
in more controlled fashion. We examine 13 probes three languages English, Spanish, and Swahili chosen for data availability and their dis- tinet morphological characteristics. In addition, we investigate the effect of presentation style on performance: that is, whether the choice between phonological transcription or orthography has a substantial effect outcomes. We report three systems which differ widely in their behavior. They often but not always make reasonable linguistic generalizations, even their incorrect predictions. In addition, we find no statistical effect for presen- tation style, even on English. This has implications for research that attempts to use neural networks for cognitive modeling and evaluates on orthographic data rather than more Nomain-approphe phono- logical transcriptions.
Modern work on the cross- lonsuuling compu- tational modeling of morphological inflection has typically employed language-independenn data splitting algorithms. In this paper, we sup- plement that approach with language-spest specific probes designed to test aspects of morpholog- ical generalization. Testing these probes on three morphologically distinct languages, En- glish, Spanish, and Swahili, we find evidence that three leading morphological inflection sys- tems employ distinct generalization strategies over conjugational classes and feature sets on both orthographic and phonologically tran- scribed inputs.
1 Introduction
The current practice in the evaluation of computa- tional morphological inflection models, such as that employed in the SIGMORPHON, CoNLL-
Languages
SIGMORPHON and SIGMORPHON-UniMory
Three languages were chosen whose inflectional morphologies range from entirely fusional (En- glish), to mixed (Spanish), to mostly agglutinative (Swahili). In highly agglutinative languages, indi- vidual features a set tend to correspond distinct morphological patterns, so a model may general- ize to unseen feature sets by mapping component features to their corresponding patterns. This is exemplified by the Swahili example (1), in which most features correspond to individual morphemes; only the person/number prefix maps to more than one feature. On the other hand, highly fusional languages map entire feature sets to single patterns, This is shown by the Spanish example (2), in which all features map together onto a single suffix.
shared tasks (Cotterell al., 2016, 2017, 2018; McCarthy et al., 2019; Vylomova al., 2020; Pi- mentel al., 2021; Kodner et al., 2022) as well as in more targeted studies focused on specific lan- guages or the generalization behavior of computa- tional models (Goldman et al., 2022; Wiemerslage et al., 2022; Kodner et al., 2023b; Guriel et al., 2023; Kodner et al., 2023a), is to train on (lemma, inflection. features) triples and predict in- flected forms from held-out (lemma, features) pairs. The algorithm for generating train-test splits is both random and inguage-independent which has proven successful distinguishing morpho- logical inflection models at the gross quantitative level. Models differ in their performance across languages and in their ability to generalize across Jemmas or feature sets.
(1) Swahili ulipika "you (singular) cooked'
li- pik-
In this paper, both replicate this type of anal- ysis and contrast it with new language-specific probes for testing models' generalization abilities
2.sG- PST- cook-
Spanish cocinaste "you (singular) cooked'
cocina- ste cnok- PST.ID
* Denotes equal contribution

---

All data was adapted from Kodner et al. (2023b), ! which was in lurn extracted from Uni- Morph 3 and (McCarthy et al., 2020; Batsuren et al., 2022). The data was subjected to additional processing as described below. For each language, only verbs were extracted, and muldi-word expres- sions were excluded. Importantly morpheme segmentation is provided in the UniMorph data unlike illustrative examples, so agglutinativity must be discovered by each system.
each split was performed five times with unique random seeds, and each seed was used to produce parallel orthographic and transcription versions of the splits for evaluating presentation style, yield- ing ten samples in total. Data sets contained 1600 training items + 400 for fine-tuning. BLIND splits contained 1000 test items.
BLIND: Following Kodner et al. (2023b), this splitting strategy ensures that approximately 50% of test items contain OOV feature sets but is other- wise blind to the identity of those feature sets. This was shown in the 2022 SIGMORPHON-UnMMopp) shared task to create more opportunities for testing dimensions of generalization across feature sets than more traditional uniform random sampling (Kodner et al., 2022).
English (Germanie): English triples were tran- scribed using an IPA translation the CMU Pro- nouncing Dictionary. Triples without available transliterations were discarded. When the dictio- nary provided multiple transcriptions, the lemma and inflection transcriptions which minimized Lev- enshtein distance between them were chosen.
English PROBE Splits
Spanish (Romance): Spanish triples were tran- scribed using the Epitran package (Mortensen et al., 2018), which does not include stress information. UniMorph treats the infinitive as the lemma and includes the 2nd person plural in its paradigms. Swahili (Bantu): Swahili triples were tran- scribed using Epitran. UniMorph treats the bare stem as the lemma. The Swahili data set is much smaller than the others and does not contain any forms with negative marking. We found many in- consistencies in the UniMorph 4 feature tags which we normalized. Most importantly, PFV and PRF, which both indicate perfect aspect, were mapped to PFV and tag order was made consistent. Triples in swc. sm with tags which could not be clearly mapped to tag sets in swc were excluded.
eN-NFIN: This probe is designed to test what the system does when it knows nothing about the rele- vant tag. The NFIN tag, which maps only uppears the infinitive and non-3rt 3rd singular present. Triples with the NFIN tag were excluded from training and presented during test. No system can know what inflection the NFIN corresponds to, so it can only succeed if it defaults to the lemma. en-PRS: This probe tests the implications of UniMorph's design choice of annotating the non- 3rd singular present with NFIN rather than PRS, We used identical splits to en-NFIN IN, but replaced the NFIN tag with PRS. Since this is shared with the present 3rd singular (PRS; SG) and participle (V,PTCP; a system should be reasonably ex- pected to generalize either the -s or -ing endings to PRS items in test.
change,
3 Data Splits
We created several random data splits to study di- mensions of morphological generalization. As in prior work, BI.IND splits were made without re- gard to specific linguistic properties of the triples. PROBE splits were sensitive to the properties of the triples: they were divided into relevant and ir- relevant sets according to properties of the feature sets or lemmas. The irrelevant triples were split in BLIND in order to pad training to the same length as BLIND, but irrelevant test triples were discarded. The relevant triples were split way specific to each probe, controlling which occurred in muin--iinetuning For both BLIND and PROBE,
en-PRS3SG: This probe also replaces NFIN with PRS but instead withholds 3 SG from training and presents during test. Success on this probe is impossible, since systems cannot learn to map this tag set to -s during training, but a system should generalize either the bare lemma or the -ing of the present participle.
Spanish PROBE Splits
eS-FUT: This probe tests the ability of systems to learn basic agglutinative suffixing pattern: the fu- ture tense FUT) is formed by suffixing regular person/number marking onto the infinitive (correr 'to run,' correr-ás 'you will run'). There are six possible person/number combinations attested in the future indicative paradigm. For each split/seed, two feature sets with IND; FUT were randomly cho-
commjkoonerr0///AL2022
RealityCheck. LingProbes
http:///luub.coo///ooeen//////// with summaries provided the Appendix. http:///tttuu..ommmmcii...k.....

---

sen, and then triples with these features were ran- domly sampled appear in training for models to learn the pattern. All other feature sets containing FUT were withheld from training, and triples containing them were sampled to appear in test.* es-AGGL: The conditional (COND) and imperfect PIN:; IPFV) are also agglutinative. Two each of conditional, imperfect, and future feature sets were independently selected to appear in the train- ing split, and the rest were withheld for evaluation. This is a more challenging version of the es-FUT probe because a system has to learn three aggluti- native patterns at once.
uation. This test similar es-FUT and es-AGGL. sw-NON3: Swahili manifests four non-3rd person subject marking prefixes as well as 3rd person prefix for each of its many noun classes. This is more challenging version of sw-1PL which with- holds all but one independently selected feature set containing each of 1; 1 1; 2; and P; from training and evaluates on the rest.
Sw-FUT: Tense is marked with an affix imme- diately following subject marking. The future is marked with -ta- This probe is set up like sw 1P-1PL except it requires system to produce string infix rather than string prefix.
eS-PSTPFV: This probe tests what system does when it is forced to predict missing parts of fusional paradigm. As shown in (2), preterite PST person/number forms are fusional, because they manifest distinctly from the mark- ing for all other tense/aspects (except for the 1st person plural) and are not decomposable into preterite part and person/number part. Since system cannot predict these forms from their com- ponent person/number features matched with other tenses, should perform poorly even it succeeds at es-FUT and es-AGGL. A system should general- ize person/number marking from the other tenses. eS-IR: This prohe tests generalization across conjugational classes rather than across feature sets. Spanish verbs fall into three classes clearly indi- cated by their infinitive suffix: -ar, -er, and -ir. -ir shares many, but not all, of its inflections with -er. For this probe, all but 50 randomly chosen -ir lem- mas are banned from train sampling, which results in 10-18 -ir training triples sampled per seed. A system should predict the appropriate -er form or one with -i- for each -ir evaluation item or overap- ply the majority
Sw-PST: The simple past (PST) is marked with -li-. This probe similar to sw-FUT but with a distractor: the past perfective (PST;PFV) is actually fusional, and is expressed as -me- without -li-. A system could thus produce -me- forms instead of the expected -li-.
SW-PSTPFV: This probe is similar sw-PST ex- cept tests the past perfective (PST; while the simple past (PST) serves a distractor.
Systems
CHR-TRM (Wu al., 2021) is character-level transformer that was used baseline in the 2021 and 2022 SIGMORPHON shared tasks. We use default hyper-parameters parameters for low-resource scttings. CLUZH (Wehrli et al., 2022) is a character-level transducer which performs well but showed some weakness in feature set generalization in the 2022 shared task. We used beam decoding, size = 4.
ENC-DEC (Kirov and Cotterell, 2018) is an ASTM-based encoder-decoder which was argued to provide evidence for the cognitive plausibility of connectionist models as follow-up to the Past Tense Debate.
ending.
eS-IRAR: This probe is similar to es-IR but much more challenging because all -ar and -ir verbs are withheld from training. We predict that a system should produce the -er inflected form or replace -e- in forms with the appropriate -a- or -i-.
Results
5.1 Orthography and Transcription
This section analyzes the effect of presentation style on performance. In addition to visual inspec- tion of Figure 1, which shows little difference be- tween orthography and phonological transcription. there are at most moderate differences in mean ac- curacy between the two. Differences range from +4.07 points in favor of orthography for English, -2.80 for Spanish, to only -0.45 for Swahili.
Swahili PROBE Splits
sw-1PL: As (1), person/number and tense/aspect marking is marked agglutinatively before the stem in Swahili. The 1st person plural (1 (PL) is marked with a tu- prefix. Two feature sets containing PL were randomly selected by seed and allowed to appear in train, and the rest were withheld for eval-
English may favor orthography because re- moves the three-way allomorphy of past -fe)d and
"Illustrations for two seods are provided in the Appendix.

---

BL IND
en-PRS
e8-AGGL
e6-IR
eB-PSTPPV
Sw-FOT
SW-PST
e0-MP IN
en-PRS3SG
es-FUT
es-IRAR
Sw-1PP 1PL
Sw-NON3
SW-PSTPEV
English
Spanish
Swahili
Ortlograplyy Fraisription
75
50
69 Accuracy
25
0-
0.0
100
25
CHR-TRM
CLUZH
ENC.DEC
CHR-TRM
CLUZH
ENC-DEC
CHR-TRM
CLUZH
ENC-DEC
Figure 1: Accuracy on each split (color) for each seed by language (major column), system (minor column/shape), and presentation style (row).
3rd singular present -(e)s which is indicated in tran- scription. However, is also possible that the differ- ence is due the choice of transcription dictionary or how we processed it. For Spanish, a transcrip- tion scheme that retained stress may have proven more challenging than Epitran's which lacks it.
System CHR-TRM
BoOOV
ImOOV
fSOO
bothOOV
94.40%
82.68%
52.90%
36.46%
CLUZH
95,43
47.12
48.93
ENC-DEC 93.79
86.01
1.43
Table 2: Average performance on BLIND orthography across seeds and languages. no00V = triples where both lemma and feat. set observed; 1m00v lemma is OOV; fs00V = feature set is OOV. bothhov = lemma and feature set are OOV.
Variable system seed
F-statistic
p-value
68,093
<2e-16
0.223
0.925
presentation
0.014
0.906
For CHR-TRM and CLUZH, we observe much smaller performance gap OOV feature sets for the more agglutinative languages (Swahili -9.5 points Spanish -49.76 English -81,38), which indicates that these systems can perform some de- gree of generalization across feature sets. This contradicts the prior work, which uncovered no substantial difference between fusional and aggluti- native languages, indicating an inability to perform this kind of generalization.
language
76,588
<2c-16
presentation lang
1.061
0.351
Table I: ANOVA analysis on BLAND showing signifi- cant effect for system and language but not presentation.
We follow this with an ANOVA analysis of five variables: the system, seed, presentation style, lan- guage, and the interaction between presentation and language, determine which differences in mean accuracy are unlikely to be due chance. Summarized Table we find that only the choice system and the language are significant, not pre- sentation style. This is consistent with visual in- spection. The conclusion is the same on BLIND, PROBE, and both combined.
This discrepancy may be explained by how we processed the Swahili data. The published Uni- Morph data contains several inconsistencies, sum- marized in Section 2, in its feature tags which made the task of generalization unfairly challenging in prior work. We corrected this by normalizing the feature sets and removing triples which we could not adequately fix. However, this would only ex- plain the middling reported performance on other agglutinative languages if their UniMorph data sets turn out to be highly inconsistent well. This question is left for future work.
5.2 Generalization and Linguistic Analysis
Feature Set Generalization in BLIND
A breakdown of BLIND test triple by type in Ta- ble 2 replicates prior work (Kodner al., 2022, 2023b) demonstrating that generalization unseen feature sets is particularly challenging. All sys- tems showed lower accuracy on OOV feature sets (fs00v bothOov) than on other triples. ENC- DEC shows virtually no ability to do 5
Generalization on Swahili PROBE Splits
CHR-TRM very successful at generalizing the pre- fix sw-1PL and string infix in Sw-FUT. CLUZH sometimes applies the 1st singular prefix instead
'A complete breakdown is provided in the Appendix.

---

of the plural in sw-1PL, but is less consistent for Sw-FUT, where produces many infixes belong- ing to other tenses in addition to nonsense outputs. sw-NON3 was developed as a more challenging ver- sion of sw-1PL, and performance was indeed lower. However, instead of "near-miss" errors suggesting linguistic generalization, both systems produced range of person/number, tense, and nonsense errors not clearly related to the probe.
perform in (Kirov and Cotterell, 2018) rewarded analogy across lemma forms rather than unseen fea- ture sets. This is similar to the es-IR and es-IRAR probes but distinct from all the other probes which were focused on feature generalization.
Generalization on English PROBE Splits
Accuracy was near-zero on every English PROBE. This was expected, because the tasks were effee- tively impossible. however, qualitative analysis is insightful. For en-NFIN and en system would succeed if it defaulted to the bare lemma as an inflection of last resort. However, no sys- tem took that approach. For en-NFIN, CHR-TRM consistently outputted -ing forms with other errors, CLUZH produced -(e)d forms, and ENC-DEC pro- duced mostly -(e)s forms with other errors.
Sw-PST was designed to be similar to sw-FUT but with distractor fusional string infix. As expected, performance was lower, especially for CLUZH, which often substituted the distractor past perfect, other tense marking, or omitted tense al- together similar its eπrоrs on Sw-FUT. Its errors SW-PSTPFV were overwhelmingly application of the simple past, which can be explained as a gener- alization of PST while ignoring the PFV tag. These are mostly reasonable errors that point to some de- gree of generalization. ENC-DEC showed no ability to generalize according to component morphologi- cal features even this highly agglutinative setting.
The replacement UniMorph's NFIN with PRS did indeed have an effect, indicating that some behavior on English can be attributed to this corpus design choice: Both CHR-TRM and CLUZH now consistently produced present participle -ing forms, while RNC-DEC instead produced -(e)d or -(e)s. This indicates generalization over the PRS feature. For en-PRS3SG, CLUZH always produced the bare lemma form, showing clear generalization of PRS, CHR-TRM only produced nonsense errors, while ENC-DEC produced nonsense errors suffixed with -())d or -ing.
Generalization on Spanish PROBE Splits
For es-FUT, CHR-TRM and CLUZH identified the correct pattern of infinitive + person/number mark- ing but often produced slightly incorrect forms. the orthographic tests, many these errors in- volved incorrect stress marking. These could be considered reasonable "near-misses" that point to generalization. However, for the more challenging eS-AGGL, both produced less interpretable errors. es-PSTPFV was designed to be impossible but in- sightful, since preterite person/number marking cannot be predicted from the person/number mark- ing of other tenses. Indeed, CLUZZ generalized the person/number marking from the other tenses as well as the few preterite person/number mark- ings presented during training, indicating that it can employ the relevant generalization. However, CHR-TRM and ENC-DEC were less interpretable.
Conclusions
In this work, we present language-specific probes to evaluate the ability of computational systems to perform morphological generalizations as a complement to prior work relying on large-scale language-independen data-splitting. Systems dif- fer substantially both in their ability perform generalizations and in their problem-solving strate- gies. Of the two systems showing generalization ability, both perform well on the simplest probes but struggle on more complex but feasible probes. On probes designed to be impossible but insight- ful, both systems show some degree of reasonable generalization. In addition to this, we find no sig- nificant effect of presentation style on the behavior of the three systems. While we maintain that the choice of presentation style should be driven by application when possible (usually orthography for NLP or transcription for cognitive modeling), our results suggest that it can be reasonable to use or- thography when transcription impractical.
es-IR and es eS-RRAR differ from the other probes in that they require generalization across conjuga- tional classes indicated by the form of the lemma rather than generalization across feature sets. This proved appropriately challenging for all systems, which all produced many nonsense errors and some near-miss overuse of -er endings combined -ater and -iter endings. Notably, these were also the only probes for which ENC-DEC showed some success. The tasks that the system was asked to

---

Limitations
Krizhanovskaya, Elizabeth Salesky, Clara Vania, Sar- dana Ivanova, Jennifer White, Rowan Hall Maud- slay, Josef Valvoda, Ran 7migrod, Paula Czarnowska, Irene Nikkarinen, Aclita Salchak, Brijesh Bhatt, Christopher Straughn. Zoey Liu, Jonathan North Washington, Yuval Pinter, Duygu Ataman, Marcin Wolinski, Totok Suhardijanto, Anna Yablonskaya, Niklas Stoehr, Hossep Dolatian, Zahroh Nuriah, Shyam Ratan, Francis M. Tyers, Edoardo M. Ponti, Grant Aiton, Aryaman Arora, Richard Hatcher. Ritesh Kumar, Jeremiah Young, Daria Rodionova, Anastasia Yemelina, Taras Andrushko, Igor Marchenko, Polina Mashkovtseva, Alexandra Serova, Emily Prud hommeaux, Maria Nepomni- ashchaya, Fausto Giunchiglia, Eleanor Chodroff, Mans Hulden, Miikka Siliverberg, Arya D. Mc- Carthy, David Yarowsky, Ryan Cotterell, Reut Tsar- faty, and Ekaterina Vylomova. 2022. UniMorph 4.0: Universal Morphology. In Proceedings of the Thir- teenth Language Resources and Evaluation Confer- ence, pages 840-855, Marseille, France. European Language Resources Association.
The main limitation of the the trgguage-specifii splitting approach compared to the traditional language-independent splitting approach is the language-specific and domain-specifie expertise needed to design the probes themselves. Never- theless, this work successfully demonstrates the utility of such probes. Future work with a larger group of experts in a wider range of languages could extend this approach to more test cases. We believe that this could be implemented on a larger scale such as for a complementary shared task in the future.
Ethics Statement
To the best of our knowledge, all results published in this paper are accurate. All data sources are free, publicly available, and cited in the article. No sensitive data was used which could violate individuals' privacy or confidentiality. Authorship and acknowledgements fairly reflect contributions.
Ryan Cotterell, Christo Kirov, John Sylak-Glassmen Géraldine Walther, Ekaterina Vylomova, Arya D. Mc- Carthy, Katharina Kann, Sabrina J. Mielke, Garrett Nicolai, Miikka Silfverberg, David Yarowsky, Ja- son Eisner, and Mans Hulden. 2018. The CONL.I- SIGMORPHON 2018 shared task: Universal mor- phological reinflection. In Proceedings of the CuNLL-SIGMORPHER 2018 Shared Task: Univer- sal Morphological Reinflection, pages Brussels. Association for Computational Linguisties.
Acknowledgements
We thank Zoey Liu for motivating discussion. Ex- periments were performed on the SeaWulf HPC cluster maintained by RCC and the Institute for Advanced Computational Science (IACS) at Stony Brook University and made possible by National Science Foundation (NSF) grant No. 1531492. The third author gratefully acknowledges funding through the IACS Graduate Research Fellowship and the NSF Graduate Research Fellowship Pro- gram under NSF Grant No. 2234683.
Ryan Cotterell, Christo Kirov, John SylakGGlassman, Géraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sandra Kübler, David Yarowsky. Jason Eisner, and Mans Hulden. 2017. CoNLL- SIGMORPHON 2017 shared task: Universal mor- phological reinflection in 52 languages. In Proceed- ings of the SIGMOOPHON 2017 Shared Task: Universal Morphological Reinflection, pages 1-30, Vancouver. Association for Computational Linguis- tics.
References
Ryan Cotterell, Christo Kirov, John Sylak-Glassman David Yarowsky, Jason Eisner, and Mans Hulden. 2016. The SIGMORPHON 2016 shared Task- Morphological reinflection. In Proceedings of the 14th SIGMORPHION Workshop on Computational Research in Phonetics, Phonology, and Morphology. pages 10-22, Berlin, Germany. Association for Com- putational Linguistics.
Khuyagbaatar Batsuren, Omer Goldman, Salam Khal- ifa, Nizar Habash, Witold Kieraš, Gábor Bella, Brian Leonard, Garrett Nicolai, Kyle Gorman, Yusti- Ghanggo Ate, Maria Ryskina. Sabrina Mielke, Elena Budianskaya, Charbel El- El-Khaissi, Tiago Pi- mentel, Michael Gasser, William Abbott Lane, Mohit Raj. Matt Coler, Jaime Rafael Montoya Samame, Delio Siticonatzi Camaiteri, Esaú Zu- maeta Rojas, Didier López. Francis, Arturo Once- vay, Juan López Bautista, Gema Celeste Silva Vil- legas, Lucas Torroba Hennigen, Adam Ek. David Guriel, Peter Dirix, Jean-Philippe Bernardy, An- drey Scherbakov, Aziyana Bayyr-ool, Antonios Anastasopoulos. Roberto Zariquiey, Karina Sheifer. Sofya Ganieva, Hilaria Cruz, Ritván Karahóga, Stella Markantonatou, George Pavlidis, Matvey Plu- garyov, Elena Klyachko, Ali Salehi, Candy An- gulo, Jatayu Baxi, Andrew Krizhanovsky, Natalia
Omer Goldman, David Guriel, and Reut Tsarfaty. 2022. (un)solving morphological inflection: Lemma over- lap artificially inflates models' performance. In Pro- ceedings the 60th Annual Meeting the Associa- tion for Computational Linguistics (Volume 2: Short Papers), pages 864-870, Dublin, Ireland. Association for Computational Linguistics.
David Guriel, Omer Goldman, and Reut Tsarfaty. 2023. Morphological inflection with phonological features. In Proceedings of the 61 Annual Meeting of the

---

Association for Computational Linguisties (Volume 2: Short Papers), pages 013-622, Toronto, Canada. Association for Computational Linguistics.
Florence, Italy, Association for Computational Lin- guistics.
David R. Mortensen, Siddharth Dalmia, and Patrick Littell. 2018. Epitran: Precision G2P for many lan- guages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Christo Kirov and Ryan Cotterell. 2018. Recurrent neu- ral networks in linguistic theory; Revisiting pinker and prince (1988) and the past tense debate. Transac- fions of the Association for Computational Linguis- tics, 6:651 665.
Jordan Kodner, Salam Khalifa, Khuyagbaatar Bat- suren, Hossep Dolatian, Ryan Cotterell, Faruk Akkus, Antonios Anastasopoulos, Taras Andrushko, Arya- man Arora, Nona Atanalov, Gábor Bella, Elena Budianskaya, Yustimus Ghanggo Ate, Omer Gold- man, David Guriel, Simon Guriel, Silvia Guriel- Agiashvili. Witold Kieraš, Andrew Krizhanovsky. Natalia Krizhanovsky, Igor Marchenko, Magdalena Markowska, Polina Mashkovtseva, Maria Nepomni- ashchaya, Daria Rodionova, Karina Scheifer, Alexan- dra Sorova, Anastasia Yemelina, Jeremiah Young. and Ekaterina Vylomova. 2022. SIGMORPHON- UniMorph 2022 shared task O: Generalization and typologically diverse morphological inflection. In Proceedings of the 19th SiGMORPHION Workshop on Computational Research Phonetics, Phonology, and Morphology, pages 176-203, Seattle, Washing- ton. Association for Computational Linguistics.
Tiago Pimentel, Maria Ryskina, Sabrina J. Mielke, Shijie Wu, Eleanor Chodroff, Brian Leonard, Gar- rett Nicolai, Yustimus Ghanggo Ate. Salam Khalifa, Nizar Habash, Charbel El- El-Khaissi. Omer Goldman, Michael Gasser, William Lane, Matt Coler. Arturo Oncevay, Jaime Rafael Montoya Samame, Gema Ce- leste Silva Villegas, Adam Ek, Jean-Philippe Bernardy, Andrey Shcherbakov, Aziyana Bayyr-ool, Karina Sheifer, Sofya Ganieva, Matvey Plugaryov, Elena Klyachko, Ali Salehi, Andrew Krizhanovsky, Natalia Krizhanovsky, Clara Vania, Sardana Ivanova, Aclita Salchak, Christopher Straughn, Zoey Liu, Jonathan North Washington, Duygu Ataman, Witold Kieraš, Marcin Woliński, Totok Suhardijanto, Niklas Stochr, Zahroh Nuriah, Shyam Ratan, Francis M. Tyers, Edoardo M. Ponti, Grant Aiton, Richard J. Hatcher, Emily Prudeaomm Ritesh Kumar, Mans Hulden, Botond Barta, Dorina Lakatos, Gá- bor Szolnok, Judit Ács, Mohit Raj. David Yarowsky, Ryan Cotterell, Ben Ambridge, and Ekaterina Vy- lomova. 2021. SIGMORPHON 2021 shared task reinflecctir Generalization across morphological reinflection: languages. In Proceedings of the 18th SIGMOR- PIONN Workshop on Computational Research in Pho- netics, Phonology, and Morphology, pages 229-259, Online, Association for Computational Linguistics.
Jordan Kodner, Salam Khalifa, Sarah Payne. and Zoey Liu. 2023a. Re-Evaluating the Evaluation of Neural Morphological Inflection Models. In Proceedings of the 45th Annual Meeting of the Cognitive Science So- ciery (CogSci), volume 45. pages 3259-3267, Sydney, NSW, Australia. Cognitive Science Society.
Jordan Kodner, Sarah Payne, Salam Khalifa, and Zoey Liu. 2023b. Morphological inflection: A reality check. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers). pages 6082-6108 1182-6101,11 Toronto, Canada. Association for Computational Linguistics.
Ekaterina Vylomova. Jennifer White, Elizabeth Salesky, Sabrina J. Mielke, Shijie Wu, Edoardo Maria Ponti, Rowan Hall Maudslay, Ran Zmigrod, Josef Valvoda, Svetlana Toldova, Francis Tyers, Elena Klyachko, Ilya Yegorov, Natalia Krizhanovsky. Paula Czarnowska, Irene Nikkarinen, Andrew Krizhanovsky. Tiago Pimentel, Lucas Torroba Henni- gen, Christo Kirov, Garrett Nicolai, Adina Williams. Antonios Anastasopoulos, Hilaria Cruz, Eleanor Chodroff, Ryan Cotterell, Miikka Silfverberg, and Mans Hulden. 2020, SIGMORPHON 2020 shared task O: Typologically diverse morphological inflec- tion. In Proceedings of the 17th SIGMORPHION Workshop on Computational Research in Phonetics, Phonology. and Morphology, pages 1-39, Online. Association for Computational Linguistics.
Arya D. McCarthy, Christo Kirov, Matteo Grella, Amrit Nidhi, Patrick Xia, Kyle Gorman, Ekate- rina Vylomova, Sabrina J. Mielke, Garrett Nico- lai, Miikka Silfverberg, Timofey Arkhangelskiy, Na- taly Krizhanovsky, Andrew Krizhanovsky, Elena Klyachko. Alexey Sorokin, John Mansfield, Valts Emstreits, Yuval Pinter, Cassandra L. Jacobs, Ryan Cotterell, Mans Hulden, and David Yarowsky. 2020. UniMorph 3.0: Universal Morphology. In Proceed- ings of the Twelfth Language Resources and Evalua- tion Conference, pages 3922-3931, Marseille, France. European Language Resources Association.
Silvan Wehrli, Simon Clematide, and Peter Makarov. 2022. CLUZH at SIGMORPHON 2022 shared tasks on morpheme segmentation and inflection generation. In Proceedings of the 19th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology. and Morphology, pages 212-219. Seattle, Washing- ton. Association for Computational Linguistics.
Arya D. McCarthy, Ekaterina Vylomova, Shijie Wu, Chaitanya Malaviya, Lawrence Wolf-Sonkin, Garrett Nicolai, Christo Kirov, Miikka Silfverberg, Sabrina J. Mielke, Jeffrey Heinz, Ryan Cotterell, and Mans Hulden. 2019. The SIGMORPHON 2019 shared task: Morphological analysis in context and cross- lingual transfer for inflection. In Proceedings of the 16th Workshop on Computational Research in Pho- netics Phonology, and Morphology, pages 229-244,
Adam Wiemerslage. Shiran Dudy, and Katharina Kann. 2022. A comprehensive comparison of neural net- works as cognitive models of inflection. In Proceed-

---

ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1933-1945. Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Ortho
BoOOv
lm00v
600V
bothOOV
Total
CHR-TRM
99.16%
92.75%
75%
50,00%
95. 95.92%
(1.40)
(3.40)
(50,00)
(100)
(1,40)
CLUZE
98.48%
84.14%
100%
1003
91.32%
(0.99)
(6.35)
(0.00)
(0.00)
(2,50)
ENC-DEC
8050
1.00%
75.00%
0.00%
8558
(2:20)
(3.20)
(50.00)
(0.00)
(2.30)
Shijie Wu, Ryan Cotterell, and Mans Hulden. 2021. Ap- plying the transformer to character-level transduction. In Proceedings of the 16th Conference of the Euro- pean Chapter of the Association for Computational Linguistics: Main Volume, pages 1901-1907, Online. Association for Computational Linguistics.
Transer
BoOOV
lm00v
&OOV
bothOOy
Total
CHR-TRM TRM
99.04%
91.71%
50,00%
50.00%
95.32%
(1,00)
(1.22)
(700)
(100)
(1 (0)
CLUZH
98.56%
88.55%
100%
100%
93.56%
(1,20)
(6.78)
(0,00)
(1.29)
(1.70)
ENC-DEC
98.76%
0.48%
75 .000
0.00%
60%
(1.80)
(1.60)
(50,00)
(0.00)
(0.60)
Table 6: Average percent accuracy and (accuracy range) on BLIND Swahili across seeds for each system and presentation style. no00V triples where both lemma and feature set were observed; 1m00v = lemma is OOV; fs00v = feature set is OOV. both0ov = lemma and feature are 00V.
A Appendix
Lemmas
# Feature Sets
# Tripples 27836
English 9188
Spanish 7326
152
1077655
Swahili 131
169
10925
Ortho
en-NFIN
en-PRS
en en-PRS356
Table 3: Type frequencies for lemmas, feature sets, and triples for each language data set before splitting.
CHR-TRM
0.06%
0.00%
(0.29)
(0.60)
(0.00)
CLUZH
1.11%
0.00%
(1.24)
(0.00)
(0.47)
ENC-DEC
0.35%
0.23%
0.00%
(1.17)
(1.15)
(0.00)
Ortho
B1OOV
lmOOV
60OV
bothOOV
Total
Transer
en-NFIN
en-PRS
en-PRS356
CHR TRM
95.83%
14.32%
95.32%
7.67%
52.22%
CHR TRM
0.00%
0.00%
(11.76)
(39.60)
(7.34)
(20.30)
(11.60)
(0.00)
(0.00)
(0.00)
CLUZH
96.60%
20.43%
95.28%
12.85%
54.96%
CLUZH
0.92%
0.09%
(7.84)
(89.60)
(7,83)
(24.00)
(10.60)
(2,05)
(0.00)
(0:47)
ENC-DEC
8.58%
95.37%
5.24%
50.763
ENC-DEC
0.63%
0.00%
(7.84)
(31.60)
(7,59)
(15.79)
(8:50)
(1.69)
(2.79)
(0.00)
Transer
#200V
bnOOV
600V
bothOOv
Total
CHR-TRM
1418%
89.86%
7.04%
48.80%
Table 7: Percent accuracy and (accuracy range) on English PROBE splits by system and presentation style.
(25.72)
(38.38)
(11.67)
(17.25)
(10.30)
CLUZH
85.77%
18.10%
90.59%
11.44%
$1.48%
(13.77)
(38.48)
(f4,38)
(20. (2)
(6.00 00)
ENC-DEC
85.44%
4.12%
22%
2.37%
45.82%
Ortho
e5-AGGL
es-FUT
06-PSTPFV
eS-IR
6S-IRAR
(26.03)
(12.12)
(13.64)
(6.08)
(6.00)
57.47%
38.93%
8.31%
48.61%
6.52%
(3.54)
(56.4T)
(27.27)
(21,13)
(5.42)
Table 4: Average percent accuracy and (accuracy range) on BLIND English across seeds for each system and presentation style. no00V triples where both lemma and feature set were observed; 1m00v = lemma is OOV; fs00v= = feature set is OOV. both00v lemma and feature set are OOV.
24.99%
5.19%
3.44%
32.39%
37.05%
(16.64)
(12.50)
(10.26)
(73.57)
(12.09)
0.107%
0.08%
0.00%
21.2%
5.98%
(0.00)
(0.00)
(0.00)
(28.01)
(8.76)
Transer
AGGAGGL
FUT
es-PSTPFV
es-IR
eS-TRAR
CHR-TRM
57.61%
44.57%
51.12%
7.86%
(28.98)
(72.62)
(27.27)
(12.25)
(6.05)
30.05%
36.41%
35.13%
40.36%
(25
(46.15)
(19.05)
(10.59)
(12.09)
0.19%
0.009
0:00%
21.08%
8.55%
(0.96)
(0.00)
(0.00)
(8.57)
(6.69)
B2OOV
lm00V
K00V
bothOOV
CHR-TRM
93.38%
48.14%
91.92%
47.98%
70.16%
Table 8: Percent accuracy and (accuracy range) on Spanish PROBE splits by system and presentation style.
(6.87)
(18.4))
(1.67)
(5.43)
(4.90)
CLU1ZH
91.98%
33.14%
93.38%
32.5%
62.74%
(9.63)
(21.45)
(2.45)
(13.06)
(7.50)
ENC-DEC
90.54%
0.59%
90.27%
0.25%
45.34%
Ortho
SM-1PL
SW-NON3
SM-FUT
SW-PST
SM-PSTPFV
(9,56)
(2,97)
(5,55)
(0.5%)
(1,40)
CHR-TRM
99.57%
85.00%
98. 35%
60.48%
98. 19%
Transer
#e00V
lm00
60OV
bathOVV
(1.14)
(11.77)
(3.60)
(40.12)
(9.07)
CHR-TRM
93.55%
56.40%
93.9%%
56.18%
78.26%
CLUZH
66.71%
53.80%
10%
12.50%
37.66%
(7.27)
(29.78)
(3.03)
(7.40)
(6.70)
(29.70)
(28.52)
(64.49)
(35.44)
(73,24)
CLUZH
92.22%
58.37%
93.3
57.21%
65.32%
ENC-DRC
D00S
0.05%
0.00%
0.00%
0000%
(10.41)
(29.89)
(3.35)
(25.79)
(13.50)
(0.00)
(0.24)
(0.00)
(0.00)
(0.00)
ENC-DEC
92.01%
0.39%
91.20%
0.71%
46.05%
Transer
Sw-1PL
SW-NON3
SM-FUT
Sw-P5T
SM-PSTPPV
(7.46)
(1.96)
(3.54)
(2.78)
(1.20)
CHR-TRM
99.39%
85.56%
96.94%
70.56%
96.24%
(1.14)
(8.20)
(5.38)
(25,74)
(9.79)
Table 5: Average percent accuracy and (accuracy range) BLIND Spanish across seeds for each system and presentation style. no00V= = triples where both lemma and feature set were observed; 1m00v lemma is OOV; fs00v feature set is OOV. bothoov lemma and feature set are O0V.
CLUZH
68.00%
47.94%
36.25%
21.68%
37.61%
(26.82)
(15.57)
(36.8))
(55.85)
(77.64)
ENC-DRC
DOUS
081%
DA00S
0.00%
0.00%%
(0.00)
(2.16)
(0.00)
(0.00)
(0.00)
Table 9: Percent accuracy and (accuracy range)
Swahili PROBE splits by system and presentation style.

---

FS-FUT
Train/TTune Only Feature Sets
VEIND: 2;; TORM
FUTN
Tost-Only Feature Sets
V; FUT: 1 FPL
V; TND; FUT; 2;5G,
V:IND; 2;; 2; PL;ININ INUM 3
Tmiis-Only Triples Sampled
8 Other Train/Tune Tume Triples Sampled
1962
Only Triples Sampled
Unique I-nily Sets Sampled
# Unique Other Train/FT F. Sets Sampled
145
Unique Tesr-Only Sampled
FU-FUT Seed
Train/ lune Only Feature Sets
V; FUTT (TTTTT SSE
FUIND; INFM
Test-Only Festure Sets
V; FU 1
FUT; 2;56; TNFH
LND; 2;56;
FUT: 3:56 3UT,
Only Triples Sampled
# Other Train/FTune Triples Sampled
1966
Tely-Only Triples Sampled
Unique Trail Sets Sampled
# Unique Other Train/FT Sets Sampled
145
Unique Tost-Only Sets Sampled
Table 10: Description of es-FUT train and test feature for seeds and 4. Seeds 1-3 provide similar results, and similar splitting logic applies to the other PROBE splits. Triples with feature sets containing FUT are par- titioned random by seed into those that can only be sampled for finetueee and those that can only be sampled for test. Otherwise, data is split uniformly at random in most SIGMORPHON shared tasks. No relevant PROBE feature set appears both train and but all other feature sets may. Evaluation is only performed on sampled triples with the relevant PROBE feature set. This approach allows us to test the specific impact of the PROBE otherwise typical setting.


Reviews:
[{"review_id": "3Hssmp6KRC", "reviewer": "Reviewer_1U8W", "summary": "", "questions": "* line 240: which feature tags inconsistencies in the UniMorph data are you referring to?\n* Limitations Section: you mention that future work includes a 'larger group of experts'. How many experts were involved in the current experiments? or any relevant information\n* Appendix A (Tables 8,9): The model ENC-DEC performs significantly worse than the other two, achieving 0% accuracy on most of the Spanish and Swahili probes. Are there any insights as to why it underperforms in languages other than English? Also, does this suggest that character-level models are generally better at morphological generalization?", "limitations": "", "rating": 5, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review.\n\nRegarding both reproducibility and clarity, we will publish our cleaned data sets as well as the splits that we used on Github if the paper is accepted.\n\nRegarding tagging inconsistencies, in Swahili, we find that the tags for the same inflection are sometimes presented in a different order, and sometimes they use different abbreviations for the same feature (e.g., PFV vs PRF for perfective). As part of the published data set, we will explain what cleanup we did in the README.\n\nThe number of experts involved in our data preparation is the same as the number of authors on the paper. A small number which will be clear once the paper is deanonymized. Even twice that number would be \"a large number\" in our estimation, but still well down in the single digits. This is nice, because it means the effort to create larger and more varied probe data splits from existing UniMorph is very feasible.\n\nIf accepted, we intend to spend most of the 5th page better explaining the probes and their purpose as well as the implications of the analysis. We have some vague intuitions about why ENC-DEC underperforms relative to the other models that we can add into the paper. The ENC-DEC model seems to struggle with keeping track of the contribution of the individual features in the feature sets, so it often makes strangely lateral guesses when just a single one is altered. Notably, the only Spanish probes that it succeeds on at all are the lexical ones involving conjugational class, not probes that held out features or sets of features, which further suggests that it's the feature sets that it struggles with. It was originally published in a paper that focused on English, which only has small feature sets, so this issue didn't really crop up.\n\nWe will work on improving the color differentiation for Swahili in Figure 1. The colors were hand-selected, so they won't be hard to modify.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper is well written. The method and ideas are well structured and clearly presented, which greatly helps the reader to follow and understand the paper.\n* The experimental setup is technically sound and the authors provide a thorough comparison and evaluation on different models and data splits (i.e. BLIND similar to prior work and PROBE, which targets specific probes).\n* The selection of languages (i.e. English, Spanish, and Swahili) is also worth pointing out, as they represent distinct morphological features.", "reasons_to_reject": "I did not identify a potential reason to reject the paper.", "questions_to_authors": "* line 240: which feature tags inconsistencies in the UniMorph data are you referring to?\n* Limitations Section: you mention that future work includes a 'larger group of experts'. How many experts were involved in the current experiments? or any relevant information\n* Appendix A (Tables 8,9): The model ENC-DEC performs significantly worse than the other two, achieving 0% accuracy on most of the Spanish and Swahili probes. Are there any insights as to why it underperforms in languages other than English? Also, does this suggest that character-level models are generally better at morphological generalization?", "paper_topic_and_main_contributions": "The paper proposes a set of 13 language-specific probes for English, Spanish, and Swahili and evaluate the ability of 3 different models from prior work to generalize morphological patterns. This work supplements prior work on morphological inflection, which focuses on language-independent data splitting.", "excitement": 5, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Figure 1: I find hard to distinguish between different shades of the same colour, and specifically, the sw-1PL and sw-FUT you mention in line 252."}, "extra_text": {}}, {"review_id": "Lb2BESqVXD", "reviewer": "Reviewer_gRQQ", "summary": "", "questions": "1. Why did you select these specific probes? Explanations are very condensed (I understand the lack of space in a short paper) and they are hard to understand and to know the motives behind them.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review.\n\nWe agree that the explanations of the probes can be improved and we should take the space to explain why we chose them (It's tricky getting everything into 4 pages for a short paper submission). If the paper is accepted, we intend to use most of the 5th page for expanding the explanations of the probes and expanding on the analysis. That should improve clarity substantially.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Language-specific, linguistic-motivated experiments.\n- Include 3 languages with different typology.\n- Statistical and error analysis.\n- Clear and concise writing.", "reasons_to_reject": "None", "questions_to_authors": "1. Why did you select these specific probes? Explanations are very condensed (I understand the lack of space in a short paper) and they are hard to understand and to know the motives behind them.", "paper_topic_and_main_contributions": "This paper introduces language-specific linguistic probes to assess the generalization of morphological inflection systems in English, Spanish, and Swahili. The probes are designed to be meaningful, considering linguistic information, providing a novel approach compared to previous language-independent methods. The study explores the impact of ambiguous examples and evaluates model performance on out-of-vocabulary (OOV) data. Additionally, the paper compares the use of orthography and phonological transcriptions, concluding that there are not significand differences. Statistical analysis confirms the significance of model and language factors on the results. Results prove the usefulness of the proposed probes to test the strengths and weaknesses of morphological inflection models.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Explanation of probes is a bit hard to follow. It would be helpful to have some qualitative examples on a Figure to make it easy to have an idea of what processes are being test by the probes."}, "extra_text": {}}, {"review_id": "jre9CsfV9o", "reviewer": "Reviewer_FThw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We fully agree with the reviewer here that the term \"cognitive\" should not be thrown around lightly, and that it hasn't really been demonstrated that these LMs make appropriate cognitive models. Our use of the term was meant to convey the position taken in some of the literature that attempts to use NNs for cognitive modeling. For example, the paper that introduces our ENC-DEC model explicitly positions itself as a cognitive modeling paper. We will either clarify this in the text or just remove the term altogether as the reviewer suggests.\n\nWe will find a way to make Figure 1 clearer. The shapes are meant to be visually redundant ways to differentiate the models, but if that's confusing, we don't necessarily need them, since the models are already grouped into sub-columns. In any case, we plan on expanding the Fig 1 caption with a lot more explanation on how to read the plot if the paper is accepted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is very interesting to me because it is linguistically informed. It provides a detailed and meaningful description of the data and methods. The topic falls into the category that many consider the most important today: how well models encode language structure. It goes beyond most papers that are published on this topic in that the design is actually linguistically meaningful. The authors provide insightful and detailed comments on their results.", "reasons_to_reject": "I would like to see this paper accepted. The only thing I object to it is the somewhat unnecessary use of the term \"cognitive\" a couple times; so far nobody has really shown that any of the LMs encode anything truly \"cognitive\", and for the given paper, it doesn't seem necessary to repeat that dubious term.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This short paper designs and presents linguistically-motivated morphological probes for inflectional morphology models. The authors observe that most prior work relies on language-independent probes while they would like to see how well the models do with respect to common language specific features, e.g. the fact that in Spanish, -ir verbs share some but not all inflections with -er verbs; the ways Spanish tenses are formed, and so forth. They design probes for three languages: English (little morphology; fusional), Spanish (rich morphology; fusional), and Swahili (rich morphology; agglutinative). They hypothesize that some probes will be more challenging than the others, and show that to be the case.", "excitement": 5, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "It took me a bit to figure out how to read the results Figure (Figure 1). Maybe a comment on the different icon shapes would help?"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The short paper proposes a set of 13 language-specific (English, Spanish, Swahili) probes for morphological generalization that are based on conjugation classes.

All reviewers agree that the work presents a nice, well-written linguistically-motivated study and strongly suggest it for acceptance. I do not see any clear objections and/or potential risks and would be happy support the paper and see it accepted.
Decision: Accept-Main