A Study on Accessing Linguistic Information in Pre-Trained Language Models by Using Prompts
Authors: Marion Di Marco
Katharina Hämmerl
Alexander Fraser
Venue: EMNLP 2023 Main (2023)
Keywords: prompting for linguistic information
morphology
morphological features
TL;DR: We use natural language prompts to formulate linguistic tasks in pre-trained language models.

Abstract: We study whether linguistic information in pre-trained multilingual language models can be accessed by human language: So far, there is no easy method to directly obtain linguistic information and gain insights into the linguistic principles encoded in such models. We use the technique of prompting and formulate linguistic tasks to test the LM’s access to explicit grammatical principles and study how effective this method is at providing access to linguistic features. Our experiments on German, Icelandic and Spanish show that some linguistic properties can in fact be accessed through prompting, whereas others are harder to capture.

A Study on Accessing Linguistic Information in Models by Using Prompts
Language
Marion Di Marco! and Katharina Hämmer1 1.2 and Alexander Fraser CCenter for Information and Language Processing, LMU Munich, Germany Munich Center for Machine Learning, Germany (dimarco, haemmerl, Jecis. 1mu.
Abstract
tion, and what type of information can be obtained, versus what out of reach for the method. There has been limited work on prompting English mod- els for English linguistic information, while our focus is on other languages. We are also interested in prompting in one language and obtaining lin- guistic information in another, such as language learner linguist might do. This will be also of interest for working with linguistic information in low-resource languages, both when looking only at a low-resource language individually, or in combi- nation with another (high-resource) language, such as the pair of Spanish and Mayan languages.
We study whether linguistic information in pre- trained multilingual language models can be accessed by human language: So far, there is no easy method to directly obtain linguistic in- formation and gain insights into the linguistic principles encoded in such models. We use the technique of prompting and formulate linguis- tic tasks test the L.M's s access explicit gram- matical principles and study how effective this method is at providing access to linguistic fea- tures. Our experiments on German, Icelandic and Spanish show that some linguistic proper- ties can in fact be accessed through prompting, whereas others are harder to capture.
In our experiments, we formulate question prompts for different linguistic tasks, for exam- ple to "ask" for the grammatical case of a word in given context sentence. The tasks are designed classification tasks, where the labels correspond linguistic properties (e.g., singular, plural for the feature number). Our experiments address standard morphological features, such as tense, case, num- ber, gender, but also syntactic information, such as the distinction between subject and object, and the distinction between active and passive voice.
1 Introduction
With pre-traine pre-trained language models becoming ever larger and continuously better at solving variety of tasks, there is growing interest in understand- ing how they represent knowledge, particularly lin- guistic information. While pre-trained language models carry out impressive feats of linguistic gen- cralization, there is no simple way to directly access the linguistic knowledge encoded in them. Easy access to linguistic information, and - in a more long-term vision - the ability to modify parts of the linguistic representation, would be powerful instrument. For instance, linguist without expert knowledge in machine learning could use linguistic querying to gain insights into a model's 's linguistic properties and to pinpoint problems with its linguis- tic representation. Similarly, for a language learner, interacting with a language model in the form of textual instructions could provide interesting new learning opportunities.
Our findings indicate that features corresponding to concrete and locally defined properties like num- ber of nouns easier to obtain than more abstract features like the grammatical case of nouns. They also support the hypothesis that abstract grammat- ical terms (c.g., dative case) are not always well- suited for formulating prompts or answers as they are not necessarily well-covered in the grammatical sense in general- langral-l data.
We present this analysis some first steps in evaluating how relevant linguistic properties of language can be made accessible through human language queries. Possible next steps can be fo- cus on low-resource languages, as well as identify- ing problems in representing linguistic information and strategies to solve such problems.
In this work, we focus accessing linguistic information through natural language queries. We propose using prompting a simple way test the LM's access to explicit grammar principles. We study on German, Icelandic and Spanish how effec- tive prompting is for querying linguistic informa-

---

2
Related Work
Sentence given context:
Label: SG
Die Regelung ist umstritten.
There is strong interest in how linguistic informa- tion is represented in pre-trained LMs: Liu et al. (2019) is well-known probing study transferring linear probing techniques to contextual models. Shapiro et al. (2021) study multilabel probing to assess the morphosyntactic representations of mul- tilingual word embeddings. Stanezak al. (2022) investigate whether the same subsets of neurons en- code universal morpho-syntactic information and therefore enable cross-lingual generalization. Lasri et al. (2022) study how BERT encodes grammat- ical number in number-agreement task. Hewitt and Manning (2019) by contrast focus on struc- tural probes to find out whether BERT and similar models encode parse trees according their the- ory. Warstadt et al. (2020) provide challenge set using acceptability contrasts to evaluate the mor- phosyntactic and semantic knowledge LMs, test- ing for a range of phenomena. Liu et al. (2021) is a probing study of English RoBERTa across time.
The regulation is highly controversial. Sentence. Regelung" is _MMSKK Sentence. The number of Regelung iss _ASK_ Sentence. Number of Regelung _MASK_..
gloss
Verbalizer mapping: PI. plural, SG singular
Table 1: Prompts for the nominal feature number.
a set of possible output tokens. A verbalizer maps the class labels to natural-language outputs, and complctions are sampled exclusively from this set of output options. One example from the original paper is predicting restaurant ratings given reviews, where the verbalizer maps "terrible" rating and "great" to stars. In our experiments, the output classes typically correspond to the val- ues of grammatical features to be predicted, for example plural, singular for the feature number.
1-star-
Typically, the training of PET relies on several putterns, namely variations of the prompts. In a first step, individual models fine-tuned small training set for each pattern. The resulting models are then used label a larger set of unla- beled examples with soft labels as training data for a final classifier with a regular sequence classifica- tion head, similar to knowledge distillation.
The authors test range of probes and downstream tasks at dozens of checkpoints. Li et al. (2022) argue that prompting acts model-free probe, thus eliminating the distinction between what the model knows and what the probe learns. They compare prompting to linear regression and MI.P probing. Zhang et al. (2022) also probe GPT-3 using prompts, specifically on tense and number features. Blevins et al. (2023) prompt GPT models to append POS tags to each word in a given sen- tence. However, most of the above works focus English-only settings. We work with multilin- gual models and for the presented experiments, use mixed- language prompts, where the question is for- mulated in English to obtain information about a another language, in which the target word and the context sentence are given.
PET has two advantages for our experiments: It is designed to work well with small sets of training data, which might be useful for tasks with only few annotated data available; and it combines several prompt formulations for the same task, which alle- viates the problem of finding the "right" prompt.
Prompt Design and Data
To train PET, we need linguistically annotated data to derive prompts and the respective labels which then mapped natural-language words didates for the mask position in the LM.
3 Methodology: PET
Prompts and Labels For the formulation of prompts, we first present sentence as context, and then variations of statements like "the feature of W is _ASK_" " with W being a word in the sentence. Table 1 shows example prompts and the label vocabulary for the feature number.
We apply Pattern- Exploiing Training (PET: Schick and Schütze (2021a), Schick and Schütze (2021b), Schick and Schütze (2022)). PET is semi- supervised training procedure which combines tex- tual instructions with fine-tuning based on labeled examples. Its key idea is providing textual instruc- tions for the task (prompts) as cloze-style phrases for masked language model (Devlin et al., 2019). where the mask token is then substituted by one of
We mix the language of the prompt (English) and the languages of the queried feature (German, Icelandic, Spanish). Even though multilingual lan- guage models are robust to mixed-language input, there are some considerations: The terminology used in the prompts, namely grammatical features
WW use the implementation in https:///ithuul com/
timoschick/pet

---

and their values, might be insufficiently covered in the underlying training data of the language model in general, and potentially even more so for English labels referring to another language's grammar. For some terms, there might be domain shift (gram- matical gender vs. gender in the societal sense).
1S
ES
train
test
train
Test
train
Test
Tense
past
27
380
80
29
448
pres
73
446
117
47
fut
24
83
Number
sing
74
563
145
58
608
Nouns
plur
26
263
42
324
Number
sing
74
514
60
137
54
615
With the success of the method hinging the coverage of the relevant terminology (in English otherwise), we also experiment with substituting grammatical terms with general language discrim- inative context the respective target languages (cf. "variants" in Section 5.1).
Verbs
plur
26
312
60
46
317
Gender
mase
36
338
71
58
489
fem
40
342
41
64
42
443
146
Case
ucc
41
254
33
70
dat
25
228
27
57
gen
nom
29
259
26
59
Data Our d stem from the Universal De- pendency Treebank (Nivre al., 2020), which con- tains high-quality data for many languages. Our main focus on German, which has a complex nominal morphology with many syncretic forms making it often impossible to derive a word's fea- tures without context. In addition, we choose Ice- landic, which has a similar feature distribution (three gender values, four case values), but is lower-resource language compared to German. On the other hand, Spanish has a comparatively simple nominal morphology (two gender values, no noun cases), but a more complex verbal morphology, largely without syncretic forms.
Table Label distribution per feature and language.
DE
ES
IS
Tense
past
past
past
present
present
present
future
Number
singular
singular
singular
plural
plural
plural
Gender
masculine
masculine
masculine
feminine
feminine
feminine
neuter
neuter
Case
accusative
accusative
dative
dative
genitive
genitive
For the experiments morphological features, we select the target words (nouns and finite verbs) such that that each combination of lemma and fea- tures occurs only once, and that there is overlap between test and training data. The target word can only occur once the sentence, and we restrict sen- tence length to 50 words. Table in the Appendix shows the amount of training data per language.
nominative
nominative
Table 3: Labels the morphological features.
cur with proportional frequency, leading in some cases to very unbalanced data sets (see Table 2). We decided not to balance the data sets, but use the frequencies as they occurred on randomly cho- sen words to preserve the approximate proportions between the different features.
Table 3 provides an overview of the predicted features and the respective labels. For the feature Verb Tense, only look at finite verbs, but do not consider composed tenses (e.g. machte habe gemacht (made vs. have made)). For Spanish, the corpus annotation differentiates between two differ- ent forms of past tense (Past: preterite tense) and Imp: imperfect tense). To limit this prediction task to the more general prediction of past present fut ture, the two Spanish past tenses were mapped into one "Past" label. Furthermore, Spanish expresses future tense morphologically, whereas German and Icelandic use an auxiliary: (yo) andaré vs. ich werde gehen (I will go).
Experiments
For the experiments on morphological features, we use finite verbs and nouns as words W. While the nominal features number, gender and case also present for adjectives and determiners, to only look the phrase head for the sake of simplicity, assuming that the feature of the head is known, it also known for the rest of the phrase. In our tables, we give the accuracy of the final distilled model for bert-base berr-base-multiinnuua -aliimmmilnngll-caaaee and him--rbbrtaa-arge (Devlin et al. (2019), Con- neau et al. (2020)). We report the average over
chose
The different feature values do not always oc-
"The data sets can be found at mar wwtoonmimrc///mao wwwohhhe___dhe _aa___dattse//
com/

---

runs with different seeds, using the default parameters of PET.
finite VERBS
DE
mbert
0,841
0.594
0,932
Tense
ximr-large
0.965
0.594
0.990
For each set of experiments in this section, we use 100 training examples, 1000 (DE/ES) or 500 (IS) examples to be labeled for the final model (cf. Table 8). Using the same sets of prompts for all languages, formulated in English, we assume that the different prompts do do equally across the languages. We use the dev set to select the best out of initially given patterns.
mbert
0.792
0.790
0.995
Number
slrgelarge
0.725
0.954
1.00
NOUNS
DE
IS
ES
mbert
0.914
0.762
0.999
Number
ximr-large
0.942
0.843
0,998
mbert
0.414
0.325
0,483
Gender
ximr-large
0.765
0.544
0.993
Case
DE-100
DE-250
DE-500
IS
mbert
0.328
0.367
0.444
0.363
5.1 Morphological Features: Nouns and Verbs
xlmr-large
0,313
0.334
0.377
0.307
Table 4 gives an overview of the experiments: For number' and tense, the predictions are quite high (with the exception of Icelandic tense), even when considering that the number of output classes is not balanced (cf. Table 2). This is contrasted by the performance for gender and particularly case; increasing the amount of labeled training examples only leads to moderate improvement in German.
Table 4: Accuracy for tense, number, gender and case.
means the two runs differed by > 0.2.
Gender
DE-100
DE-250
DE-500
IS
mbert
0.815
0.838
0.851
0,498
larr-large
0.412
0.594
0.412
0.904
Case
DE-100
DE-250
DE-500
IS
mberr
0.370
0.470
0.541
0.323
ximr-large
0.330
0.610P
0.430
0.289
Case is complex and context-dependent it often corresponds the syntactic function of subcat- egorized noun, and thus requires a general under- standing of the entire sentence. Gender a local feature innate to a noun, with mostly arbitrary val- ues that do not correspond to real-world properties and are inconsistent between languages. The fea- tures tense and number are more tangible in the sense that the terms used for their description are meaningful and also prevalent in general language. Particularly for German, the high number of syn- cretic forms makes the learning task even more difficult. While there can be context keys, such determiners and inflectional suffixes within the noun phrase, this is, however, limited to certain contexts and inflectional morphemes often not directly accessible due fusional morphology and inconsistent subword segmentation.
Table 5: Discriminative context as labels.
systems, the less-represented labels (for example genitive in German case) are not well represented in the models, and are thus under-predicted
Variants We explored using discriminative con- text words for gender and case to have a more common-language label vocabulary: By associat- ing the target word with a context word indicative of gender/case, aim at directing the model's attention relevant context in general while not impeding the prediction with presumably insuffi- ciently covered terminology. For gender, we used personal pronouns in nominative case, assuming they provide useful content, e.g. when referring to the target noun. For casc, we use definite (German) and indefinite (Icelandic) articles as labels. How- ever, articles are highly syneretic (ef. Table 10), and thus not always distinet. In general, we ob- served some improvements for gender, and even for case (cf. Table 5), which is nevertheless still far from good. Interestingly, xlmr-large tends to pro- duce lower results than mbert for German gender, in contrast to the results from using English labels.
The results of Icelandic are notably worse than those of the other languages. One possible factor might be that the proportion of Icelandic in the PLM training data is considerably smaller than for German Spanish; furthermore, data set for Icelandic also comprises less training data.
Another issue is the unbalancedness of some of the data sets; as a result, we observed that in some
This result indicates that abstract grammatical terminology is not necessarily the best and only way to address linguistic features. However, one should also bear in mind that the formulation prompts has been shown to be finicky (Webson
*While most systems showed consistent results the two runs, some had large differences (marked with for a difference > 0.2. This concerned to large part Icelandic systems, and some the German system based zimrelarge. NNumeer for German nouns is not lexicalized (unlike Span- ish). Singular and plural can be the same, cf. Table 9.

---

and Pavlick, 2022), and that non-optimal prompt formulation does not always lead bad perfor- mance: this adds a new layer of difficulty to our task as it makes a clear interpretation difficult.
Subj/Obj
DE000
DE250
DE
ES
mbert xlmr-1
0.635
0.736
0.817
0.752 0.832
0.692
0.785
0.880
0.709
0.739
Table 6: Distinction of subject and direct object.
5.2 A Broader Variety of Linguistic Tasks
German
verb particles
pp-attachm.
passive
Here, we study linguistic tasks that go beyond querying for standard morphological features.
mbert
0.80
0.522
0.674
xlmr-large
0.89
0.501
0.692
Subject-Object This task consists in deciding whether word is the subject or direct object of verb in given sentence context. We only use nouns, no pronouns, as target words. While this is similar to predicting case, this variant is eas- ier with only the two labels subject and object, which have the additional benefit of being more commonly used than the actual labels for the differ- ent grammatical cases. In contrast to the previous experiments, we designed the set to have approxi- mately balanced classes (see Table 11 for the label distribution). Furthermore, as many nouns can take both the subject and the object position, the same verb-noun pairs can occur several times across the data sets. The same applies in general for the three further syntactic tasks, as the realization of the re- spective features depends on the context.
Table 7: Further linguistic tasks for German.
wurde den Spitzenclubs... die Möglichkeit geboten. auf höherem Niveau uls gegen die besten Clubs zu spielen.
were the top clubs the opportunity offered higher level than against the best clubs play.
verb=gehoten, pr=auf Niveau NO verb=spielen, przauf Niveau
YES
Passive Voice
The verb werden can occur in sev- eral functions, for example as an auxiliary for pas- sive voice or future tense, see the examples below:
Der Reaktor muß für eine Woche abgestellt werden YES The reactor must be turned off for week
wird in künftigen Fällen nicht mehr ausreichen ". will not be enough future cases
NO
daB er ... Mitglied Republikaner geworden that ... became member of Republicans
NO
Particles German verb particles can occur sepa- rately from the verb, for example ausschneiden schneidet ... aus (Io cul oul). There can be a large distance between particle and verb, making this task not trivial. As contrastive examples, we used sentences where a preposition (restricted to those that share the same form as frequent particles) is attached to verb. The queries are formulated as variants of "Is P particle of V?".
To identify passive structures, the queries are for- mulated as 'Is "werden" a passive auxiliary?". Table 6 shows a reasonable performance for the task of subject-object identification task across lan- guages; as before, we observe that increasing the number of training examples leads to improvement at least for mbert. The other three tasks are carried out only for German; the results are showed in Ta- ble 7. We observe mixed performance, indicating that PET can be applied some linguistic tasks.
There are many verbs that occur with and without particle, as shown in the example below:
er spielte noch in drei Tonfilmnn mit
YES
Conclusion
he played in three sound movies particle
spielte die Band Konzerte mit Bands wie
NO
We presented a study on using prompting to ac- cess linguistic information in pre-trained LMs for German, Icelandic and Spanish. We evaluated pre- dicting set of morphological features, as well a broader variety of linguistic tasks. Our results indicate that some features are indeed accessible, whereas the approach fails for features such as the complex and abstract grammatical case.
played the hand concerts with hands
PP-attachment For this task, we select sentences with (at least) two verbs, and ask whether a prepo- sitional phrase PP is attached to verb V, which is either the verb the PP is attached to, or randomly selected, other verb. Due to German sentence order, there can be long distance between the verb and attached PP, making this task very challenging. The prompts are formulated as variants of "Is at- tached to V?". We show an example below, where the PP auf Niveau is attached to the verb spielen (to play):
Using natural language to obtain linguistic in- formation is potentially useful in many scenarios; querying for variety of non-standard linguistic tasks might be an especially interesting use case, in particular for low-resource languages.

---

Limitations
whether it is just not accessible by means of our current method. We plan to work on separating these distinctions the future.
The work reported in this paper, study on ac- cessing linguistic information in pre-trained lan- guage models through natural language instruc- tions, presents an initial but targeted study on Ger- man, Icelandic and Spanish data.
Ethics Statement
The authors see no ethical concerns with the work presented in this paper.
An obvious limitation lies in basing the exper- iments on three European languages that are not representative of many different language families. We chose German as our main target language be- cause it has a complex enough morphology to pro- vide a set of challenging tasks, but at the same time has excellent linguistic tools available to be used in the analysis. We furthermore chose Ice- landic as related, but lower-resourced language and Spanish with its comparatively simple nominal morphology as a contrasting language. Similarly, we tried to model features from different linguistic levels (both morphological and syntactic features) in order to provide a reasonably comprehensive overview. Exploring more language families and their respective grammatical features, as well as ap- plying the methodology to low-resource languages makes for an interesting project that is, however, beyond the scope of this work.
A Acknowledgements
This publication was supported by LMUexcellent, funded by the Federal Ministry of Education and Research (BMBF) and the Free State of Bavaria un- der the Excellence Strategy of the Federal Govern- ment and the I Länder; and by the German Research Foundation (DFG; grant FR 2829/4-1).
References
Terra Blevins, Hila Gonen, and Luke Zettlemayer 2023. Prompting language models for linguistic structure. In Proceedings of the 61st Annual Meeting of the Association for Compulational Linguixtics (Volume 1: Long Papers), pages 6649-6663. Toronto, Canada. Association for Computational Linguistics.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning scale. In Pro- ceedings of the 58th Anmual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
We primarily focused on accessing information through natural-language instructions, but without actual knowledge whether the queried information is really encoded in the model. To extract linguis- tic information, one needs to address two ques- tions, namely, "Is the information there at all?" and, "How can we access it?" In this study, we skip the first question but just assume that the information is available. While this is generally a reasonable ussumption, as previous work has shown that there is knowledge about grammar and linguistic struc- ture in LLMs, we do not know for sure whether information about the feature we are interested in is encoded in way that corresponds to the query and the pre-defined label vocabulary.
Jacob Devlin. Ming-Wes Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
John Hewitt and Christopher D. Manning, 2019. A structural probe for finding syntax in word represen- tations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Pupers), pages 4129-4138, Minneapolis, Minnesota. Association for Computational Linguistics.
With view to the comparatively good results of some linguistic features, we might say that informa- tion about those feature is there and can be obtained through natural- language prompts. For other fea- tures, we cannot really draw a conclusion. For example, consider grammatical case, where the prediction model clearly failed, but we saw evi- dence of awareness for related features in the task of distinguishing between subject and object. We thus cannot say for sure whether there is no knowl- edge about case in terms of the labels we used, or
Karim Lasri, Tiago Pimentel, Alessandro Lenci, Thierry Poibeau, and Ryan Cotterell. 2022. Probing for the usage of grammatical number. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages

---

8818-8831 Dublin, Ireland. Association for Compu-
Timo Schick and Hinrich Schüitze. 2022. True few-shot learning with Prompts-4 A real-world perspective. Transactions of the Association for Computational Linguistics, 10:716-733
tational Linguistics.
Jiaoda Li, Ryan Cotterell, and Mrinmaya Sachan. 2022. Probing via prompting. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages I 1474 Seattle, United States. Association for Computational Lin- guistics.
Naomi Shapiro, Amandalynne Paullada, and Shane Steinert-Thhrlkeee 2021. A multilabel approach to morphosyntactic probing. In Findlings of the Associ- ation for Computational Linguisties: EMNLP 2021. pages 1486-4524 Punta Cana, Dominican Republic. Association for Computational Linguisties.
Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith, 2019. Lin- guistic knowledge and transfcrability contextual representations. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, Volume 1 (Long and Short Pa- pers). pages 1073-1094, Minneapolis, Minnesota. Association for Computational Linguisties.
Karolina Stanczak, Edoardo Ponti, Lucas Torroba Hen- nigen, Ryan Cotterell, and Isabelle Augenstein. 2022. Same neurons, different languages: Probing mor- phosyntax in multilingual pre-trained models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 1589- 1598, Seattle, United States. Association forr Computational Linguistics.
Zeyu Liu, Yizhong Wang, Jungo Kasai, Hannaneh Ha- jishirzi, and Noah A. Smith. 2021. Probing across time: What does RoBERTa know and when? In Findings of the Association for Computational Lin- guistics: EMNLP 2021. pages 820 842, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Mariona Taulé, M. Antônia Martí, and Marta Recasens. 2008. AnCora: Multilevel annotated corpora for Catalan and Spanish. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08), Marrakech, Morocco. European Language Resources Association (ELRA).
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mo- hamancy, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The benchmark of linguis- tic minimal pairs for English. Transactions of the Association for Computational Linguistics, 8:377- 392.
Ryan McDonald, Joakim Nivre, Yvonne Quirmbach- Brundage, Yoav Goldberg, Dipanjan Das, Kuzman Ganchev, Keith Hall, Slav Petrov, Hao Zhang, Osear Täckström, Claudia Bedini, Núria Bertomeu Castelló, and Jungmee Lee. 2013. Universal Dependency an- notation for multilingual parsing. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers), pages 92-97, Sofia, Bulgaria, Association for Com- putational Linguistics.
Albert Webson and Ellie Pavlick. 2022. Do prompt- based models really understand the meaning their prompts? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 2300-2344, Scattle, United States. Association for Computational Linguistics.
Joakim Nivre, ter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo. Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal Dependencies v2: An evergrowing multilingual treebank collection. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4034-4043, Marseille, France. European Language Resources Association.
de Marneffe, Filip Gin-
Lining Zhang, Mengchen Wang, Liben Chen, and Wenxin Zhang. 2022. Probing GPT-3's linguistic knowledge on semantic tasks. In Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and In- terpreting Neural Networks for NLP, pages 297-304, Abu Dhabi, United Arab Emirates (Hybrid). Associa- tion for Computational Linguistics.
Timo Schick and Hinrich Schütze. 2021a. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Asso- ciation for Computational Linguistics: Main Volume, pages 255-269, Online. Association for Computa- tional Linguistics.
Timo Schick and Hinrich Schütze. 2021b. It's not just size that matters: Small language models are also few- shot learners. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 2339-2352, Online. Association for Computational Linguistics.

---

A Data Information and Test Set Creation
DE
IS
ES
train
object: 49
object: 52
object: 45
We used the UD Treebank corpora de ge_gsd-ud- (dev/est/traini) (MeDonald et al. 2013) for German, es_ cuccor--uddddevvtss//riii (Taulé et al., 2008) for Spanish and is modern-ud ud- (dev/esst/rriin) for Icelandic. Table gives the amount of training data for all three languages.
subject: 51
subject:
subject: 55
test
object: 505
object: 408
object: 500
subject: 495
subject: 402
subject: 500
Table 11: Overview of the label distribution of the subject-object experiment.
German
Spanish
Icelandic
Particles
PP attachm.
Passive
selected from (lest/dev/rain)
385/186/1941
932//96/3118
197/114/606
train
part-yes: 54
attach-yes: 40
pass-yes: 50
linal test
826
932
197
part-no: 46
attach-no: 60
pass-no:
final dev
184
696
114
test
part-yes: 160
attach-yes: 504
pass-yes: 224
final train
100/250/500
100
100
part-no:
attach-no:
pass-no:
unlabeled data
1000
1000
500
Table 12: Overview of the label distribution of the Ger- man syntactic tasks.
Table 8: Overview of the training data: number of sen-
tences. (*: Some of the unused training data was added to the test data.)
training conditions between PET and the probing models are not comparable in a straightforward way. The probing models are based on 100 training sentences (PET training data) and the additional 1000 sentences used in the distillation step (1 100 sentences total). We observe that for most fea- tures, at least one variant of PET is better than the respective probing result, with the exception of case, for which PET is always worse. This shows that prompting is generally well-suited to accessing linguistic information, and is in some cases even superior to prompting, but also that some features out-of-reach with this method. See Tables 13 and 14 for detailed results.
B Background: German Grammar
Tables and 10 provide examples for syncretism in definite articles and nouns.
case
singular der Elefant den Elefanten
plural
nominative
die Elefanten
accusative
die Elefanten
dative
dem Elefanten
den Elefanten der Elefanten
genitive des Elefanten
Table 9: Inflected forms of Elefant (elephanr): there is only one distinct singular form (nominative case), whereas all other forms are identical with the plural form.
To investigate how PET fine-tuning changes the models, we also probed the resulting PET models. Overall, the PET models tend to perform worse on the probing task, except for those that obtain very high results in the morphological prediction task. seems that PET training does not add (much) information relevant for probing, and that less well- performing PET models impair the original model with regard to the probing task. See Table 15 for detailed results.
(cf. www.duden.ccm wtt.dd/wwww.ee...eiaiiiaaaaaaaae
MASC
FRM
NEUT
MASC
FRM
NEUT
NOM
der
die das die
die
die
ACC
den
die das
die
die
die
DAT
dem
der dem den
den
den
GEN
des
der
des
der
der
der
SINGULAR
PLURAL
Table 10: Definite articles: bold-faced are used labels for the prediction of case (der, den, dem, des).
For each morphological feature, trained separate probing model based on linear classifier, running over 25 epochs. In order be comparable to the training conditions of PET, the training data consists of the labels of the respective feature for nouns (number, gender or case) and verbs (number. tense), whereas all other words in the sentence are labelled NA.
Further Linguistic Tasks: Data Details
Table 11 shows the label distribution for the subject- object distinction task. Table 12 shows the label dis- tribution for the three syntactically inspired tasks.
D Alternative Approach: Probing
As the comparison between the training condi- tions of PET (100 labeled training examples in ad- dition to 1000 unlabeled examples to train the final
As an ablation, we trained linear classifier prob- ing models for each morphological feature. The

---

DE
IS
ES
ES
verb
mbeet
0.814
0.766
0.864
verb
PET mbert
0.860
0.736
#0.926
tense ximr-large
0.831
0.761
0.838P
tense
PET ximu-large
0.874
0.777
90,954
verb
mbert
0.895
0.797
0.902
verb
PET mbert
0.866
0.812
0.947
number ximr-large
0.823
0.766
0.866
number
PET xlmu-large
0.910
*0.853
0.951
noun mbert
0.834
0.792P
0.856"
PET mbert
0.852
0.726
0,912
number
PET xlmu-large
0.849
0.838
$0,872
number xImr-large
0.752P
0.761
0.791
PET mbert
*0. 729
0.624
0,821
noun
mbert
0.677
0.584
0.829
gender
PET xlmu-large
*0,780
0.746
*0,865
gender
xlmr-large
0.551
0.65P
0.714P
noun
PET mbert
0.609
0.665
mbert
0.429
0.731
case
PET xlmu-large
0.630
*0.807
case
xlmr-large
0.368
0.635
Table 15: Prohing experiment with n= 1 100 sentences
Table 13: Probing experiment with n=100 sentences training data (corresponding to n=100 labeled exam- ples in the PET experiment). P: the respective PET- 100 model (EN labels or discriminative context), in Tables 4 or better.
(600 for IS) training data applied to the respective best PET-100 model (EN labels or discriminative context). #; PET model is better than the respective original model in Table 14.
E Computational Resources
DE
IS
ES
We use the implementation of PET available from https:/ //gi thub. com/ coon/timscchcckppet
verb
mbert
0.891
0.812
0.912
tense
xImr-large
0.89
0.792
0.938
We ran the PET experiments on CPU with up to 20 threads (Intel(R) Xeon(R) CPU E5-2630 v4). The time used per experiment depends the par- ticular setting (mainly number of patterns) and the model size. For example, training the model for German noun number with n=100 training sen- tences on three patterns took approximately 2 hours for mbert and 6.5 hours for ximr-large.
verb mberrt
0.92
0.817
0.954
number ximr-large
0.915
0.832
0.959P
noun mbert
0.87
0.817
0.916"
number xlmr-large
0.861
0.868
0.87P
noun mbert
0.723
0.629
0.879
gender
xlmr-large
0.755
0.746
0.813P
noun
mberr 0.643
0.766
case
ximr-large
0.682
0.802
Table 14: Probing experiment with n=1 100 sentences training data (n=600 for IS), corresponding to the 100 sentences for PET training and the additional sentences used for the distillation step. the respective PET- 100 model (EN labels or discriminative context), in Tables 4 better.
classifier) and the probing models is not straightfor- ward, the probing models are based on 100 training sentences (PET training data), as well as on 1100 training sentences (PET training data 1000 extra sentences), with the respective feature labels given on nouns and finite verbs. This provides more train- ing data to the probing models, as PET only sees the label for one noun or verb per sentence in the training data, and also does not know the labels of the data for the distillation step.
We evaluate the probing models on the same target words as we do PET, counting prediction accuracy on those words while ignoring predictions for other words the sentence.


Reviews:
[{"review_id": "lwSG5dQIoj", "reviewer": "Reviewer_SkRy", "summary": "", "questions": "A. The paper should explain why the authors chose to mix the language of the prompt with the language of the queried feature. This seems like it would introduce some complexities, for example (as the paper states) English labels referring to another language's grammar. \n\nB. For the \"subject/object\" discrimination task, the paper should clarify whether these were the only choices in a given prompt and that one of the choices was correct (that is, no prepositional objects or nouns in compound noun expressions).\n\nC. What does \"best patterns\" mean at line 544? \"Using the same questions, formulated in English, for all languages, we used the dev set to select the 3 best patterns out of 5 initially given patterns assuming the respective patterns lead to different outcomes across languages.\"\n\nD. Line 318 \"We are also interested in code-switched prompts.\" This is the first time the paper refers to code-switching. Is that the same as \"mixed-language\" prompts? If not, it should be explained. If so, the terminology should be kept consistent.\n\nE. Why are the Icelandic results in Table 2 so much worse than the German results? The Icelandic results are almost at chance. This should be discussed.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. You are right that precision/recall/F1 scores would be helpful. As they were not output by the PET tool, we did not initially think to calculate these, but we can add this for the camera-ready based on the saved outputs. Yes, we moved a lot of details to the Appendices due to space constraints, and we will evaluate closely which points can or need to move up for the camera-ready. However, we believe that the main story is indeed contained in, and can be understood from, the body of the paper. Any missing elements will be straightforward to fix in the writing.\n\nIn response to your questions:\n\nA) Question about mixed-language prompts:\nIt is true that mixed-language queries add a certain complexity. However, it allows to keep the setting the same when testing several languages. Furthermore, many features have the same general meaning across languages (such as singular vs. plural) and might be better represented in English due to the larger amount of PLM training data. For other features, this is less clear, which is why we also investigate discriminative context as labels in the target language (cf. Table 3).\n\nB) Question about labels in the subject-object task:\nThe task is formulated with the two labels “subject” and “object”. And yes, one of the labels always applies. \n\nC) Question about best patterns:\nThis refers to the best-performing patterns on the development set.\n\nD) “Code-switched” vs. “mixed language”:\nYes, “code-switched” here refers to the prompts mixing languages. We will adjust the terminology.\n\nE) Performance of Icelandic: \nIcelandic has less training data overall in our data set, and the percentage of Icelandic in the PLM training data is much smaller than that for German and Spanish, (cf. https://data.statmt.org/cc-100/). We assume that this is the main factor for the comparatively bad outcome of Icelandic.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Although as reviewed in section 6, \"Related work\", there seems to be quite a bit of previous work in attempting to extract grammatical information from LLM's, with similar goals to those in this paper, this paper presents results on an interesting variety of grammatical features and compares the results for three non-English languages. For this reason, the results are interesting and could be valuable to the EMNLP community.", "reasons_to_reject": "Because the relative frequency of each of the items being distinguished can be quite different, (Table 7), the paper should also present results as precision, recall, and F1, in addition to accuracy, in order to clarify the meaning of the results because of the unequal numbers of each category in the data. \n\nThere is a lot of information included in the Appendices which I found necessary to read in order to evaluate the work. For example, Table 7 \"label distribution of features per language\" is very important for understanding the meaning of the accuracy results. Another example is the results for the probing task (Section 5) can't really be evaluated without referring to Tables 13 and 14 in the Appendix. In Section 5 the approaches being compared (PET vs. probing) are just described as being better or worse. The quantitative results are only provided in the Appendix. To compare the PET method to probing, there should ideally be a chart comparing the results in Table 2 to those in Table 13.  Another important piece of information found only in the Appendix is the definition of \"tense\". The main paper does not provide a definition of tense -- it is necessary to read Appendix A to find out what \"tense\" means in this experiment.", "questions_to_authors": "A. The paper should explain why the authors chose to mix the language of the prompt with the language of the queried feature. This seems like it would introduce some complexities, for example (as the paper states) English labels referring to another language's grammar. \n\nB. For the \"subject/object\" discrimination task, the paper should clarify whether these were the only choices in a given prompt and that one of the choices was correct (that is, no prepositional objects or nouns in compound noun expressions).\n\nC. What does \"best patterns\" mean at line 544? \"Using the same questions, formulated in English, for all languages, we used the dev set to select the 3 best patterns out of 5 initially given patterns assuming the respective patterns lead to different outcomes across languages.\"\n\nD. Line 318 \"We are also interested in code-switched prompts.\" This is the first time the paper refers to code-switching. Is that the same as \"mixed-language\" prompts? If not, it should be explained. If so, the terminology should be kept consistent.\n\nE. Why are the Icelandic results in Table 2 so much worse than the German results? The Icelandic results are almost at chance. This should be discussed.", "paper_topic_and_main_contributions": "This paper tries to determine if linguistic information about words and sentences can be accurately obtained from LLM's with prompts that directly request it. The paper describes experiments that test this question based on Icelandic, German and Spanish, which represent some diversity in morphology. It shows that some information is available from LLM's at a modest level of accuracy.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "I didn't note any missing references."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Line 241: \"While this is similar to the prediction of case, the prediction task is easier with only the two labels subject and object, that have the additional benefit of being rather common-language.\" What does \"common-language\" mean, and why is this helpful?\n2. There are many references to \"probing tasks\", especially in the section on previous work. it would be helpful to have a brief overview definition of this task."}}, {"review_id": "Wu1R9kFjmX", "reviewer": "Reviewer_CAHM", "summary": "", "questions": "No specific questions, but below I list some comments and suggestions that the author(s) might consider when revising the paper.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your kind review and helpful notes. We will address them as best we can in the camera-ready.\nYou are right that moving all or most of the appendices to the body would change the submission type. As we noted in another rebuttal, we do believe that the most important information is included in the paper body, but we will keep a close eye on what details can and should move up when revising.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Interesting issue that is addressed convincingly with results that are discussed clearly and correctly. Sound and valid methodology. Good discussion of the limitations of the study in the dedicated section (lines 335-394). The motivation and explanation for the interesting choice of languages covered in the paper (i.e. German, Icelandic and Spanish) given in lines 150-162 is very good and convincing.", "reasons_to_reject": "None particularly that is worth mentioning or that struck me particularly negatively.", "questions_to_authors": "No specific questions, but below I list some comments and suggestions that the author(s) might consider when revising the paper.", "paper_topic_and_main_contributions": "The paper addresses the very interesting, challenging and relevant question whether linguistic information in pretrained multilingual language models can be accessed through human language. The experiments reported cover German, Icelandic and Spanish. The manuscript addresses an original and timely issue and given its limited length (4 pages of actual paper) provides an interesting and well-argued discussion of the topic.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Nothing to report, except that the first reference of the list (Blevins et al., 2023) seems to be incomplete, i.e. some details probably missing.", "Justification_for_Ethical_Concerns": "N/A"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The paper is well-structured and generally well-written, it was generally a pleasure to read.\n\nSome comments:\n\n- The German of this reviewer is a bit rusty, but I'm fairly sure that \"(made vs. has made)\" in line 553 should instead be \"(made vs. (I) have made)\", as a gloss for \"e.g. machte vs. habe gemacht\" (with italicised words where appropriate, as is done in the manuscript), on the grounds that the auxiliary verb in the compound tense \"habe gemacht\" in lines 552-553 is first person singular. For this reason, I believe that the English gloss is incorrect here, please double-check and amend if/as required.\n\n- Line 555: \"(Past: preterite tenses)\" should probably be \"(Past: preterite tense)\", in the singular, not plural form (italicised words are fine).\n\n- I found it odd that section 6 on related work (lines 286-319) is located towards the very end of the paper, just before the conclusion. Unless there are requirements specifically against this, I would suggest to move it earlier on in the paper as customary, e.g. after the Introduction section. Also, this section on related work could highlight better and more clearly the specific contributions of the manuscript under review.\n\n- The table numbers (and the sections A / B / C etc.) are not mentioned in the right sequence in the running text of the manuscript, which is quite annoying (and has no specific reason, I believe, unless I missed something?).\n\n- In Section 1 (introduction), paragraph with lines 067-074, the author(s) could formulate those points as research questions and hypotheses that guided their study, rather than as a pre-empted discussion of the findings. In addition, in a similar vein the author(s) could discuss the potential applications of their research in a bit more detail.\n\n- What the author(s) call(s) \"section A / B / C\" etc. seem more appendices to me. In addition, sections (or possibly appendices) A, B, C and D (to a lesser extent section/appendix E) seem integral part of the paper, insofar as they report data and results of the experiments that were conducted for the study, and for this reason should probably be actual parts/sections of the main body of a long(er) paper, rather than \"left-overs\" lumped together at the end of the manuscript. But I understand that this would probably change the submission type of the manuscript, also for the purposes of its refereeing and evaluation for acceptance...\n\n"}}, {"review_id": "WPEEzCR9gJ", "reviewer": "Reviewer_uGEU", "summary": "", "questions": "- A: 4-5, 23-24: you say that there is no easy method to directly obtain linguistic information from LMs. Can you please substantiate this statement (maybe by citing some work)?\n- B: In Table 1, you give the English equivalent of the sentence given as context. It is not entirely clear whether you are using it here simply as a gloss, or you are giving that to the model too. I guess it is the former (and if so, clarify that it is just a gloss).\n- C: 146-147: \"general-language discriminative context\"; what does it mean? And, can you provide an example?\n- D: Sec. 5: why do also study probing? Is it relevant/comparable to PET? Do you need Sec. 5 at all?\n- E: Are you going to release the exact data you used?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. We would like to submit that the main points you raise are all writing and terminology issues and can be easily resolved for a camera-ready. Of course, we will do our best to make the terminology more consistent and clear.\nAs we said above, we will keep a close eye on what details can and should move up from the Appendix when revising.\n\nTo answer your questions:\n\nA) Question about “no easy method to obtain linguistic information” in the introduction: \nThis refers to obtaining information in an intuitive way, e.g. through natural language queries, that is also flexible to handle different types of (language-specific) linguistic information. We will make this more precise. \n\nB) Question about gloss in table 1:\nYes, this is a gloss, indicated by the italics. We can make this more explicit.\n\nC) Question about “general-language discriminative context”: \nThis refers to the experiments in Table 3, where we don’t use the actual labels (such as nominative or feminine) for case and gender, but a general-language context word indicative of the feature (We use articles and pronouns, cf. lines 214 ff. For example, the definite article “dem” only occurs in a Dative context in German, and we thus use it as a label instead of the word “Dative”). \nWe will make sure to explain this better already in Section 3.\n\nD) Question about probing:\nFunnily enough, this was requested in a previous round of reviews, as traditional probing is an existing, well-known method to access morphosyntactic information. As well, reviewers worried that the pattern-exploiting training was significantly changing the information available in the model. We will clarify the motivations for this.\n\nE) Question about releasing datasets:\nYes, we can release the data sets we used.\n\nThank you as well for the typos you caught!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The use of PET helps in mitigating some problems that may arise with prompting.\n- The work has a clear scope and the Limitations are explicitly defined.", "reasons_to_reject": "- I am not entirely clear with the terminology used. For example, what is the difference between \"tangible\" and \"abstract\" features? Where do these terms come from? Did you come up with them? If so, where do you draw the line? If not, please cite a work where these terms are defined.\n- The previous point affects the clarity of the manuscript's exposition. For example, in line 196, the term \"tangible\" is used again with a dubious definition, and \"complex and abstract\" is used in line 328. In addition, other points would need rephrasing. For example: line 209 \"common-language label vocabulary\" and the subsequent paragraph are unclear to me.\n- Sec. 4.2 is \"rushed\" with many details that are crucial for understanding better what was done relegated to App. C.\n- I would like to see a conclusive discussion clearly summarizing the results obtained.", "questions_to_authors": "- A: 4-5, 23-24: you say that there is no easy method to directly obtain linguistic information from LMs. Can you please substantiate this statement (maybe by citing some work)?\n- B: In Table 1, you give the English equivalent of the sentence given as context. It is not entirely clear whether you are using it here simply as a gloss, or you are giving that to the model too. I guess it is the former (and if so, clarify that it is just a gloss).\n- C: 146-147: \"general-language discriminative context\"; what does it mean? And, can you provide an example?\n- D: Sec. 5: why do also study probing? Is it relevant/comparable to PET? Do you need Sec. 5 at all?\n- E: Are you going to release the exact data you used?", "paper_topic_and_main_contributions": "This work studies the ability to access linguistic information by various LMs via prompting. The authors use PET as methodology and experiment with German, Icelandic, and Spanish data from UDT. They test morphological and syntactical features, framing the task as classification and formulating prompts asking for, e.g., the grammatical case of a word-form. The results show that some features are more easily accessible than others, with substantial differences among different languages.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- It would be advisable to move Related Work after Introduction to give the unaccustomed reader a comprehensive background.\n- Line 185: 'T'able 7 (check all instances).\n- Note 2: open inverted quote.\n- All links are not clickable.\n- All tables could be more appealing (e.g., changing fonts, using booktabs, etc.)"}}]

Author rebuttal:

Meta-review: The authors develop prompts to shed light on LLM’s access to explicit grammar principles. The authors evaluate on Icelandic, German, and Spanish. All reviewers appreciated the motivation of this work, but noted that it could benefit from clarifying its exposition.
Decision: Accept-Main