Prompting is not a substitute for probability measurements in large language models
Authors: Jennifer Hu
Roger P. Levy
Venue: EMNLP 2023 Main (2023)
Keywords: Large Language Models
Behavioral Testing of Language Models
Metalinguistic Judgment
Prompting
Minimal Pairs

Abstract: Prompting is now a dominant method for evaluating the linguistic knowledge of large language models (LLMs). While other methods directly read out models' probability distributions over strings, prompting requires models to access this internal information by processing linguistic input, thereby implicitly testing a new type of emergent ability: metalinguistic judgment. In this study, we compare metalinguistic prompting and direct probability measurements as ways of measuring models' linguistic knowledge. Broadly, we find that LLMs' metalinguistic judgments are inferior to quantities directly derived from representations. Furthermore, consistency gets worse as the prompt query diverges from direct measurements of next-word probabilities. Our findings suggest that negative results relying on metalinguistic prompts cannot be taken as conclusive evidence that an LLM lacks a particular linguistic generalization. Our results also highlight the value that is lost with the move to closed APIs where access to probability distributions is limited.

Prompting is not a substitute for probability measurements in large language models
Jennifer Hu Kempner Institute
Roger Levy
Department of Brain and Cognitive Sciences Massachusetts Institute of Technology rplevyêmit.. edu
Harvard University jenniferhu@fas harvard. edu
Abstract
current models that make them unsuitable as cog- nitive theories (e-g.. Dupre, 2021; Lan et al., 2022; Katzir, 2023; Milway, 2023; Murphy, 2023).
Prompting is now a dominant method for eval- uating the linguistic knowledge of large lan- guage models (LLMs). While other methods directly read out models' probability distribu- tions over strings, prompting requires models to access this internal information by processing linguistic input, thereby implicitly testing new type of emergent ability: metalinguistic judg- ment. In this study, compare metalinguis- tic prompting and direct probability measure- ments as ways measuring models' linguistic knowledge. Broadly, find that LLMs' met- alinguistic judgments are inferior to quantities directly derived from representations. Further- more, consistency gets worse as the prompt query diverges from direct measurements of next-word probabilities. Our findings suggest that negative results relying on metalinguistic prompts cannot taken as conclusive evidence that an LLM lacks particular linguistic gener- alization. Our results also highlight the value that is lost with the move to closed APIs where access to probability distributions is limited.
No matter their theoretical position, researchers need a way to assess the capabilities of LLMs order to substantiate such claims. The fundamen- tal unit of LLM computation is [[ttoke][onnexty which, in principle, can be directly read out from a model by accessing its output layer of vocabulary logits. The distribution that this implies over word strings reflects the model's linguistic generaliza- tions: that is, a generative model of the language seen during training, which can be used to evaluate the likelihood of previously unseen strings. Direct measurements of model-derived string probabilities have revealed capabilities such as syntactic gener- alizations (e.g.. Linzen et al., 2016; Futrell et al., 2019; Hu al., 2020; Warstadt al., 2020), seman- tic plausibility judgments (Kauf al., 2022), and certain coherence inferences (Beyer et al., 2021).
Recently, there has been a growing trend use prompting to evaluate L.LMs' capabilities. Prompt- ing (popularized by Brown et al., 2020) enables end-to-end interaction with models through natural language, and can be done entirely through infer- ence (i.e., without gradient updates). This method has revealed new classes of emergent abilities in LLMs, such arithmetic, instruction-follooiing and grounded conceptual mappings (Brown et al., 2020; Wei al., 2022a; Patel and Pavlick, 2022), well as the ability to render sentence acceptability judgments (Dentella et al., 2023). From a sociolog- ical angle, prompting has also made LLM evalua- tions more accessible for domain experts in linguis- tics and cognitive science, sparking new discussion about the limitations of LLMs' abilities (Katzir, 2023; Dentella et al., 2023; Ullman, 2023). For ex- ample, Dentella al. (2023) prompt GPT-3 to pro- duce grammaticality judgments of infrequent lin- guistic constructions, and conclude that the model "show[s] a critical lack of understanding even of
1 Introduction
Few technologics have been as exciting - and di- visive for language science as large language models (LLMs). LLMs are capable of incredibly sophisticated linguistic behaviors, which emerge through statistical learning with massive amounts of text data and highly expressive, domain-agnostie learning architectures. On the one hand, the suc- cess of these models has sparked growing move- ment to treat them as candidate models of human language acquisition and processing (c.g., Baroni, 2022; Warstadt and Bowman, 2022; Wilcox et al., 2022; Contreras Kallens et al., 2023) indeed, Piantadosi (2023) even claims that they "refute" Chomsky's approach to language. On the other hand, linguists have highlighted shortcomings of
Code and data are available at https://gituh..om/jmmii mntalinguistii-promping..

---

high-frequency words" (p. 1). Similarly, Katzir (2023) argues that "LLMs are poor theories of hu- man linguistic cognition" (p. 2) based on prompting ChatGPT to compare the well-formedness of two English sentences.
on this theoretical debate. Rather, our goal is evaluate the validity of metalinguistic prompting as a way of measuring LLMs' internal knowledge. We pose two research questions: (1) How well do models perform under direct and metalinguistic evaluation methods? and (2) How consistent are the metalinguistic methods with the direct method? We investigate these questions through four exper- iments, covering range of tasks and linguistic domains. Our findings (and their supporting fig- ures) are summarized below:
As these examples illustrate, researchers have been making substantial theoretical claims based on prompting. However, there is an important caveat to using prompts to evaluate models' lin- guistic knowledge. Prompt-based methods test not only whether a model represents the generalization of interest (e.g.. a certain ordering of probabilities), but also whether the model report the outcome of applying the generalization the sentence in the prompt. In this way. prompting implicitly tests a new type of emergent ability - metalinguistic judgment- - which has not yet been systematically explored as a way to evaluate model capabilities.
1. The metalinguistic judgments elicited from LLMs through prompting are not the same quantities directly derived from model repre- sentations. (Figures 2 and 3)
2. Direct probability measurements generally yield better or similar task performance, com- pared to metalinguistic prompting. (Figure 2)
To demonstrate the difference between di- rect probability measurements and metalinguistic prompting, consider the case of English subject- verb agreement. A direct approach might compare the probability assigned by the model to singular and plural verbs, given particular subject noun phrase (e.g., Linzen et al., 2016). For example, we might compare P("is") and P("are"). conditioned on the prefix given by (1-a). In contrast, prompt- ing approach might present a sentence prefix, and pose a question about this linguistic content, For example, we might compare P("is") and P""aree",, conditioned on the prompt in (1-b).
Minimal pairs help reveal models' general- ization capacities, compared to isolated judg- ments. (Figure 2c vs. Figure 2d)
4. In general, the less similar the task/prompt is to direct probability measurement, the worse the alignment between metalinguistic and direct measurements. (Figure 3)
Taken together, our findings suggest that nega- tive results relying on metalinguistic prompts can- not be taken conclusive evidence that an LLM lacks a particular linguístic generalization. These findings suggest a possible basis for a competence- performance distinction in LLMs: namely, the distinction between the information encoded in a model's isolated-sentence string probability distri- bution versus the model's 's behavioral responses to prompts. We discuss this topic in greater detail in Section 6.1.
(1) a. The keys to the cabinet
b. Here is sentence: The keys to the cabinet... What word is most likely to come next?
A model that perfectly performs the metalinguistic task posed in prompt (1-b) should have identical probability distributions over the next word given (I-a) and (1-b). However, there is no guarantee that model's response to metalinguistic prompt will match its underlying internal representations. In light of this, how should we interpret models' responses to metalinguistic prompts? How do these responses correspond to models' internal represen- tations? And when should we use metalinguistic prompts as opposed to direct measurements? These questions are becoming increasingly important, as prompting plays growing role in the debate about LLMs models of human language processing.
Our results also highlight the value that is lost as researchers move toward interacting with LLMs through closed APIs, where access to models' un- derlying probability distributions is limited. Indeed, only two days before acceptance of this paper to EMNLP, the ability to obtain arbitrary token log- probabilities from gpt-3. 55.55turrm innrru- was removed from the OpenAI API, reinforcing the timeliness of the issue. We urge future research to clearly motivate the evaluation methods used assess [.LM abilities, and highlight the importance of developing and using open- source models with access to internal probabilities.
In this study, we do not intend to take a stance

---

Experiment
Targeted ability
Task
Dataset(s)
1 (Section 4.1)
Word prediction
Predict final word sentence
Pereira al. (2018); arti- cles from March 2023
2 (Section 4.2)
Semantic plausibility
Determine which word (of two options) is most likely, given preceding context
Vassallo et al. (2018)
3a (Section 4.3)
Syntax
Determine which sentence (of two op- tions) is "better", in isolation
SyntaxCiym (Hu et al., 2020);
BI LMM (Warstadt et al,, 2020)
3b (Section 4.4)
Syntax
Determine which sentence (of two op- tions) is "better", given both options
SyntaxCiym (Hu et al., 2020);
BLiMP (Warstadt et al., 2020)
Table I: Overview of experiments in our study.
2
Related work
judgments about linguistic objects given linguistic input. In contrast, our study investigates whether models can access and report their internal proba- bility distributions through linguistic prompts.
Our study is related to model calibration, or the problem of estimating predictive uncertainty (e.g.. Guo et al., 2017; Minderer et al., 2021; Kadavath al., 2022; Mielke et al., 2022). Most relevant to our work, Kadavath et al. (2022) perform a broad- scale evaluation of "honesty" in LMs. In particu- lar, they analyze models' "truthfulness" and "self- knowledge", which are conceptually similar to our ground-truth and internal consistency evaluations. Similarly, Mielke et al. (2022) find that chatbots' expressions of confidence and doubt are poorly calibrated with the likelihood that the models' re- sponses are actually correct. These approaches dif- fer from ours in that they analyze models' metacog- nition by annotating verbalized expressions (e.g.. "I don't know, but..."), or by using models to evaluate the correctness of their own generated answers.
3 General methods
3.1 Overview of tasks
Our experiments feature four tasks, summarized in Table 1. Together, the tasks cover both word- and sentence-level computations, as well both isolated judgments and minimal-paii comparisons. Since we aim to analyze the validity of using met- alinguistic prompts to reveal linguistic knowledge, the experiments also cover semantic plausibility and syntax as linguistic domains of interest.
As mentioned in Section 1, word prediction (Expt. 1) a fundamental task for LLMs, and in- volves the most straightforward operation on mod- els' representations: reading out the logits over the vocabulary in the output layer. Because we know that models represent can treat the direct probability measurements providing ground-truth values. We therefore treat word prediction as baseline task, where we can perform a tightly controlled comparison be- tween probability measurements and metalinguistic prompting. The other three tasks induce an intu- itive (partial) ordering in terms of their similarity to the baseline word prediction task (word compari- son, sentence judgment, sentence comparison). We return this ordering in Section 5.2.
Other studies have also demonstrated that mod- els can respond prompts in unexpected ways (Khashabi et al., 2022; Min al., 2022; Webson and Pavlick, 2022: Webson et al., 2023; Prasad et al., 2023), in some cases achieving successful outcomes even when prompts are misleading or incoherent. Similarly, Turpin et al. (2023) find that explanations elicited through chain-of of-thought prompting (Nye et al., 2021; Wei et al., 2022b) can systematically misrepresent the underlying reasons and causes for a model's prediction. McCoy al. (2023) also demonstrate how LLMs' responses to prompts are highly sensitive to word probabilities, due to their original training objectives.
we
Beguš et al. (2023) also evaluate LLMs' "met- alinguistic" abilities, but in different sense of the word. They investigate whether LLMs can perform theoretical analyses of the structures and regulari- ties of linguistic expressions for example, draw- ing the syntactic tree diagram of a given sentence in valid LTTX code. Beguš et al.'s 's tested abilities are metalinguistic in the sense that they require
3.2 Overview of prompts
In each experiment, evaluate models using four methods: direct method, and three zero-shot met- alinguistic prompting methods. The direct method involves computing probabilities of tokens or full sentences based on the models' internal logits over vocabulary items. contrast, the metalinguistic

---

Direct
The Flan-T5 models have nncoder architecture and were pre-trained on span cor- ruption task before fine-tuning on large collec- tion of instruction-ban word probabilities under these models, we take the sentence prefix (W1, w2...., wx-1) 102....,,x-11. (where Wi is et al., 2022),² and three GPT-3/3.5 models (text- curic-001/GPT-3, text-ddvini--002/G1--3 text- davinci-003//PT--33.)) Flan-T5 models were ac- cessed through Huggingface, and GPT-3/3.5 mod- els were accessed through the OpenAl API.
VOcaó distnbbtion
A butterfly is flying insect with four large
!|?
Metalinguistic
Here is a sentence: A butterfly is flying insect with four large.. What word is most likely to come next?
vocab distnbution
tasks. To measure next-
(a)
Direct
the word to be predicted) and append the sentinel token cextra_id__ We then create the output sequence textra_id_@> Wn: and sum the prob- abilities corresponding the tokens of Wn- To measure full-sentence probabilities, we compute pseudo-likelihood inspired by Salazar et al. 's (2020) method for scoring sentences under masked language models. Going left to right, we mask out each word in the sentence using the T5 sentinel tokens, and then sum the log probabilities assigned to each true word.
S
Every child has studied.
P(S.)
P(S))
P(S)}
2 Every child have studied.
P(S))
Metalinguistic
Here are two English
sontences: 1) Every
stdied. 2) Every thild studied. Which sentence is better Engish serrence? Respond either 2 your
VO06O distribution
('1")"
While the Flan-T5 models differ only in size, the OpenAl models also differ training regime: text- curie-001 an autoregressive language model at its core (GPT-3; Brown et al., 2020), whereas text- davinci-002 has additional supervised fine-tuning, and text-davinci-005 has additional reinforcement learning (Ouyang et al., 2022).
(b)
Figure 1: Conceptual illustration of direct probability measurements vs. metalinguistic judgments. (a) Ba-
sic word prediction task (Experiment Section 4.1). (b) Sentence comparison task (Experiment 3b, Section 4.4).
Details of experiments
prompts ask a question or specify task requiring a judgment about a linguistic expression.
4.1 Experiment 1:
Word prediction
Taking direct probability measurements our baseline evaluation method, also identify or- dering of the three metalinguistic methods in terms of their similarity baseline. In the MetaQues- tionSimple and MetaQuestionComple prompts, the linguistic object of interest (c.g., a sentence prefix) is linearly closest and farthest from the po- sition where the model is asked to make a predic- tion, respectively. The Metalnstruct prompts are structured as imperative instruction, and fall in between the two MetaQuestion prompts. Tables 2- 4b show example prompts for all experiments."
Task and stimuli. The aim of this experiment is to evaluate models' ability to predict the next word given a preceding context. Instead of a standard language modeling task, where models are evalu- ated on their ability to predict every word in a text, the simplified task of predicting the final word of a sentence. This makes the metalinguis- tic evaluations more tractable, as we only need construct single prompt for each item.
We use two datasets with contrasting style. Our first dataset, taken from Pereira et al. (2018) ("P18"), consists of 384 simple declarative sen- tences that state fact about familiar concepts, such
3.3 Models
We test six models across all of our experiments: three F1an-T5 models (small, large, XL; Chung
PParaeeter counts: small 80M, large 780M, XI. 3B.
'Many LLM evaluations use ChatGPT (Piantadosi, 2023; Katzir, 2023) or GPT-4 (Beguš et al., 2023; Webb et al., 2023: Moskvichev et al., 2023). However, we exclude these models from our analyses hecause the OpenAl API does not provide access to token probabilities for chat-hased models.
'To test the gencralizability of our findings beyond English, we also conducted preliminary experiments in Mandarin Chi- nese. Details and results can be found in Appendix D.

---

Type of prompt
Example
Direct MetaQuestionSimple
A butterfly is flying insect with four large wings
What word most likely to come next in the following sentence? A butterfly is flying insect with four large wings You a helpful writing assistant. Tell what word most likely come the following sentence: A butterfly is flying insect with four large wings Hore the beginning
Mctalnstruct
MctaauestionComplee
English sentence: butterfly is a flying insect with four large... What the best next word? Answer: wings
Table 2: Example prompts for Experiment 1, Region where we measure probability is marked in boldface. Ground- truth sentence continuations shown in blue.
Type of prompt
Example
Direct MetaQuestionSimple
The archer released the [arrow, interview]
What word is most likely to come next in the following sentence (arrow, interview)? The archer released the (arrow, interview) You a helpful writing assistant. Tell what word most likely come next in the following sentence (arrow, interview?): The archer released the (arrow, interview] Here is the beginning of an Fnglish sentence: The archer released the... What word more likely come next: arrow, or interview? Answer: [arrow, interview]
Metalnstruct
MctaQuestionComplee
Table 3: Example prompts for Experiment 2. Region where we measure probability marked in boldface. Semantically plausible continuations shown in blue; implausible in red.
as accordion or butterfly. All pronouns are derefer- enced, making the dataset useful for testing predic- tion in simple contexts with minimal dependencies. There are at least two concerns with the PI8 sentences. First, their simple structure might not be representative of the text that models encounter during training. Second, they have been publicly available since 2018, making it possible that they may be in the models' training data. To address these concerns, we constructed a second dataset ("News") containing sentences that are more natu- ralistic, but highly unlikely to occur in the training data. We used the NewsData tool' to find English articles published in the United States in the date range of March 20-26, 2023.5 The articles cover a span of topics, such as business, politics, and food. For each article, we construct a prefix by concatenating the headline with the first sentence (up to, but not including, the last word), separated by the string "". There are 222 items in total.
in blue boldface). The other prompts are designed to elicit metalinguistic judgments, through ques- tions (MetaquestionSinple, MetaQuestionCom. plex) and instructions (Metalnstruct). As concep- tual illustration, Figure la shows a comparison of the Direct and MetaQuestionComples methods.
Evaluation.
Our measure of task performance is the log probability assigned by each model the ground-truth sentence continuation. To measure internal consistency (see Section 5.2), we analyzed the relationship between log probabilities assigned to ground-truth continuations, as measured by the direct method and each metalinguistic method.
4.2 Experiment 2:
Word comparison
Task and stimuli. The aim of this experiment is to evaluate models' ability judge which of two words is more likely continuation of sentence. While Experiment tested word prediction with- out focus on any particular linguistic phenomenon, here we use the word-com compproopa task to assess knowledge of semantic plausibility.
Prompts. Example prompts are shown in Table (only examples from the P18 corpus are shown, for simplicity). The Direct prompt feeds the sentence prefix to the model, and we measure the model's probability of the ground-truth next word (indicated
We use a set of 395 minimal sentence pairs from Vassallo et al. (2018). Each pair consists of two sentences that differ only in the final word, which alters the plausibility of the described event (e.g., "The archer released the arrowinterview"), Each sentence has a simple syntactic structure.
'htps://newsdatt.io/
"These are unlikely to be in the Flan-T5 training data, as the models were publicly released in 2022. The OpenAl model documentation also states that text-curic-001 only re- ccived training data up to October 2019, and the text-davinci- models only received training data up to June 2021.
Prompts. The prompts are similar to those from Experiment 1, but here we ask models to make a

---

Type of prompt
Example
Direct MetaQuustionSimple
(Every child has studied, Every child have studied] Is the following sentence a good sentence of English? Every child has studied. Respond with either Yes No your answer. (Yes, No}
Mctalnstruct
You are a helpful writing assistant. Tell if the following sentence is good sentence of English. Every child has studied. Respond with either Yes or No as your answer. (Yes, No) Here sentence: Every child studied. Is the sentence good sentence of English? Respond with either Yes or No as your answer. Answer: (Yes, No}
MctaaucstionComplec
Type of prompt
Example
Direct MetaQuestionSimple
[Every child has studied, Every child have studied]
Which sentence is a better English sentence? 1) Every child has studied. 2) Every child have studied. Respond with either or as your answer. 21
Metalnstruct
You are helpful writing assistant. Tell me which sentence is better English sentence. 1) Every child has studied. 2) Every child have studied. Respond with either 1 or 2 as your answer. (1. 2] Here are two English sentences: 1) Every child have studied. 2) Every child has studied. Which sentence better English sentence? Respond with either or 2 your answer. Answer: (1. 2]
MetaQuestionComllex
Table Example prompts for Experiments 3a (a) and 3b (b). Region where we measure probability is marked boldface. Grammatical sentences and correct answer options shown in blue; ungrammatical//noomcc in red.
comparison between two potential continuations of the sentence prefix (Table 3). For the Direct method, we present the model with the shared sen- tence prefix, and compare the probability of the plausible (e.g., "arrow"; indicated in blue) and im- plausible (c.g., "interview"; indicated in red) con- tinuations. For the Meta prompts, we create two versions of the prompt by shuffling the order in which the answer options are presented.
We use minimal pairs from two datasets de- signed to test knowledge of English syntax: Syn- taxGym (Hu et al., 2020; Gauthier et al., 2020), and the Benchmark of Linguistic Minimal Pairs (BLiMP; Warstadt et al., 2020). Since SyntaxGym was not designed for full-sentenee probability com- parisons, first converted the SyntaxGym mate- rials into sentence-level minimal pairs. We then took random sample of 15 items from each of the 23 remaining suites, resulting in 345 items. For BLiMP, we extracted the items that were compat- ible with the "simple LM method" and then took random sample of 30 items from each of the 13 categories, resulting in 390 items. See Appendix A for details the tested phenomena.
Evaluation. Accuracy is the proportion of items where the model assigns higher probability to the plausible sentence continuation than to the implau- sible continuation. Random performance is 50%.
To measure internal consistency, for each evalu- ation method we computed the item-level log prob- ability differentials between the plausible and im- plausible sentence continuations. We then com- puted the correlation between the differentials elicited by the direct method and the differentials elicited by each metalinguistic method.
Prompts. For the Direct method, we measure the probability of each sentence in the minimal pair. For the Meta prompts, we construct a separate prompt for each sentence in the minimal pair asking whether the sentence is "a good sentence of English" (see Table 4a), and then compare the probability assigned by the model to "Yes" versus "No".
4.3 Experiment 3a: Sentence judgment
Evaluation. To measure accuracy for the direct condition, we compute the proportion of items where the model assigns higher probability to the grammatical sentence in the minimal pair. For
Task and stimuli. The aim of this experiment is to evaluate models' ability to judge whether sentence is "good" sentence of English. For any particular sentence, we compare the model's judg- ment of that sentence to the model's judgment of another sentence that forms minimal pair, only differing in a critical syntactic feature that manipu- lates grammaticality or acceptability.
"We created full sentences by combining content across regions, and then turned each success criterion inequality into a minimal pair (see Hu et al., 2020, for details). For simplic- ity, we omitted test suites with success criteria involving the conjunction of 2 3 incqualities, or probability differentials.

---

the metalinguistic prompts, we report balanced ac- curacy, or the mean of the true positive rate and true negative rate. A true positive occurs when the model assigns higher probability to "Yes" than "No" for grammatical sentence, and a true negative oc- curs when the model assigns higher probability to "No" than "Yes" for an ungrammatical sentence. To measure internal consistency, we compare the log probability differentials for each method. For the direct method, the differential is the difference in log probability of the grammatical and ungram- matical sentences. For each metalinguistic method, the differential is the difference in log probability of the "Yes" token conditioned on the grammatical and ungrammatical sentence prompts.
direct method) the log probability differentials between the answer options corresponding the grammatical and ungrammatical sentences (mea- sured by each metalinguistic prompting method).
Results
We now return to our main research questions, laid out in the Introduction: (1) How well does each evaluation method perform on each task? (2) How consistent are the metalinguistic evaluation meth- ods with the direct evaluation method? We address these Sections 5.1 and 5.2, respectively.
5.1 Task performance
Result #1: Metalinguistic judgments not the same as direct measurements, Figure 2 shows task performance for each experiment, At the coars- est level, the different methods (hues) yield differ- ent performance scores, demonstrating that met- alinguistic and direct responses are not identical.
4.4 Experiment 3b:
Sentence comparison
Task and stimuli. As in Experiment 3a (Sec- tion 4.3), the goal is to measure models' syntactic judgments. However, instead of presenting the model with sentences in isolation and asking for judgments, in Experiment 3b we present the model with the minimal pair of sentences, and probe which sentence takes to be a "better" sentence of English. For our stimuli, we use the same subsets of SyntaxGym and BLiMP as in Experiment 3a.
Result #2: Direct measurements generally per- form > metalinguistic methods. Aross all ex- periments, the direct method nearly always yields best performance of all tested methods. There are few exceptions: in Experiment 1, T5-SM performs best under Metalnstruct, and Flan-T5-XL performs relatively well under MetaQuestionCom- plex, as do the davinci models in Experiment 2.
Prompts. The direct evaluation method is the same as in Experiment 3a: we compare probabili- ties of each sentence in the minimal pair. For the metalinguistie prompts, we have a single prompt for each minimal pair that presents both sentences at once. We assign an identifier (1 or 2) to each sentence in the pair, present multiple-choice ques- tion comparing both sentences, and compare the probabilities assigned by the model to each answer option (i.e., "1" or "2"). As in Experiment 2, we av- erage model results over two versions of the prompt that counterbalance the order of answer options (for metalinguistic prompts). Example prompts are shown in Table 4b. As a conceptual illustration, Figure 1b shows comparison of the Direct and MetaQuestionComplex methods.
Result #3: Minimal pairs help reveal models' generalization capacities. The difference be- tween Experiments 3a and 3b lies in the presen- tation of minimal pairs. Comparing Figures 2c and 2d, we first note that the direct results (darkest bars) are identical by definition: they reflect com- parisons of full-sentence probabilities. For the met- alinguistic prompts, there is an increase in accuracy going from the isolated sentence judgments (Fig- ure 2c) minimal -airr comparisons (Figure 2d), for all models with above-chance performance.
5.2 Internal consistency
Evaluation. Accuracy is measured as the pro- portion of items where the model assigns higher probability to the grammatical sentence in the min- imal pair (direct method), or to the answer option corresponding the grammatical sentence (met- alinguistic prompts). Random performance is 50%.
Result #4: Consistency gets worse as we get further from direct measurement of noxt-word probabilities. Figure 3 illustrates alignment be- tween direct and metalinguistic measurements (see Appendix Figure for by-model internal consis- tency results). Each cell shows the average corre- lation coefficient (Pearson's r) between the item- level differentials measured by the direct method
To measure internal consistency, we compare the log probability differentials between the grammati- cal and ungrammatical sentences (measured by the

---

Direct
MetaquesttonSimple
Metalnstruct
MetaQuestionComplex
1.0
0.0
Logprob of final word
0.8
-2.5
Accuracy
-5.0
-7.5
0.4
-10.0
0.2
0.0
Flan-T5 SM
Flan LS LG
Flan-T5 Xt.
tent teut-curie-0e 0011
text sed--svinct-002
text ded--davinc--033
Flan-T5 SM
Flan-TS LG
Flan-T5 XL
text text-curie-001
ted cet--dainci-0022
next-davinck003
(a) Experiment 1: Word prediction
(b) Experiment 2: Word comparison (Semantic plausibility)
1.0
1.0
Accuracy
Accuracy
0.6-
0.6
0.4
0.4
0.2
0.2
0.0
0.0
Flan- T5 SM
Flan-TSL LG
Flan-T5 XI.
text tat-davinn--022 uev--ciro-001
eett-davinc-003
Flan- T5 SM
Flan-T5 Rlan-TS LG
xl.
text tat-davinci-002 text-curi-0011
eext-davinc-003
(c) Experiment 3a: Sentence judgment (Syntax)
(d) Experiment 3b: Sentence comparison (Syntax)
Figure 2: Task performance: Direct probability measurements generally outperform metalinguistic prompts. (a) Log probability assigned to ground-truth sentence continuation, averaged over items and datasets. (b) Proportion of items where model prefers semantically plausible continuation over implausible continuation. (c)-(d) Proportion of items where model prefers grammatical sentence over ungrammatical sentence in minimal pair, averaged over datasets. Error bars denote bootstrapped 95% Cls. Dashed lines indicate random baseline.
and a particular metalinguistic prompting method." Recall from Section 3 that the tasks and prompts in our study induce intuitive orderings in terms of how similar they are word prediction (at the task- level) and direct probability measurements (at the prompt-level). The columns of Figure (prompt types) are loosely ordered by similarity to direct probability measurements, and the rows (tasks) are loosely ordered by similarity to word prediction. Broadly speaking, find that as distance in either dimension increases, the correlations get weaker. These results further support Result #1: while di- rect and metalinguistic responses are highly corre- lated for some combinations of tasks and prompts. the relationship is far from perfect.
LLMs' linguistic knowledge, Broadly, we find that metalinguistic judgments are inferior to direct mea- surements of token- sentence-leved probabilities. We also find evidence that minimal-pair compar- isons help reveal models' generalization capacities.
We do not intend to claim that prompting should be categorically dispreferred in favor of other eval- uation methods. Prompting is useful for eliciting open-ended responses, such as chain-of-thought reasoning. Metalinguistic prompting could also be used to ask questions that would be challenging to translate into direct probability measurements (c.g.. "Which of the following two sentences is more se- mantically plausible, but less syntactically well- formed?"). In addition, prompting lowers the tech- nical barrier for domain experts to conduct LLM evaluations, which could contribute to the develop- ment more robust behavioral benchmarks.
6 Discussion
In this study, compared metalinguistic prompt- ing and direct measurements as ways of evaluating
What do our findings mean for researchers in- terested in the linguistic abilities of LLMs? While there are valid reasons to prefer both prompting
For Experiment 1, the correlation is computed between the item-level prohabilities of the ground-truth final word.

---

has been a topic of debate. Katzir (2023) argues that "[LLMs") behavior directly reflects their com- petence, and when they fail it is their competence that is at fault" (pp. 4-5). We take Katzir to mean that an LLM's next-word probability distribution is deterministic given its architecture, weights, and the preceding context, and this probability distri- bution is always computed when the LLM is used. Therefore, as Katzir says, while humans may re- cover from linguistic errors (e.g., in agreement or parsing) given additional time or resources, "fur- ther time and resources are no use" to LLMs. Firestone (2020) and Lampinen (2023), in contrast, argue that performance conditions need to be care- fully controlled in both humans and machines in order to make fair comparisons between the two.? Our work suggests that if competence- performance distinction is to be made for LLMs, a natural locus is the contrast between the informa- tion contained in an LLM's s string probability distri- bution (corresponding to its competence) versus the behavior the L.M exhibits when prompted (corre- sponding to its performance). A model's failure exhibit linguistic generalization when prompted might not reflect a lack of the relevant information in its underlying conditional probability distribu- tions, but instead an inability to access and behave in accordance with that information in response to prompt that poses a metalinguistic query. This view remains consistent with the fact that LLM be- havioral errors may be corrected when illustrative examples included in the prompt whether not such prompts pose metalinguistic queries, they offer opportunity for in-context learning (Brown et al., 2020) - or by allowing the model to produce a reasoning trace before outputting an answer (Nye et al., 2021; Wei et al., 2022b; Kojima et al., 2022). "competence" and "performance" in an Al model
Sentence comparison
1.0
(Exp 3b)
0.23
0.24
0.22
More distant task
Sentence judgment (Fxp 3a)
0.2
0.2
0.2
comparison (Exp
0.38
0.5
prediction (Exp 1)
0.78
-1.0
Meta Question Instruct
Mata
Question Complex
Simple
More distant prompt
Figure 3: Internal consistency: Correlation between metalinguistic and direct responses gets weaker as prompts become less direct. Pearson correlation be- tween response magnitudes (averaged over models and datasets) measured by direct prompts versus each met- alinguistic prompt. See Appendix C for more details.
and direct probability measurements, these meth- ods will not necessarily generate consistent results. More specifically, our findings suggest that nega- tive results relying on metalinguistic prompts can- not be taken as conclusive.
While our paper focuses on the value of direct probability measurements for evaluating linguistic generalizations, other endeavors in cognitive sci- ence and machine learning also rely access to LLM probabilities. For example, token probabili- ties enable multiplechoice evaluation and beam search, as well as investigating mode collapse (Janus, 2022) and obtaining quantities for Bayesian inference (e.g.. Choi et al., 2022; Li et al., 2023; Lipkin et al., 2023). 8 Thus, the value that is lost we move toward closed APIs extends beyond linguistic analysis, further underscoring the impor- tance of using and developing LLMs with open access internal probabilities.
To conclude, prompting is not substitute for direct probability measurements in LLMs. We un- derscore the importance of specifying the assump- tions underlying methodological choices LLM evaluation, and using open models with direct ac- cess to probabilities for scientific research. our interactions with LLMs are limited to high-level prompting, we lose the opportunity to measure ca- pabilities that could advance our understanding of these models and their relation to human language.
6.1 Competence vs. performance in LLMs
Humans use language in diverse contexts with vary- ing task demands and constraints, but the underly- ing linguistic knowledge relatively stable. This is the Yngve, 1960; Chomsky, 1965): an individual's performance in a particular context may not re- flect an individual's full underlying competence. Whether it is productive distinguish between
distinction (e.g.,
"Hahn et al. (2022), for example, show that imposing per- formance constraints on LLMs (by degrading context repre- sentations) derives human language processing behavior in complex nested dependency constructions.
"These examples were inspired by discussion by Tan Zhi Xuan on Twitter http::/twwittr.co//uuaangggggge 777305007399847699

---

Limitations
Acknowledgements
We thank Jon Gauthier and the anonymous re- viewers for feedback earlier versions of this work, as well as Peng Qian for translating the prompts into Chinese J.H. gratefully acknowl- edges support from an NSF Graduate Research Fellowship (# 745302), an NSF Doctoral Disserta- tion Research Improvement Grant 1661 8), and the Simons Center for the Social Brain at MIT, R.P.L. gratefully acknowledges support from NIH grant U01-NS121471, Newton Brain Science award, and the Simons Center for the Social Brain at MIT.
Our experiments only test three types of metalin- guistic prompts, and only perform zero-shot eval- uations. In practice, the small number of metalin- guistic prompt types was sufficient to illustrate the difference between direct and metalinguistic re- sponses. However, it would be beneficial to con- sider more types of prompts to determine how well the phenomenon generalizes. We also note that models might achieve better task performance under the metalinguistic prompts with few-shot prompting or in-context learning. We did not in- clude few-shot analyses due to space limitations, and because many recent LLM evaluations rely on zero-shot metalinguistic prompts. It remains to be seen whether metalinguístic and direct responses are better aligned when models have access ex- amples the prompt.
References
Marco Baroni. 2022. On the proper role of linguistically-ooreneed deep net analysis in linguis- tic theorizing. In Shalom Lappin and Jean-Philippe Bernardy, editors, Algebraic Structures in Natural Language. Taylor & Francis.
Another limitation of our study is that we only tested small class of models. An important direc- tion for future work would be to replicate our ex- periments models of different sizes and training objectives (e.g., chat-based models). We also note that the results from the OpenAI models are not necessarily reproducible due to the models being behind a closed API. Timestamps of our calls to the OpenAl API are available in our data files (https: /ggituu.coom/cnntt//naaaaggggggt pronsticc-pomppinn.....
Gašper BeguR, Maksymilian Dąbkowski, and Ryan Rhodes. 2023. Large Linguistic Models: Analyz- ing theoretical linguistic abilities of LLMs.
Anne Beyer, Sharid Loáiciga, and David Schlangen. 2021. Is Incoherence Surprising? Targeted Evalua- tion of Coherence Prediction from Language Models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 4164-4173, Online. Association for Computa- tional Linguistics.
Ethics Statement
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin. Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Tlya Sutskever, and Dario Amodei, 2020. Language Models are Few-Shot Learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
This work does not release any new models or datasets. Instead, the goal is to provide insights into methodology for evaluating the internal knowl- edge of modern LLMs, and in turn contribute to the interpretability of these models. We hope that our results illustrate the importance of open access to model representations and the risks of relying on high-level API interactions for scientific research.
With that said, the broader ethical concerns about LLMS are still relevant to our work. LLMS have been shown to produce output that is factually incorrect, offensive, or discriminatory, and should therefore be used with extreme caution, especially in commercial applications or user-facing settings. Any demonstrations of LLMs' linguistic general- izations should not imply that they are safe use, or can be expected to behave in a way that is aligned with human preferences and values.
Kristy Choi, Chris Cundy, Sanjari Srivastava, and Ste- fano Ermon. 2022, LMPriors: Pre- Preined Language Models as Task-Specific Priors.
Noam Chomsky, 1965, Aspects the MIT Press.
the Theory of Syntax.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang,

---

Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu, Zhuyun Dai. Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022. Scaling Cnoruction-Finetunee Language Mod- els.
1725-1744, Online. Association for Computational Linguistics.
Janus. 2022. Mysteries of mode collapse.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield- Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hemaande,, Josh Jacohson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Ka- mal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. 2022. Language Models (Mostly) Know What They Know
Pablo Contreras Kallens, Ross Deans Kristensen- McLachlan, and Morten H. Christiansen. 2023. Large Language Models Demonstrate the Potential of Statistical Learning in Language. Cognitive Science, 13256. Publisher: John Wiley & Sons, Ltd.
Vittoria Dentella, Elliot Murphy, Gary Marcus, and Evelina Leivada. 2023. Testing Al performance on less frequent aspects of language reveals insensitivity to underlying meaning.
Roni Katzir. 2023. Why large language models are poor theories of human linguistic cognition: A reply to Piantadosi (2023).
Gabe Dupre. 2021. (What) Can Deep Learning Con- tribute Theoretical Linguisties? Minds and Ma- chines, (1(4):617-635.
Carina Kauf, Anna A. Ivanova, Giulia Rambelli, Em- manuele Chersoni, Jingyuan S. She, Zawad Chowd- hury, Evelina Fedorenko, and Alessandro Lenci. 2022. Event knowledge in large language models: the gap hetween the impossible and the unlikely.
Chaz Firestone. 2020. Performance vs. competence in human-machine comparisons. Proceedings of the National Academy of Sciences, 117(43):26562 26571. Publisher: Proceedings of the National Academy of Sciences.
Daniel Khashabi, Xinxi Lyu, Sewon Min. Lianhui Qin, Kyle Richardson, Sean Welleck, Hannaneh Ha- jishirzi, Tushar Khot, Ashish Sabharwal, Sameer Singh, and Yejin Choi. 2022. Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Prompts. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3631-3643, Seattle, United States. Association for Computational Lin- guistics.
Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. 2019. Neural language models psycholinguistie subjects: Representations syntactie state. Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers). pages 32-42. Minneapolis, Minnesota. Association for Computational Linguistics.
Jon Gauthier, Jennifer Hu, Ethan Wilcox, Peng Qian, and Roger Levy. 2020. SyntaxGym: An Online Plat- form for Targeted Evaluation of Language Models. Praceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 70-76, Online. Association for Computational Linguistics.
Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large Language Models are Zero-Shot Reasoners. In Ad- vances in Neural Information Processing Systems, volume 35, pages 22199-22111 22213. Curran Associates. Inc.
Andrew Kyle Lampinen. 2023. Can language models handle recursively nested grammatical structures? A case study on comparing models and humans.
Chuan Guo. Geoff Pleiss, Yu Sun, and Kilian Q Wein- berger. 2017. On Calibration of Modern Neural Networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pages 1321-1330.
Nur Lan, Emmanuel Chemla, and Roni Katzir, 2022.
Large Language Models and the Argument From the Poverty of the Stimulus.
Michael Hahn, Richard Futrell, Roger P. Levy, and Ed- ward Gibson. 2022. A resource-rational model of human processing of recursive linguistic structure. 119(43).
Belinda Z. Li, William Chen, Pratyusha Sharma, and Jacob Andreas. 2023, LaMPP: Language Models as Probabilistic Priors for Perception and Action.
Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox. and Roger Levy. 2020. A Systematic Assessment of Syntactic Generalization in Neural Language Mod- els. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies. Transactions of the Association for Computational Linguistics, 4:521- 535.

---

Benjamin Lipkin, Lionel Wong, Gabriel Grand, and Joshua B. Tenenbaum. 2023. Evaluating statistical language models as pragmatic reasoners. In Proceed- ings of the Annual Meeting of the Cognitive Science Society, volume 45.
Francisco Pereira, Bin Lou, Brianna Pritchett, Samuel Ritter, Samuel J. Gershman, Nancy Kanwisher. Matthew Botvinick, and Evelina Fedorenko. 2018. Toward a universal decoder of linguistic meaning from brain activation. Nature Communications, 9() (1)9963.
R. Thomas McCoy, Shunyu Yao, Dan Friedman, Matthew Hardy, and Thomas I. Griffiths. 2023. Em- hers of Autoregression: Understanding Large Lan- guage Models Through the Problem They are Trained to Solve.
Steven T. Piantadosi. 2023. Modern language models refute Chomsky's ' approach to language.
Archiki Prasad, Peter Hase, Xiang Zhou, and Mohit Bansal. 2023. GrIPS: Gradient-fie free, Edit- based In- struction Search for Prompting Large Language Mod- cls. In Proceedings of the 17th Conference of the Eu- ropean Chapter of the Association for Computational Linguistics, pages 3845-3864, Dubrovnik, Croatia. Association for Computational Linguistics.
Sabrina J. Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. 2022. Reducing Conversational Agents' Overconfidence Through Linguistic Calibra- tion. Transactions of the Association for Computa- tional Linguistics, 10:857-872.
Daniel Milway, 2023. A Response Piantadosi (2023).
Julian Salazar, Davis Liang. Toan Q. Nguyen, and Ka- trin Kirchhoff. 2020. Masked Language Model Scor- ing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699-2712, Online. Association for Computational Linguistics.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettle- moyer: 2022. Rethinking the Role of Demonstrations: What Makes Ln-Context Learning Work? In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048-1100 1064, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. 2023. Language Models Don't Always Say What They Think: Unfaithful Explana- tions
Prompting.
Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlshy, Dustin Tran, and Mario Lucie. 2021. Revisiting the Cali- bration of Modern Neural Networks. In Advances in Neural Information Processing Systems, volume 34, pages 15694. Curran Associates, Inc.
Tomer Ullman. 2023. Large Language Models Fail Trivial Alterations to Theory- of---ind Tasks.
Paolo Vassallo, Emmanuele Chersoni, Enrico Santus, Alessandro Lenci, and Philippe Blache. 2018. Event Knowledge in Sentence Processing: A New Dataset for the Evaluation of Argument Typicality. In LREC 2018 Workshop on Linguistic and Neurocognirive Resources (LiNCR), Miyazaki, Japan.
Arseny Moskvichev, Victor Vikram Odouard, and Melanie Mitchell. 2023. The ConceptARC Bench- mark: Evaluating Understanding and Generalization in the ARC Domain.
Yiwen Wang, Jennifer Hu, Roger Levy, and Peng Qian. 2021. Controlled Evaluation of Grammatical Knowl- edge in Mandarin Chinese Language Models. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 5604- 5620, Online and Punta Cana, Dominican Republic. Association for Computational Linguisties.
Elliot Murphy. 2023. Notes on Large Language Models and Linguistic Theory.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieher, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2021. Show Your Work: Scratchpads for Intermedi- ate Computation with Language Models.
Alex Warstadt and Samuel R. Bowman. 2022. What artificial neural networks tell us about human language acquisition. In Shalom Lappin and Jean- Philippe Bernardy, editors, Algebraic Structures in Natural Language. Taylor & Francis.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback.
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mo- hananey. Wei Peng, Sheng Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The Benchmark Linguis- tic Minimal Pairs for English. Transactions of the Association for Computational Linguisties, 8. Pub- lisher: MIT Press.
Roma Patel and Ellie Pavlick. 2022. Mapping Language Models to Grounded Conceptual Spaces. In Interna- tional Conference on Learning Representations.
Taylor Webb, Keith J. Holyoak, and Hongjing Lu. 2023. Emergent Analogical Reasoning in Large Language Models.

---

Albert Webson, Alyssa Marie Loo, Qinan Yu, and Ellie Pavlick. 2023, Are Language Models Worse than Humans Following Prompts? It's Complicated.
Figures 7 to 10 show the relationship between sponses measured by the direct and metalinguistic prompting methods, for Experiments 1-3b, respec- tively.
Albert Webson and Ellie Pavlick. 2022. Do Prompt-
Based Models Really Understand the Meaning of Their Prompts? In Proceedings of the 2022 Con- ference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies, pages 2300-2344, Seattle, United States. Association for Computational Lin- guistics.
D Experiments in Mandarin Chinese
To explore the robustness of our results beyond En- glish, performed preliminary investigation of GPT-3.5 (1ext-davinii-003) on materials in Man- darin Chinese. We first consulted native speaker to translate the metalinguistic prompts into Chinese. We then tested GPT-3.5 on two datasets: (1) set of recent news articles for word prediction, mirroring Experiment 1; and (2) set of controlled minimal pairs that cover semantic and syntactic phenomena (Wang et al., 2021), mirroring a combination of the phenomena tested in Experiments 2 and 3. We tested the tasks Experiments 3a and 3b on this set minimal pairs.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud. Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi. Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022a. Emer- gent Abilities of Large Language Models. Transac- tions Machine Learning Research.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. 2022b. Chain of Thought Prompt- ing Elicits Reasoning in Large Language Models. In Advances in Newral Information Processing Systems.
Figure 11 shows the results (task performance) for Experiments I, 3a, and 3b in Chinese. Like our English experiments, in all our Chinese exper- iments we find that metalinguistic prompting and direct probability measurements do not yield the same results. Like our Experiment in English, we also find that in the Chinese word prediction task, GPT-3.5 assigns highest probability to the ground- truth continuation under the "Direct" method. We also find that model performance improves when using minimal pairs, mirroring our original findings comparing Experiments 3a and 3b. These findings demonstrate how English results may general- ize languages with different syntactic structures and grammatical properties.
Ethan Wileox, Jon Gauthier, Jennifer Hu, Peng Qian, and Roger P. Levy. 2022. Learning syntactic struc- tures from string imput. In Shalom Lappin and Jean- Philippe Bernardy, editors, Algebraic Structures in Natural Language. Taylor & Francis.
Victor H. Yngve. 1960. A Model and an Hypothesis for Language Structure. Proceedings of the American Philosephical Society, 104(5):444-466. Publisher: American Philosophical Society.
Details of syntactic phenomena
Table 5 summarizes the syntactic phenomena cov- ered in the datasets used in Experiments 3a and 3b, which were taken from SyntaxGym (Hu al., 2020; Gauthier et al., 2020) and BLiMP (Warstadt et al., 2020).
However, we also found differences between the Chinese and English results: in the Chinese version of Experiment 3b, the "Direct" method un- derperformed the metalinguistic methods (accuracy scores: Direct 0.6; others 0.8). One potential ex- planation for this is that according to the OpenAl documentation, the models are "optimized for use in English."" although in practice they may work well for other languages." 10 Therefore, the models may not be well-suited scoring probabilities of Chinese sentences with no context (as in the Direct condition); instead, they may benefit from seeing additional Chinese text in the prompt before the sentence (as in the metalinguistic conditions).
B Dataset-level task performance
Figure 4 shows task performance for each of the tested datasets in Experiment 1 (left: P18; right: News). Figure 5 shows task performance for each of the tested datasets in Experiments 3a and 3b (left: SyntaxGym; right: BLiMP).
Relationship between metalinguistic and direct predictions
Figure shows the average Pearson r correlations between metalinguistic and direct responses, for each the tested models.

---

Dataset
Phenomenon
# items
SyntaxGym
Center embedding
15
SyntaxGym
Center embedding (modifier)
15
SyntaxGym
Cleft
15
SyntaxGim
Cleft (modifier)
15
SyntaxGim
Filler-gap gap dependencies (3 sentential emheddings)
15
SyntaxGym
Filler-gal dependencies (4 sentential embeddings)
15
SyntaxGym
Filler-gap dependencies (hierarchy)
15
SyntaxGym
Filler-gap gap dependencies (object extraction)
15
SyntaxGym
Filler-gay gap dependencies (prepositional phrase extraction)
15
SyntuxGym
Filler-gap dependencies (subject extraction)
SyntaxGym
Subject-verb number agreement (object relative clause)
15
SyntaxGym
Subject-verb number agreement (prepositional phrase)
SyntaxGym
Suhject-verb number agreement (suhject relative clause)
SyntaxGym
Reflexive number agreement (object relative clause, feminine) Reflexive number agreement (object relative clause, masculine) Reflexive number agreement (prepositional phrase, feminine)
SyntaxGym
Syntax
SyntaxGym
Reflexive number agreement (prepositional phrase, masculine) Reflexive number agreement (subject relative clause, feminine) Reflexive number agreement (subject relative clause, masculine) Suhordination
SymtaxGym
SyntaxGym
SyntaxGim
SyntaxGym
Subordination (object relative clause)
SyntaxGym
Suhordination (prepositional phrase)
SyntaxGym
Subordination (subject relative clause)
15
BLiMP
Anaphor agreement
BLiMP
Argument structure
30
BLiMP
Binding
30
BLiMP
Control raising
30
BLiMP
Determiner-noun agreement
30
BLiMP
Ellipsis
30
BLiMP
Filler-gap dependency
30
BLIMP
Irregular forms
30
BLiMP
Island effects
30
BLiMP
NPI licensing
30
BLiMP
Quantifiers
30
BLiMP
S-selection
30
BLiMP
Subject-vert agreement
30
Table 5: Coverage of syntactic phenomena in stimuli used in Experiments 3a and 3b.
Direct
MetaQuestionSimple
Metalnstruct
MetaQuestionComplex
P18
News
0.0
Logprob of final word
-2.5
-5.0
-7.5
-10.0
-12.5
Flan-T5 SM
Flan-T5
Flan-T5 XL
text-curie-001
text-davinci-002
text- 00x-i
Flan-T5 SM
Flan-T5 LG
Flan-T5 XL
text-curie-Oo)
text-davinci-002
dexi-davinci-003
Figure 4: Task performance for each tested dataset in Experiment (word prediction).

---

Direct
MetaQuestionSimple
Metalnstruct
MetaQuesstonCCmmple
SyntaxGym
BLiMP
1.0
0.8
Accuracy
0.6
0.4
0.2
0.0
Flan-T5 SM
Flan-T5 LG
Flan-T5 XL
text-curie-001
lext-davinci-00Z
lext-davinci-003
Flan-T5 SM
Flan-TP Man-T5 LG
Flan- Flan-T5 XL
text-curie-001
text-davinci-002
text lext-davinci-003
(a) Experiment
Direct
MetaQuestionSimple
Metalnstruct
MetaQuessionComplex
SyntaxGym
BLiMP
1.0
0.8
Accuracy
0.6
0.4
0.2
0.0
Flan-T5 Flan-T5 SM
LG
Flan-T5 XL
text-curie-001
text-davinc-0022
text-davinci-003
Flan-T5 SM
Flan-T5 LG
Flan-T5 XL
text-curie-O0)
-axi-davin2--002
rext-davinci-003
(b) Experiment 3b
Figure 5: Task performance for each tested dataset in Experiments 3a (top) and 3b (bottom).

---

Flan- SM
Flan-
Flan- T5
Sentence comparison (Exp 3b)
00.2
-0.23
-0.26
0.45
0:48
0.49
0.44
0.45
0.44
Sentence judgment (Exp 3m)
0.08
0.09
0.09
0.24
0.27
0.25
0.38
0.37
0.40
1.0
Word comparison (Exp 2)
0.75
0.78
0.52
0.45
0.43
0.21
0.60
0.62
0.44
-05
Word prediction (Exp 1)
0.86
0.87
0.79
0.59
0.56
0.58
0.58
0.50
0.61
text-curie-001
text-davinci-002
text-davinci-003
Sentence comparison (Exp 3h)
0.000
-0.06
-0.20
0.41
0.41
0.42
0.42
0.39
0.43
-0.5
Sentence judgment
(Exp 36)
-0.02
-0.08
0.04
0.30
0.29
0.23
0.25
0.27
0.19
Word comparison
(Exp 2)
0.40
0.38
0.06
0.43
0.44
0.55
0.39
0.34
0.50
-1.0
Word prediction
(Exp 1)
0.95
0.95
0.43
0.90
0.89
0.72
0.87
0.B8
0.65
Moka Question
Mota Instruct
Meta Quest on Complex
Mesa Question Simple
Meta Instruct
Mota Question Complex
Meta Question Simple
Meta Instruct
Maka Question Complex
Figure 6: Internal consistency: Correlation between metalinguistic and direct responses gets weaker prompts become less direct. Pearson correlation between response magnitudes (averaged over datasets) measured by direct prompts versus each metalinguistic prompt.

---

MetaQuutsieeSmmreee
r.80
0.48.6
0.44 p 0.001
MeeiauessoonSimpl
r= 0,83 p 0.001
r= 0.71
r=
p 0.001
p.000001
p 0.001
p < 0.001
10-
-20-
F'
I= 0.34 p 0.001
F
I
0.66
Mecainstruct
1.0
P
p 0.001
Measinstuct
-10
p. 0.0001
P 0.001
p 0.001
-00
20
-30
See@uesforCumpple..
MeBuQusionCuure
p.x
I=m
p
p
F=
P.c.
p
p 0.001
-20-
-10-
-20
-20-
-30-
-20
-20 Direct
-20
-20 Direct
-20 Dirett
-00 Direct
Direct
Direct
Flan TS/
(b) Flan-TS/ News
text-curie 001
text dovinci 002
lext devinci-000
text-curie 0OL
Text wewiddiviiicoo 002
bext devtina 003
MessQussting Simple
McaaQusdiorSmmp
0.001
p
0.001
0.001
-20
-00
.20-
F= 0.94 0.001
F= 0.67 0.001
0.85 0.001
r= 0.97 00.001
0.91
Metainstrust
0.001
0.001
Metal
-10
-00
-00
-20
TeesQuestioCommpll Complex
MesoQuesiooiCamppe
0.551
== 0,84
== 0.80
0.
r= 0.35 0.001
r= 0.60
== 0.53
0.002
0.001
0,001
0.001
0,001
-30
10
-20
-20
-10
-20
-10 Direct
-20
-10
-20
-10
-20
-10
-20
-10
Direct
Direct
Direct
Direct
Direct
(c) GPT-3* /P18
(d) GPT-3* News
Figure Direct vs. metalinguistic responses for Experiment Relationship between log probability assigned model to ground-truth sentence continuation under the direct method and each metalinguistic prompting method. Dashed line indicates y.

---

Flan-T5 SM
Flan-TS LG
Flan-T5 XL
MeequestonEingg
r= 0,75 0.001
r= 0.45 0.001
0.60 0.00.
r= 0.78 p 0.001
I= 0.43 0.001
r= 0.62
Messinsrual
p 0.001
20
Meraquestio CenstonCCmmfee
[= 0.52
0=0.21
I= 0.44
20
p 0.001
g 0.001
p 0.001
0-
-30
Direct
Direct
Direct
(a) Flan- T5
text-davinci-002
MerqQuesioon Stmple
[=
== 0.43
== 0.39
p 0.001
x.c. 0.001
p 0.001
-20
Meteinstrus
P
P 0.001
p
0-
MetaquestaanCmmpe Compless
p
-20
20
Dirett
Direct
Direst
(b) GPT-3*
Figure 8: Direct vs. metalinguistic responses for Experiment 2. Relationship between log probability differentials
(plausible implausible) measured by direct method and each metalinguistic prompting method. Dashed line indicates

---

Flan-T5 SM
Flan-T5
Flan-T5
Flan
Flan
= 0.07
== 0.38
0.45 P 0.001
MelaQuestionSimii SimplennSSmplee
4 r=
0.10
0.30 P 0.000
Metquesionsse
0.202
p 0.001.
p 0.102
P 0.056
0=17 0.002
0.35
0=2 p 0.00
[= 0.00 p 0.933
0.19
0.31 p 0.0000
Metainsruct
0.00..01..
Metainssudt
g. 0.001
Monquesttsnoogge
0.001
0.00
Mels CeaaQuestionCupppe
[= 00.02 p 0.706
g. 0.001
== 0.22
p 000000
0
(a) Flan-T5 SyntaxGym
(b) Flan-T5 BLiMP
tnxt-curie-011
toxt-davinci-002
test-davincc-008
text-dastnc--022
test-davinc--003
MesaQuratioSSimgg
20-
p= 0.813
-0.03
== 0,28
< 0.001
MedeQuestiinSinkk
20
p= 0.644
0.01
0.30 p 0.001
24 0.001
20
[= 0.07 p= 0.156
0.27
Netainsiuu
p
x
Metainstruct
P 0.000001
= 0:24 p.c. 0.001
p.
-20
MetaQuesstion CumplionnoCCleee Comples
MesaQuestonCamuuee
20
0.04 p 0.447
r= 0.24
0.001
R0.
[= 0.04
p= 0.491
P 0.001
0.2.2
0.15 p 0.004
pnd 0.001
-25
Direct
Direct
Direct
Direct
Direct
Direct
(c) GPT-3* SyntaxGym
(d) GPT-3* PPTT3*/BLiMP BLiMP
Figure Direct vs. metalinguistic responses for Experiment 3a. Relationship between log probability differentials measured by direct method and each metalinguistic prompting method. For the direct method, we compute the difference in log probabilities assigned to the grammatical and ungrammatical sentences. For each metalinguistic prompting method, we compute the difference in log probabilities assigned the "Yes" token conditioned on the grammatical and ungrammatical sentence prompts (see Section 4.3 for details).

---

Flan
Flan
Flan-T5
Flan
MetaQuestioniin
-0.44 p 0.001
I= 0.53
p 0.001
0.46 p 0.00L
MetaQuesiion Simple
r= 00.21 p 0.001
0.37
p 0.001
ps 0,000
r= 0.42
0-
I 0.57
0.47 p 0.00y
r= 00.55 p
= 0,40
r=0.42
Metairstruct
p 0.00000
p 0.001
Metainsenud Q
0.001
p&00000
-5
MetaQuestorCampee
I 0.56
r= 0.47 e 0.001
Meta MelaQuestionoommge
g 0.000001
p 0.001
p
-5
25
(a) Flan-T3s 15/ SyntaxGym
(b) Flan-T5 BLiMP
inxt-curie-01
toxt-daviinc--002
nost-davinci-008
text-davinni-002
davinci-00
MesaQursienSrnui
0.02
== 0.52
MenQueeatonSinph
[= 0.02
r= 0.30
10
p= 0.763
< 0.001
10
p= 0.678
P 0.000
100
0.48 0.001
-10
0.35 p
MesaQueddonCammpe Metalnstuu
0.035
0.001
Metalnettull
-20-
"rum 0.45 p.c 0.001
-10
37
d 0.33
p.0.001
--2-0.27
r= 0.52
RenaQuedionCamppl
F= -=1-0:13 P.00.1
0.32
0.001
pix 0.001
10-
p p=000.00
0-48 p 0.001
p 0.001
-25
20
Direct
Direct
Direct
Direct
Direct
Direst
(c) GPT-3* SyntaxGym
(d) GPT-3* / HDTT33/BLiMPP BLiMP
Figure 10: Direct vs. metalinguistic responses for Experiment 3b. Relationship between log probability differentials measured by direct method and each metalinguistic prompting method. For the direct method, we compute the difference in log probabilities assigned to the grammatical and ungrammatical sentences. For each metalinguistic prompting method, compute the difference in log probabilities assigned to the "I" "2" answer options corresponding to the grammatical and ungrammatical sentences (see Section 4.4 for details).

---

Direct
MetaQuestion
Metalnstruct
MetaQuesttonComplex
Logprob final word
1.00
1.00
-2-
Accuracy
0.75
Accuracy
0.75
0.50
0.50
0.25
0.25
-6
text-davinci-003
text-davinci-00s
text-davinci-003
(8) Experiment
Experiment
Experiment
Figure 11: Task performance from preliminary experiments in Mandarin Chinese, See Appendix D for details.


Reviews:
[{"review_id": "PTnv2VRRaY", "reviewer": "Reviewer_vZ47", "summary": "", "questions": "Question A: How generalizable are your findings to language models trained on non-English languages, especially those with significantly different linguistic structures?\nQuestion B: Have your research considered or explored other possible evaluation methods? If not, what are your thoughts on broadening your investigation to include these methods?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments and feedback. Below, we respond to specific concerns and questions.\n\n**Generalizability beyond English (Question A):** Thank you for raising this point. To explore your suggestion, we performed a preliminary investigation of GPT-3.5 (text-davinci-003) on materials in Mandarin Chinese, which differs from English in many grammatical features. We consulted a native speaker to translate the metalinguistic prompts into Chinese. We then tested GPT-3 on two datasets: (1) a set of recent news articles for word prediction, mirroring Experiment 1; and (2) a set of controlled minimal pairs that cover semantic and syntactic phenomena (Wang, Hu, Levy, & Qian EMNLP 2021 https://aclanthology.org/2021.emnlp-main.454/), mirroring a combination of the phenomena tested in Experiments 2 and 3. We tested the tasks of Experiments 3a and 3b on this set of minimal pairs.\n\nLike our English experiments, in all our Chinese experiments we find that metalinguistic prompting and direct probability measurements do not yield the same results. Like our Experiment 1 in English, we also find that in the Chinese word prediction task, GPT-3.5 assigns highest probability to the ground-truth continuation under the “Direct” method. We also find that model performance improves when using minimal pairs, mirroring our original findings comparing Experiments 3a and 3b. These findings demonstrate how our English results may generalize to languages with different syntactic structures and grammatical properties. However, we also found a difference: in Experiment 3b in Chinese, the “Direct” method underperformed the metalinguistic methods (accuracy scores: Direct ~0.6; others ~0.8).\n\nOne potential explanation for this is that according to the OpenAI documentation, the models are “optimized for use in English”, although in practice they may work well for other languages (https://help.openai.com/en/articles/6742369-how-do-i-use-the-openai-api-in-different-languages). Therefore, the models may not be well-suited to scoring probabilities of Chinese sentences with no context (as in the Direct condition); instead, they may benefit from seeing additional Chinese text in the prompt before the sentence (as in the metalinguistic conditions). More broadly, while it would be interesting and useful to run our experiments in other languages, the outputs of the GPT-3/3.5 models in non-English languages may be difficult to interpret. \n\n**Exploring other evaluation methods (Question B):** Thank you for raising this point as well. We choose to focus on prompting and direct probability measurements because we have access to ground-truth probabilities (i.e., by reading out the model logits), which allows us to cleanly test the faithfulness of metalinguistic judgments. In future work, it would be very interesting to compare these approaches to other NLP evaluation methods (e.g., classification tasks like natural language inference). By considering a larger set of methods, the focus would shift away from the validity of metalinguistic prompting, and more toward consistency across methods. For this reason, we feel this warrants a separate follow-up study independent of the current paper.\n\n**Not addressing root causes:** This is a good point, and something that would be very interesting to explore in future work. Future directions could analyze whether models consistently over- or under-report internal probabilities, whether the discrepancies can be predicted from certain prompt structures, or whether certain interventions can increase models’ faithfulness in reporting internal representations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The strengths of this paper and the benefits it could offer to the NLP community include:\n\n- Detailed examination of evaluation methods: The paper deeply analyzes how large language models are evaluated, focusing on the differences between metalinguistic prompts and direct measurements. This examination contributes to understanding how well these methods can capture the linguistic competence of the models.\n\n- Practical guidance for researchers: The findings of this study provide invaluable guidance for researchers who work with language models. By highlighting the limitations of metalinguistic prompts, the paper offers insights into how these methods may misrepresent a model's understanding. It encourages researchers to cautiously approach such results, thereby promoting more accurate interpretations of experimental outcomes.\n\n- Theoretical implications: The paper contributes to the broader academic discourse surrounding language models, particularly in relation to the distinction between competence and performance. This theoretical input can stimulate further discussions and investigations in the field.", "reasons_to_reject": "Some weaknesses of this paper are:\n\n- Limited generalizability: The study primarily focuses on evaluating English language models. This might limit the generalizability of its findings to models trained on other languages, especially those with significantly different linguistic structures.\n\n- Insufficient exploration of alternative approaches: The paper focuses extensively on metalinguistic prompting and direct measurements. Including a wider range of evaluation methods or diversifying the datasets and tasks could provide a more comprehensive understanding.\n\n- Not addressing root causes: While the paper points out the misalignments in LLM responses based on the type of prompt, it doesn't deeply explore the underlying reasons for such discrepancies in LLM behavior.", "questions_to_authors": "Question A: How generalizable are your findings to language models trained on non-English languages, especially those with significantly different linguistic structures?\nQuestion B: Have your research considered or explored other possible evaluation methods? If not, what are your thoughts on broadening your investigation to include these methods?", "paper_topic_and_main_contributions": "Problem or Question Addressed:\n\nThe primary concern of the paper is determining the efficacy of metalinguistic prompts in assessing the linguistic capabilities of LLMs compared to direct measurements.It also addresses the issue of whether these two evaluation methods produce consistent results, especially given the move towards closed APIs that may restrict access to direct probability measurements.\n\nMain Contributions:\n\n- Value of minimal-pair comparisons: The paper highlights the significance of using minimal-pair comparisons, a method where two very similar sentences (differing perhaps in only one grammatical aspect) are presented, to reveal models' generalization capabilities.\n\n- Distinction between competence and performance: The study reveals that a model might have the relevant linguistic representation (competence) but may not always be able to access and describe this representation through metalinguistic evaluation (performance). This challenges prior criticisms of LLMs and underscores the difference between what a model knows and how it can express or describe what it knows.\n\n- Implications for LLM evaluations: The study underscores the importance of being cautious about the evaluation methods chosen. It emphasizes that researchers should clearly state their assumptions when selecting a particular evaluation method. Additionally, the study highlights potential challenges and losses in moving towards closed APIs that limit access to probability distributions.\n\nIn essence, the paper serves as a cautionary note on how the linguistic capabilities of LLMs are evaluated and the potential pitfalls of relying solely on metalinguistic prompts.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MubQybZLUC", "reviewer": "Reviewer_FmJA", "summary": "", "questions": "A) Do you have any speculation as to why MetaQuestionComplex performs so well in the text-davinci-002 and text-davinci-003 runs in Experiment 2?\n\nB) Are there cases where you think prompting would still be preferable to direct measurement (assuming both are feasible)?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments and feedback. Below, we respond to specific concerns and questions.\n\nWith regards to practical relevance, there is certainly a substantial body of work using direct probability measurements to evaluate LMs. However, there is also a growing trend to report the outcomes of metalinguistic prompts (e.g., Katzir 2023, Dentella et al. 2023). There will likely be increasing pressures to rely on metalinguistic prompts as our field continues to shift toward models with closed APIs. Therefore, we feel that a detailed investigation comparing the two methods is very timely and relevant, highlighting the value that would be lost if metalinguistic prompting becomes the norm in LM evaluation. We would be happy to clarify this in the revision.\n\n**Question A:** We speculate that this may be because in MetaQuestionComplex for Experiment 2, the answer options (e.g., “arrow, or interview”; see Table 3) are linearly closest to the end of the prompt, where the model begins its generation. To test this idea, we ran GPT-3.5 (text-davinci-003) on a new prompt variant, MetaQuestionComplexOptionFirst, where the answer options are stated near the beginning of the prompt (see response to Reviewer 3kd5 below for more details and examples). We found that the performance on MetaQuestionComplexOptionFirst (0.8) was more similar to the other metalinguistic prompt types, and significantly lower than performance on MetaQuestionComplex (0.96) for Experiment 2. This seems consistent with our speculation. However, the exact causes of variation across prompts are still unclear, and should be investigated in more detail in future work.\n\n**Question B:** Prompting might be useful for evaluating open-ended responses, such as explanations beyond multiple-choice tasks, or for interacting with models in flexible ways, such as chain-of-thought reasoning. Metalinguistic prompting could also allow for more targeted questions that would be challenging to translate into direct probability measurements (e.g., “Which of the following two sentences is more semantically plausible, but less syntactically well-formed?”). We also note that prompting lowers the technical barrier for domain experts outside of NLP to conduct LM evaluations, and could very well be preferred for practical reasons.\n\n**Typos:** Thank you for pointing this out. We chose to keep the x- and y-axis limits consistent through all facets in each subplot (a-d), to facilitate comparison across the facets. It turns out that in the left column of subplot Figure 10(a), most of the y-coordinates points lie within the region [-0.1, 0.1], and thus the points appear to be in a flat line when the y-axis limits are set to [-5, 5]. Apologies for the confusion, and we will do our best to clarify this in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The experimental results present convincing evidence that direct probability measurement works better for the selected tasks. The methods are clearly outlined, and the inclusion of multiple tasks differing in the directness of prompts is helpful. The paper is well written and easy to follow.", "reasons_to_reject": "The practical relevance of the findings is not fully clear. My impression is that evaluation related to uncovering linguistic generalizations (e.g. semantic plausibility judgments) is already done using direct probability measurement when possible. The paper did not seem to present examples of tasks where prompting is popular but direct measurement would be better.\n\nRevision: Having read the author response and looked at their cited sources more (e.g., the Katzir and Dentella et al papers), I am now more convinced that prompting is sometimes being used even when direct measurement might perform better. I am therefore increasing my excitement score from 3 to 4.", "questions_to_authors": "A) Do you have any speculation as to why MetaQuestionComplex performs so well in the text-davinci-002 and text-davinci-003 runs in Experiment 2?\n\nB) Are there cases where you think prompting would still be preferable to direct measurement (assuming both are feasible)?", "paper_topic_and_main_contributions": "This paper compares two methods for assessing language models' linguistic knowledge: direct probability measurements and prompting. The tasks on which these methods are compared involve comparing word and sentence probabilities to see how they reflect factors such as semantic plausability and syntactic agreement. Results show that direct probability measurement almost always leads to better performance than prompting. This could potentially motivate future work to prefer linguistic assessments involving direct measurement rather than prompting.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "In Figure 10(a), the reported correlation values don't seem to match the scatter plots in the left column."}, "extra_text": {}}, {"review_id": "NU8FTVUafy", "reviewer": "Reviewer_3kd5", "summary": "", "questions": "What linguistic knowledge is the word prediction task testing, exactly? This should be explained better in the paper imo.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments and feedback. Below, we respond to specific concerns and questions.\n\nWe would be happy to include more variations of prompts. To explore this suggestion in the timeframe of the author response period, we tested two additional metalinguistic prompt variations. The first set of new prompts (“MetaInstructKeywords”) is structured such that the relevant sentence (or sentence prefix) and answer are placed at the end of the prompt, with “Sentence:” and “Answer:” keywords. The second set of new prompts (“MetaQuestionComplexOptionFirst”) is structured such that the answer options (only applicable to Experiments 2-3) are presented at the beginning of the prompt, and otherwise mirrors MetaQuestionComplex. (Example prompts are illustrated at the very end of our response.) We re-ran GPT-3.5 (text-davinci-003) on all experiments using these new prompts, and our original qualitative findings still hold. We would also be happy to include more in the final paper, and welcome suggestions for additional prompts to try!\n\nWe are also happy to include more discussion on metalinguistic prompting as a potentially desirable property for LMs.\n\nWe will also clarify the connection between our work and the debate on “LLMs as models of human language processing”. We mention this debate in order to highlight what’s at stake – i.e., the substantial theoretical claims that researchers are making based on metalinguistic judgments. This helps to motivate our investigation of metalinguistic prompting methods. However, we don’t intend to take a stance on the theoretical debate in our paper.\n\nThe word prediction task is not testing a particular component of linguistic knowledge (in the sense of syntax or semantic plausibility) – instead, we treat it as a baseline task. Because we know the model represents next-word-probabilities, we know that the direct probability measurements are “ground-truth”, allowing us to do a tightly controlled comparison to metalinguistic prompting methods. We would be happy to clarify this in the revision.\n\n---\n\n**New prompt #1: MetaInstructKeywords**\n\n*Experiment 1*\n\nI will show you the beginning of a sentence. Tell me what the most probable next word is.\n\nSentence: [SENTENCE_PREFIX] …\nAnswer:\n\n*Experiment 2*\n\nI will show you the beginning of a sentence. Tell me which is more likely to be the next word of the sentence, from the following options: “arrow”, or “interview”.\n\nSentence: [SENTENCE_PREFIX] …\nAnswer:\n\n*Experiment 3a*\n\nI will show you a sentence. Tell me whether it is a fluent English sentence. Respond with either “Yes” or “No” as your answer.\n\nSentence: [SENTENCE]\nAnswer:\n\n*Experiment 3b*\n\nI will show you two sentences. Tell me which of the two sentences is more fluent in English. Respond with either “1” or “2” as your answer.\n\nSentence 1: [SENTENCE_1]\nSentence 2: [SENTENCE_2]\nAnswer:\n\n**New prompt #2: MetaQuestionComplexOptionFirst (only for Expts 2-3)**\n\n*Experiment 2*\n\nI will show you the beginning of a sentence, where the next word should either be \"arrow\" or \"interview\". [SENTENCE_PREFIX]... Which of these two words is more likely to come next in the sentence?\n\nAnswer:\n\n*Experiment 3a*\n\nI will show you a sentence and ask a question with “Yes” or “No” as the answer options. Is the following sentence a good sentence of English? [SENTENCE]\n\nAnswer:\n\n*Experiment 3b*\n\nI will show you two sentences and ask a question with “1” or “2” as the answer options. Which of the following two sentences is a better English sentence? 1) [SENTENCE_1] 2) [SENTENCE_2]\n\nAnswer:", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is a valid and quite thorough evaluation pointing out a relevant limitation of prompting as a way to measure model ability, specifically in linguistic knowledge.\n\nThe paper is well-written and the methods are well-described.", "reasons_to_reject": "- The common wisdom in prompting is to use more on the order of 10 different prompts. I believe it would be helpful, and make for more reliable results, to include more variations of the respective prompt types.\n- This is a minor point, but I think the discussion is missing that being able to access this knowledge via prompting may in fact be a desirable property, and should probably be improved.\n- The paper repeatedly refers to a debate about “LLMs as models of human language processing” but the connection of such a debate to the work presented is left unclear.", "questions_to_authors": "What linguistic knowledge is the word prediction task testing, exactly? This should be explained better in the paper imo.", "paper_topic_and_main_contributions": "This paper compares prompting and direct evaluation of linguistic knowledge in language models. It shows that evaluation by prompting tests for more than just the ability to encode concepts, and that models generally perform worse on metalinguistic prompts. \n\nMy impression is that this is a fairly predictable result but it is valuable to point out and quantify.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All three reviewers provided positive assessments, consistently giving a score of 4 for both soundness and excitement. They consistently agreed that this paper is thorough, well-written, and of clear relevance to the field.

The strengths and weaknesses described by the reviews may be summarized as follows.

Strengths:
- valid and thorough experimental design (R1, R2, R3)
- well-written (R2, R3)
- clear and practical implications for the field (R1)

Weaknesses:
- only tested on one language (English) (R1) and limited exploration of the space of possible prompting strategies (R3)
Decision: Accept-Main