Auxiliary task demands mask the capabilities of smaller language models
Authors: Jennifer Hu
Michael Frank
Venue: COLM (2024)
Keywords: cognitive evaluation, task demands, benchmarking, emergence, reasoning, syntax, development
TL;DR: The auxiliary demands of a task can affect LMs' performance on that task, and task demands asymmetrically affect models with fewer parameters and less training data.

Abstract: Developmental psychologists have argued about when cognitive capacities such as language understanding or theory of mind emerge. These debates often hinge on the concept of "task demands" -- the auxiliary challenges associated with performing a particular evaluation -- that may mask the child’s underlying ability. The same issues arise when measuring the capacities of language models (LMs): performance on a task is a function of the model's underlying knowledge, combined with the model’s ability to interpret and perform the task given its available resources. Here, we show that for analogical reasoning, reflective reasoning, word prediction, and grammaticality judgments, evaluation methods with greater task demands yield lower performance than evaluations with reduced demands. This "demand gap" is most pronounced for models with fewer parameters and less training data. Our results illustrate that LM performance should not be interpreted as a direct indication of intelligence (or lack thereof), but as a reflection of capacities seen through the lens of researchers' design choices.

Published as a conference paper at COLM 2024
Auxiliary task demands mask the capabilities of smaller language models
Jennifer Hu
Kempner Institute for the Study of Natural and Artificial Intelligence Harvard University
jenniferhuēfas. harvard.
Michael C. Frank
Department of Psychology
Stanford University
mcfrank@stanford. edu
Abstract
Developmental psychologists have argued about when cognitive capacities such as language understanding or theory of mind emerge. These debates often hinge on the concept of "task demands" the auxiliary challenges associated with performing a particular evaluation - that may mask the child's underlying ability. The same issues arise when measuring the capacities of language models (LMs): performance on a task is a function of the model's ss underlying knowledge, combined with the model's ability to interpret and perform the task given its available resources. Here, we show that for analogical reasoning, reflective reasoning, word prediction, and grammaticality judgments, evaluation methods with greater task demands yield lower performance than evaluations with reduced demands. This "demand gap" is most pronounced for models with fewer parameters and less training data. Our results illustrate that LM performance should not be interpreted direct indication of intelligence (or lack thereof), but reflection of capacities seen through the lens of researchers' design choices.
1 Introduction
A shared goal in psychology and artificial intelligence is to ascribe cognitive capacities to black-box agents. For example, we might be interested in whether a young child has "theory of mind", or whether language model (LM) has the ability to distinguish grammatical and ungrammatical sentences. The trouble is, although we would like to infer an underlying psychological construct, we only have access to specific observable evaluations a child's sss ability to answer question about character in a story, or model's performance on a syntax benchmark.' Inferences from evaluations to constructs are ubiquitous in developmental psychology (Frank, 2023a), and cognitive evaluations of artificial models are beginning to follow similar principles (e.g., Momennejad et al., 2023; Ivanova, 2023).
For both humans and machines, making inferences about failures is especially tricky, because failure on a task does not always indicate the absence of the underlying capacity. In developmental psychology, children often fail due to task demands: auxiliary challenges unrelated to the capacity of interest (e.g., Keen, 2003; Carruthers, 2013; Kuran-Küçüük & Kibbe, 2024). Children of given age may succeed in a simple version of a task, while failing at more complex version that incorporates other demands. Sometimes the issue is as basic as children not understanding the question being asked, while at other times task demands can include issues like remembering multiple pieces of information inhibiting a response.
Here, we use the term "evaluation" to encompass tasks, benchmarks, measures, and experiments, for both humans and machines.
1

---

Published as a conference paper at COLM 2024
A
Evaluation
B
task
Hypothesized task demand
demands
Performance
Dependent variable
Interpret question
Interpret question
High demands
What word is most likely come next the following text?
High demand evaluation
Internal predictions
Predict of sentence
Capabliity agent (age, size, training)
Text: take my coffee with cream and
Produce prediction after prompt
Transfer prediction prompt
C
Prediction:
Demand gap
(Low - High
Verbal response
Decoded output
Low demand evaluation
Comprehend sentence
Predict token
take my coffee with cream and sugar.
Capability of agent (age, size, training)
Reaction at target word
Probability of target word
Figure 1: A: Hypothetical task demands in two evaluation settings, faced by humans and machines. Both methods apparently measure the accuracy of target word prediction, but the high demand setting imposes additional auxiliary demands. B: Hypothesized pattern of results if task demands asymmetrically affect less capable agents (e-g, younger children or smaller models). C: Signature "demand gap" produced by hypothesized pattern in B.
Here, we argue that task demands also play an important role in determining evaluation success or failure for LMs, especially when comparing models of different capacities. As conceptual illustration, Figure 1A depicts hypothetical task demands faced by humans and machines for two different ways of evaluating word prediction ability. The high-task- demand evaluation presents metalinguistic prompt, which explicitly asks the agent for prediction. A correct response for this stimulus requires interpreting the question, accessing internal predictions for that target sentence, and producing them after the prompt. In contrast, lower-demand design for humans might be the measurement of reaction time upon hearing the target word ("sugar"); for LMs, a lower-demand task would be a readout of the probability of the target word. While both of these evaluations apparently measure the accuracy of the LM's predictions, models might fail in the high-demand situation due to difficulty with task demands unrelated to the accuracy of their actual target word prediction. Indeed, recent studies have demonstrated performance gaps between higher- and lower- demand evaluations in variety of linguistic domains (e.g., e.g. Hu & Levy, 2023; Hu et al., 2024; Kauf et al., 2024; West et al., 2024; Tsvilodub et al., 2024).
Our primary question is not whether these performance gaps exist, but whether task demands asymmetrically affect less capable language models, mirroring the findings in children. A null hypothesis might be that low-capability model for example, one with fewer parameters or less training data would perform equally poorly on all cognitive evaluations, regardless of the method. Inspired by the task demand debate in developmental psychology, we instead predict that less capable agents should suffer more from task demands than more capable agents (Figure 1B). If borne out, this prediction would produce signature "demand gap", which would be larger in less capable LMs (Figure 1C).
We present experiments demonstrating this signature pattern a larger demand gap for less capable models across a variety of tasks and open-source LMs. We investigate two evaluation contrasts: production vs. forced choice, and metalinguistic judgment vs. proba- bility measurement. For each of these contrasts, we measure the demand gap between the higher- and lower-demand evaluation methods on multiple cognitive domains, collectively covering analogical reasoning, reflective reasoning, word prediction, and grammaticality judgments. We find evidence that demand gaps get smaller as LMs increase in size (i.e., number of parameters) and training time. Our findings suggest that task demands can mask
2

---

Published as a conference paper at COLM 2024
the abilities of less capable models, illustrating how inferences about LM abilities depend on our choice of evaluation metrics.
The paper is structured as follows. In Section 2, we more fully describe the concept of task demands and how it relates to other issues in LLM evaluation. We describe the logic and methods behind our experiments in Section 3, and present the results in Section 4. Finally, in Section 5 we conclude and discuss broader implications for NLP and cognitive science.
2 Background and related work
Psychology focuses on connecting observable measurements to latent (unobserved) psy- chological constructs. A valid measure is one that appropriately measures the construct (Cronbach & Meehl, 1955). Task demands are one of many different factors that can un- dermine validity. In particular, the presence of auxiliary demands in task means that the inference from observed performance to construct is no longer "pure." Reducing these demands can enable more direct inference about the latent constructs of interest.
While the psychology literature does not provide precise definitions of what counts as a task demand, informally task demand can be thought of as a feature of the task that imposes challenges unrelated to the capacity that the task is designed to measure. For humans (and particularly in children), these challenges might be related to processes such memory, attention, or inhibition, which are involved in performing the task but separate from the target capacity. It is unclear how these kinds of challenges manifest in LM computation, and our work aims to explore several such possibilities.
An evaluation with fewer task demands is not necessarily easier or more lenient. Instead, better gloss might be that it is a purer task, in the sense that it better measures the targeted construct. Consider an evaluation of arithmetic ability: the size of the numbers being manipulated and the range of operations would likely change the overall difficulty of the evaluation. In contrast, the use of word problems, pictorial representations, or framing in terms of real-world objects would all change the task demands of the evaluation, even if the difficulty of arithmetic components stayed constant. Some of these changes might serve to make the task easier for example, if they engaged familiar content (Lampinen et al., 2024) while others might make the task harder (at least for some models).
Our work here builds upon the field of LM evaluation specifically, research on links be- tween evaluation metrics and underlying constructs, or in other words "machine evaluation validity". For example, Schlangen (2019) points out that individual evaluation tasks are often only loosely related to the underlying construct that researchers want to engineer into their system. Other work has questioned the validity of inferences from forced choice metrics (Tsvilodub et al., 2024; Schaeffer et al., 2023), and several studies have noted the mis- alignment between production/ forced choice (Lyu al., 2024; Röttger al., 2024; West et al., 2024) and between prompting/ measurement of underlying probabilities (Hu & Levy, 2023; Hu et al., 2024; Kauf et al., 2024). related line of work asks how design "species-fair" comparisons between humans and models (Firestone, 2020; Lampinen, 2023).
These specific investigations of validity are part of the broader effort to design robust LM evaluations. The results of any evaluation depend in part design choices, which necessarily reflect a researcher's 's broader goals or theoretical commitments, These design choices have been widely discussed (e-g., Linzen, 2020; Bowman & Dahl, 2021; Raji et al., 2021), including factors such as size and scope (Srivastava et al., 2023), dynamic updating (Kiela et al., 2021), and adversarial approaches (Nie et al., 2020; Sakaguchi al., 2021). To our knowledge, most of this work has not specifically analyzed the interaction between evaluation method and model capability that our focus (but see Schaeffer et al., 2023).
Methods
Our main research question is whether less capable language models are more sensitive to task demands. If this is the case, then we would expect to find the pattern in Figure 1C: the performance gap between low- and high-demand tasks should be smaller for more capable
3

---

Published as a conference paper at COLM 2024
Evaluation
Cognitive construct
Dataset(s)
Goal
Example item
Production (high
demands) vs.
Analogical reasoning
Webb et al. (2023)
Predict next digit(s)
[59 [89]]111 9 18 [847][1 433 4 [n[]22]]52]]
Forced choice (low demands)
sequence
Reflective reasuming
Hagendorff et al. (2025)
Override intuitive solution to correctly solve the problem
chair and euat fugether cust $13. The chair costs $10 more than the cost. How much does the cuat cust?
Metalinguistic
judgment (high demands) vs.
Word prediction
LAMBADA (Pa- pemoo et et, 2016)
Predict final word of the text (under- lined)
sun speekled shade grees beneath were welcome respite at- ter the stifling kitchen, and was glad to relax against the tree's rough, brittle bark and begin my breakfast buttery, loasted bread fresh fruit. Even the tasty, was clean and cold. almosst made up for the
Probahility measurement (low demands)
coffee
Grammaticality judgment
BLiMP (Warstadt et al., 2020); Den- tella et (2023a); Hu et al. (2024)
Identify the gram- matical sentence in minimal pair
(1) Rachelle had bought that chair. (2) 'Rachelle had bought that chairs.
Table Overview of task contrasts and cognitive domains tested in our experiments.
models. In the remainder of this section, we discuss the evaluation contrasts, cognitive constructs, and models tested in our experiments (see Table for summary).
3.1 Evaluation contrasts
We focus two evaluation contrasts that are relevant to most behavioral LM evaluations. When designing an evaluation, researchers have the choice to assess the LM in a generative (production or discriminative (forced choice) setting. Independently, there is also the choice of formulating the task as a prompt or a direct measurement of string probabilities. Both contrast between methods with higher and
of these decision points can be formulated lower levels of task demands, which we discuss in more detail below.
Production vs. forced choice. The first contrast compares production of the correct answer (higher task demands) versus forced choice over a fixed set of answer options (lower task demands). Notably, children's ability to produce a certain linguistic form tends to lag behind their ability to comprehend that form (e.g., Hendriks & Koster, 2010). Forced choice paradigms have revealed knowledge in children that would not otherwise be observed through production. For example, 6-9-month-old infants can direct their gaze to named objects, before being able to talk themselves (Bergelson & Swingley, 2012).
is not obvious that this asymmetry would also apply for LMs, for which the task of generating linguistic output which may impose auxiliary demands on children, including planning motor actions associated with speech production - is the primary rationale. But even without these demands, forced choice requires less precise estimates of a probability distribution than correct production. Conditioned on context (e.g., the statement of reasoning problem), the LM's ability to solve the task is encoded in its distribution over subsequent tokens. This distribution is the same, regardless of whether the LM is evaluated through forced choice or production. Production, which involves sampling an output from the distribution, might not demonstrate the model's 's abilities because of auxiliary factors (e-g,, the decoding method or low absolute probabilities) whereas targeted comparisons of answer options might reveal distinctions that are present in the model's probability distribution but would not be sampled. In this sense, production and forced choice can target the same cognitive construct, while imposing different kinds of auxiliary demands."
Forced choice evaluation does require researchers to design a set of answer options, which may introduce unintended artifacts (e.g., content or frequency effects). In line with this view, West et al. (2024) find that generative (production) performance is better than discriminative (forced choice) performance across range of models and tasks. However, this finding could be due to the complexity
4

---

Published as a conference paper at COLM 2024
Metalinguistic judgment vs. probability measurement. The second contrast compares two ways measuring linguistic knowledge: a metalinguusticc judgment versus a direct mea- surement of how likely well-formed linguistic expression is. From the developmental perspective, children can identify well-formed sentences of their language, without necessar- ily being able to articulate the underlying grammatical rules that govern these sentences. As previously argued by Hu & Levy (2023), the probabilities assigned to strings directly reflect model's linguistic generalizations, whereas metalinguistic prompts impose additional demands such as correctly interpreting the question (see Figure 1A).
3.2 Cognitive domains
3.2.1
Production vs. forced choice
For the production versus forced choice contrast, evaluated LMs in two cognitive domains: analogical reasoning and reflective reasoning. Reasoning abilities have been of great recent interest in LM evaluation (e.g., Kojima et al., 2022; Webb et al., 2023; Chang & Bergen, 2024), and the test sets we used feature well-motivated answer option sets that enable forced choice evaluation.
Analogical reasoning. We used the digit matrices task introduced by Webb et al. (2023) to measure LMs' ability to perform analogical reasoning. Each item presents a sequence of digits in matrix format, and the model is evaluated its ability to predict the continuation of the sequence. The relationships between the sequence and continuation involve a variety of transformations and logical patterns; refer readers to Webb et al. (2023) for details.
We followed Webb et al.'s protocol for scoring. For the forced-choias evaluation, we com- pared the log probability assigned by the model to each of the possible answer choices (summed over tokens). The set of answer choices for each problem was defined by Webb et al. (2023). For the production condition, we truncated models' responses where a closing bracket is generated, and then evaluated the response based on whether it contained the correct set of digits. We report results averaged over all problem types.
Reflective reasoning. To evaluate LMs' reflective reasoning abilities, we used the cognitive reflection tests (CRTss) described by Hagendorff et al. (2023). CRTs (Frederick, 2005) are problems that have a reflexive, intuitive solution, which must be overridden through reflective problem-solving to arrive at the correct answer. The CRTs used by Hagendorff et al. (2023) cover three conditions which probe intuitive biases observed in humans.
To evaluate models in the forced-choice condition, we compared the mean log probability assigned to the correct and intuitive answers. We average probabilities over tokens because answer options can differ in length (e.g., "$10. 00"0""""""" vs. "$1. .50"). The model succeeds on given item if it assigns higher probability to the correct answer than the intuitive answer, conditioned on the problem statement. To score the freely generated responses, manually inspected the outputs and labeled whether they corresponded to the intuitive answer, correct answer, neither. Results are averaged over the three tested conditions.
3.2.2 Metalinguistic judgment vs. probability measurement
For the metalinguistic jonaiingistijudgggee prediction and grammaticality judgment because they are the clearest cases where metalin- guistic prompting differs from established ways of using direct measurements.
judgment versus probability contrast, evaluated LMs on word
Word prediction. A natural locus for gap between direct measurement and metalin- guistic judgment is word prediction. Since the direct evaluation setting is analogous to models' autoregressive training objective,
view these predictions as "ground-truth" measure of models' underlying word prediction abilities, providing a useful reference for
of the answer options in their study, which were designed to include adversarial "hard negatives", More broadly (and in keeping with our general theme), researcher goals (in this case, adversarial testing as opposed to capacity discovery) can modulate the difficulty of any evaluation method.
5

---

Published as a conference paper at COLM 2024
evaluating metalinguistic abilities (Hu & Levy, 2023). We evaluated models' word predic- tion ability using the LAMBADA dataset (Paperno et al., 2016). The items consist of short multi-sentence passages, which are constructed such that correctly predicting the final word of the text depends on the entire context, and not just the final sentence.
To evaluate models using direct probability measurements, we computed the log probability assigned by the model to the target final word conditioned on the prefix (summing over sub-word tokens). To evaluate models metalinguistically, we computed the log probability of the final word conditioned on the metalinguistic prompt shown in (1).
(1) What word is most likely to come next following text? Text [prefix]
Prediction:
Grammaticality judgment. A prominent question in linguistics and LM research is whether LMs capture the distinction between grammatical and ungrammatical sentences. While body of work has evaluated this ability by testing whether models assign higher probability to grammatical versus ungrammatical sentences (e.g,, Warstadt et al., 2020), there has been growing trend to instead use metalinguistic prompts (e.g,, Dentella al., 2023a;b; Katzir, 2023). Hu & Levy (2023) and Hu et al. (2024) demonstrated that the proba- bility comparison method can reveal knowledge that is masked by metalinguistic methods. However, they did not investigate how this gap varies with scale or training time.
We used two datasets evaluate models' grammatical abilities: BLiMP (Warstadt et al., 2020), 4 and the set of minimal pairs based on Dentella et al.' (2023a) materials constructed by Hu et al. (2024). To evaluate models using direct measurements, we assessed whether the model assigned higher probability to the grammatical sentence than its ungrammatical counterpart. This method is designed to control for factors such as lexical items and length (see also Linzen et al., 2016; Marvin & Linzen, 2018). To evaluate models metalinguistically, prompted the model to compare the two sentences in the minimal pair:
(2) Which the following two sentences more grammatically correct English? Respond with as your answer Sentence 1: [sentencel] Sentence [sentence2] Answer:
For each minimal pair, we evaluated models on two versions of (2), swapping the order in which the sentences presented. A model succeeds if it assigns higher conditional proba- bility to the token "1" or "2" (whichever one corresponds to the index of the grammatical sentence). Accuracy is computed the average success across these two orders.
Note that we only tested a single prompt for each of the metalinguistic evaluation tasks. refer readers to Hu & Levy (2023). They tested
For results on broader set of prompts, models on 3 different metalinguistic prompts in both English and Chinese, and found similar qualitative findings across prompt types and languages.
3.3 Models
Recall that our main question is whether the demand gap shrinks over the course of model "development", or as its general capabilities increase (Figure 1C). We tested two ways of operationalizing this concept for language models: varying the size (i.e., number of parameters) while keeping other architectural and training details as similar as possible, and varying the duration of training for given model instantiation.
This analysis is not meant to draw direct parallels between development in children and scaling in LMss we don't 't claim, for example, that llama-2 7B is model of a young child and Llama-2 70B is a model of an adult. Instead, we use size and training as practical proxies for how LMs might gain resources that help with overcoming task demands.
We use the test split pre-processed by OpenAL, as exposed by the EleutherA//lambada_geee Huggingface dataset. We obtain the prefix and final target word by splitting on whitespace. We randomly sampled 50 items from each of the 13 categories, resulting in 650 items.
6

---

Published as a conference paper at COLM 2024
Model family
Sizes tested
Training tokens
Data cutoff
Pythia (deduped)
2.8, (1, 2.8, 6.9, 12 B
207 B
2020
OLMo
[1. B
[3, 2.5| T
March 2023
Gemma
[2
[2, 2T
unknown (before Feb 2024)
Llama-2
17, 13, 70] B
Sept 2022
Mistral
7B
unknown
unknown (before Oct 2023)
Table 2: Models tested. For OLMo and Gemma, the first/ model size is associated with the first/ second amount of training tokens. For Gemma and Mistral, the data cutoff date is not publicly known, but we assume it is before publication of the associated paper.
Model size. We first use model size as a way of manipulating an LM's general capabilities. Models with more parameters more expressive, suggesting that they might be able to handle auxiliary task demands that would be difficult for less expressive models (e.g., interpreting metalinguistic question, maintaining the result of computation across intervening content in prompt).
We evaluated 13 open-source autoregressive language models using the Huggingface Trans- formers library. The models range in size from 1B to 70B parameters spanning across five families: deduplicated Pythia (Biderman et al., 2023), OLMo (Groeneveld et al., 2024), Gemma (Gemma Team et al., 2024), Llama-2 (Touvron et al., 2023), and Mistral (Jiang et al., 2023). Table summarizes the sizes we tested for each model family, as well as other training details. We chose these models to cover range of sizes, while also achieving reasonable performance (ruling out smaller models such as GPT-2) and ensuring reproducibility. Since our experiments require token-level logits, we could not evaluate closed-Al API models.
Training time. Another way to manipulate the general capabilities of an LM is training time. While many studies examine emergence as a function of model size (e.g., Wei et al., 2022a), a growing line of work is concerned with "developmental interpretability" (Hoog- land et al., 2023), or how abilities and structures emerge over the course of learning (e-g., Power et al., 2022; Simon et al., 2023). For our experiments training time, evaluated OLMo 7B at 10 checkpoints over the course of training (including the final one).
Across all experiments, we computed conditional and full-sequence probabilities using the minicons package (Misra, 2022). All models were base models without any fine-tuning.
Data contamination. Some of our tested datasets were publicly available before the data cutoff of all our tested models (LAMBADA and BLiMP), and all of our datasets were released before the publication of Gemma. This raises the concern of data contamination: Le., that these datasets were seen during training. While this situation is not ideal, it is not a major concern because we are not interested in performance per se rather, we are interested in the difference in performance across evaluation methods. We do not have strong expectations that data contamination would affect one method more than another.
Results
4.1
Task demands vs. model size
Figure shows the results for our first tested contrast: production vs. forced choice. We first briefly discuss raw task performance. For the analogical reasoning domain, larger models generally achieve higher performance (compared to smaller models in the same family) under both evaluation methods (Figure 2A). Interestingly, we do not find this pattern for the reflective reasoning domain (Figure 2C), because the larger models have a strong preference for intuitive responses, whereas the small models slightly prefer correct intuitive but have a much stronger preference for atypical answers (see Hagendorff et al. 2023 for details).
Our main focus is the relationship between the demand gap (forced choice production) and number of parameters (Figure DB.D). We measure the demand gap as the difference in log odds, computed from the accuracy scores. In both domains, the difference in log
7

---

Published as a conference paper at COLM 2024
A
B
0.8
Forced choice fLow task demands)
Production (High tank demands))
Pytna
GLMo
Gemma
Liama-2
Mistral
Pyttia
Propotion carrect
0.8
ML
log
(Forced choce
0.6
0.4
Lirre-22
Mistoo
0.4
0.2
0.0
paramattrrs (log billooss)
18
1.48
2.58
6.98
12B
18
7B
28
78
138
70B
78
D
odes (Porocd Production)
Pythi
OLMo
Gemma
Llama-2
Mistral
0.6
Pythik
OLMo
0.4
Gemma
Proportin
Lisma-2
0.2
6-
1.48
2.88
128
7B
28
78
133
70B
Model
# parameters (bg billions)
Figure 2: Production (high task demands) vs. forced choice (low task demands) in two domains: analogical reasoning (top row) and reflective reasoning (bottom row). A,C: Accuracy scores across models and evaluation methods. B,D: Difference of log odds (forced choice production). Colored lines best-fit within model families. Black line best-fit across all models. Shaded region indicates bootstrapped 95% CI. (Log odds difference for Pythia 1B in panel infinite, so it is not shown.)
Direct (Low
Metslingulstic (High demands)
Logptoo of final word [Drect Metalingu (essste
Pythia
Logprob word
Pytiaa
OLMo
Gemma
10
18
1.48
2.88
6.06 12B
78
28
78
136
70B
78
parameters (log billioss)
C
Pytha
ML.Mo
Gemma
Llama-2
Mistral
(Drect- Metsing.stii Aetic
Proportonn currect
0.8
Log cdds
1.5
OLMo
Pyhia
0.6
Gemma
0.4
0.8
1.48
6:90
12B
76
76 136 70B
7B
(og billors)
Figure 3: Metalinguuitic juddigeenistiicddmmentt (high task demands) vs. direct probability measurement (low task demands) in two domains: word prediction (top row) and grammaticality judg- ments (bottom row). A: Log probability assigned to final word in word prediction domain. B: Difference of final-word log probability (direct metalinguistic). C: Accuracy in gra- maticality judgment domain. D: Difference of log odds (direct metalinguistic). Colored lines = best-fit within model families. Black line best-fit across all models. Shaded region indicates bootstrapped 95% CI.
odds tends to decrease as the number of parameters increases, demonstrating the predicted signature pattern (Figure 1C). This pattern holds for all model families except Pythia, whose log odds differences are relatively flat across model size.
Figure 3 shows the analogous results for second evaluation contrast: metalinguistic judgment vs. probability measurement. For word prediction as well as grammaticality judgment, the direct method generally yields higher performance than the metalinguistic
8

---

Published as a conference paper at COLM 2024
method, in line with Hu & Levy (2023) (Figure 3A,C). As before, we also find that the demand gap (direct - metalinguistic) decreases as model size increases for both domains. This gap is measured as log probability difference for the word prediction task (Figure 3B), and log odds difference for grammaticality judgment (Figure 3D). For the grammaticality judgment domain, we note that the log odds difference is fairly flat for the Pythia and OLMo families. The tested Pythia and OLMo models do not change in performance across sizes, and do not perform appreciably above chance (50%) under the metalinguistic method. It could be the case that due to their architectures or training, the "shrinking gap" effect might not be seen within the range of tested sizes, but could potentially emerge at larger sizes.
To formally test our demand gap prediction (visualized in Figure -C)-ies that smaller models are differentially affected by task demands- we analyzed the interaction between model size. and task demands. We used the 1me4 package in R to fit the following mixed effects model for each of the domains with binary responses (analogical reasoning, reflective reasoning, and grammaticality judgment):
correct size evalMethod (size evalMethod modelFamily)
(1)
We fit the following model for the word prediction domain:
oinal ~ size * evalMethod + (size * evalMethod | modelFamilyy (2)
We found significant interaction between model size and evaluation method for each do- main except reflective reasoning (analogical reasoning: = 2.631, p = 0.009; grammaticality 1.995, p 0.0461; word prediction: / 7.654; p 0.005).
judgment:
Overall, our results provide evidence for the pattern illustrated in Figure IC: the "demand gap" between high-demand evaluation methods and low-demand evaluation methods is smaller for more capable models, which we operationalized here using parameter count. Next, we turn to an alternate operationalization: training time of single model.
4.2 Task demands training time
We evaluated intermediate checkpoints of OLMo 7B on analogical reasoning (production vs. forced choice) and word prediction (metalinguistic vs. direct). Figure 4 compares perfor- mance across the low- and high-demand evaluation methods across training time. both domains, the low-demand method yields higher performance earlier during training. For analogical reasoning (Figure 4A), the slope of improvement early during training is flatter for production than forced choice. For word prediction (Figure 4B), the log probabilities achieved by the direct method quickly plateau at a high value, whereas they increase more consistently under the metalinguistic method.
Again, we fit generalized linear models testing for an interaction between training steps and evaluation method. Here, there is no model family grouping factor. We fit the following models for analogical reasoning and word prediction, respectively:
correct logTrainingStep evalMethod
finalWordLogprob logTrainingStep evalMethod
(4)
For both domains, we found significant interaction between the number of training steps and evaluation method (analogical reasoning: z 2.639, p 0.008; word prediction: t = 24.41; < 0.0001).
In sum, evaluation methods with lower task demands can reveal abilities in a model earlier during training that might not otherwise be revealed by higher-demand methods. These findings complement the results from Section 4.1, which showed that low-demand settings can reveal abilities in smaller models that would not be revealed by higher-demand methods.
5 Discussion
In studying the cognition of both humans and language models, one of the primary chal- lenges is linking specific evaluations to more general constructs. Designing valid evaluations
We did not perform this analysis the other two domains because the fully trained OLMo 7B model achieved poor performance under the high-demand evaluations (see Figure 2C and Figure 3C).

---

Published as a conference paper at COLM 2024
Logprosb of first word
Proportion cornect
2-
0.4
Foroid choice
-8-
Drectt Metalingussto
Figure 4: Performance across in-
termediate training checkpoints of
Production
0.2-
OLMo 7B on analogical reasoning (A) and word prediction (B).
200000
400000
200000
400000
Training slep
Training step
is thus critical for LM research. Here, we used a concept from developmental psychology auxiliary task demands - to investigate the validity of a range of LM evaluations. We found that evaluations with higher demands led to lower performance, and especially so for LMs with fewer parameters or less training a phenomenon we termed the "demand gap." " Our findings also connect to emergence (Wei et al., 2022a), and suggest that the scale at which particular abilities are observed in LMs is function of both the evaluation that is being used and the auxiliary capability of the model to handle the task demands of that evaluation. These conclusions complement the prior findings of Schaeffer et al. (2023).
We investigated two evaluation contrasts in each of two cognitive domains, which estab- lishes consistency across settings but are still only samples from the much broader literature on cognitive evaluation. More work will be needed to investigate the impacts of other popular cognitive evaluation techniques, such as in-context learning (Lampinen, 2023) or fhain-of-thoughhe prompting (Wei et al., 2022b). Similarly, we made an effort to include a representative group of openly accessible LMs (Frank, 2023b) released in multiple sizes or with multiple training checkpoints. With the exception of the Pythia family, all models showed qualitatively similar results, suggesting that our findings are at least somewhat general. Tt is unknown, however, how our pattern might vary across a broader scale of models; we speculate that the order of magnitude less data used in training the Pythia models might have resulted in the somewhat different performance we observed.
Our conception of what makes a particular task demand is informal. For example, the de- compositions given in Figure 1A rely on intuitive concepts such as " "question interpretation." In developmental psychology, we can reason about task demands using constructs such as memory, attention, or inhibition, supplementing with our knowledge about how these develop in childhood. In LMs, we judged task demands to be low when evaluations aligned with training objectives (as in measurements of token probability) or when they constrained the space of possibilities (as in forced choice tasks). A more granular notion of task demand in LMs might investigate the attentional and representational mechanisms triggered by different evaluations; we view this as a promising avenue for future work.
Another interesting (and speculative) direction suggested by our work is using LMs to study task demands in humans. The relative ordering of performance in LMs under different evaluation methods could reflect more general properties about how probabilistic predictions or representations of knowledge manifest in different evaluation settings. These insights could help us generate predictions about what kinds of methods might place greater demands on children. Indeed, recent studies are consistently showing that probability measurements reveal knowledge that can be masked by metalinguistic methods, suggesting that direct predictions might be the most diagnostic of underlying ability.
Finally, given the wide range of evaluation methods available LM researchers, and the stark differences that can emerge across methods, a natural question is which method is the "right" one to use. When designing an evaluation, we encourage researchers to think about the validity of using particular method to measure a particular construct of interest (Schlangen, 2019). As our work demonstrates, an important step is to specify the auxiliary demands that the evaluation might impose on the model, and how these demands interact with one's broader evaluation goals (e.g., trying to be adversarial, versus trying to reveal knowledge). Conversely, researchers should interpret model performance not as direct indication of some feature of intelligence (or lack thereof), but as a reflection of a model's true underlying capacities seen through the lens of researchers' design choices.
10

---

Published as a conference paper at COLM 2024
Acknowledgments
We would like thank Noah Goodman and Michael Franke for helpful discussion. This work has been made possible in part by gift from the Chan Zuckerberg Initiative Founda- tion to establish the Kempner Institute for the Study of Natural and Artificial Intelligence.
Ethics statement
Understanding language models' limitations and abilities is becoming increasingly impor- tant not only for academic research, but also for the general public. There can be dangers both to overclaiming and underclaiming the abilities of LMs (Bowman, 2022). This paper highlights task demands as potential threat to the validity of LM evaluation, which we hope will help researchers make more informed conclusions about LMs' cognitive capacities. This work could also help non-experts better understand the broader context in which LM evaluation results are collected and interpreted.
7 Reproducibility statement
We prioritized reproducibility in work. All of the tested models openly accessible through the Huggingface Transformers library. In addition, the training data for the Pythia and OLMo model families are publicly available. Inference was run on NVIDIA A100-40GB SXM GPUs during March 2024.
Our code and data are publicly available https:///ggthub..ccojjjnnuuuuuue demddadnds.
References
Elika Bergelson and Daniel Swingley. At 6-9 months, human infants know the meanings of many common nouns. Proceedings of the National Academy of Sciences, 109(9):3253-322 -22552888888 February 2012. doi: 10.1073/
10.1073/pnas.1113888 pttpp/aa.....01111111
URL https://doi.orgg// 1073/
1113380109.
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling, 2023. URL https:
//arxiv. org/ absrg//bbb/////// 20000//233.000 01373.
Samuel Bowman. The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Fail. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 7484-7499, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653 /1 22022.2cl--oong.515
URL https:tts/// hatpss////aaaahooo//o// orgoorg////// wttrg//222..oo acl- moll-oooo..16 516.
Samuel R. Bowman and George Dahl. What Will it Take to Fix Benchmarking in Natural Language Understanding? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4843-4855, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/ v1 1112022..aaclmmain. URL https: https://aclaanthologg/ org 202g:2221...o wwa.l-maiian 385.
Peter Carruthers. Mindreading in infancy. Mind & Language, 280))141-112,,201
Tyler A. Chang and Benjamin K. Bergen. Language Model Behavior:
A Comprehensive
Survey. Computational Linguistics, 50(1):293-350, March 2024. ISSN 0891-2017. doi: 10.1162/
URL https://doi..rge 01200
Lee J Cronbach and Paul E Meehl. Construct validity in psychological tests. Psychological Bulletin, 52(4):281, 1955.
11

---

Published as a conference paper at COLM 2024
Vittoria Dentella, Fritz Günther, and Evelina Leivada. Systematic testing of three Language Models reveals low language accuracy, absence of response stabillty, and a yes-response bias. Proceedings of the National Academy of Sciences, 120(51))e23399588120, December hwt.00//pp/.../// 2023a. doi: 10.1073/ pnas. 2mass2309583112 URL. hnttps:///ol http://ooooog/////////
Publisher: Proceedings of the National Academy of Sciences.
Vittoria Dentella, Elliot Murphy, Gary Marcus, and Evelina Leivada. Testing Al performance on less frequent aspects of language reveals insensitivity to underlying meaning, 2023b. URL https: wrw. wtgg/bs/////eeo/e//.///eee htsp//bbs./.....////////// 12313.
Chaz Firestone. Performance vs. competence in human- comparisons. Proceedings of the National Academy of Sciences, 117(43))2562--6671 October 2020. URL https:// // Noi.org/10.. / 1905334117. Publisher: Proceedings of the National Academy of Sciences.
Michael C Frank. Baby steps in evaluating the capacities of large language models. Nature Reviews Psychology, 268):451-45,, 2023a.
Michael Frank. Openly accessible lims can help us to understand human cognition. Nature 2023b.
Human Behaviour, 7(11)::825-1187,, 2023b.
Shane Frederick. Cognitive Reflection and Decision Making. Journal of Economic Perspectioes, 19(4):25 42, 2005. doi: 10.1257/08953330577511666 Stt.r/dcco..122/8883333777772221
aeaweb. URL https: wwwwb./ org/
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Isai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette- Choo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christian Muraru, Grigory Rozhdestvenskiy, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikula, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Pier Giuseppe Sessa, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross Mcllroy, Ruibo Liu, Ryan Mullins, Samuel L. Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojciech Stokowiec, Yu-hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy. Gemma: Open Models Based on Gemini Research and Technology, 2024. URL https://arxivv oww.obgg/////... 240./ 08295.
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, lan Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Py- atkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richard- son, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Han- naneh Hajishirii. OLMo: Accelerating the Science of Language Models, 2024. URL http://aaxivv.orr/bbbbbb/ 00838.
Thilo Hagendorff, Sarah Fabi, and Michal Kosinski. Human-like intuitive behavior and reasoning biases emerged in large language models but disappeared in Chat-
12

---

Published as a conference paper at COLM 2024
GPT. Nature Computational Science, 3(10):833-83) October 2023, ISSN 2662-8457. doi: 10.1038 -23588-023-000227 URL https: /ttp:://doo.oo//0///
Petra Hendriks and Charlotte Koster. pooouuction/cmppreeenntt asymmetriess in language acquisition. Asymmetries in Language Acquisition, 120(8):1887-1897, August 2010. ISSN
0024-3841. doi: /ttp0///...nguaa200.002202 Sccience/arricceee////944444844
URL https://ww..cciencedreee com/
Jesse Hoogland, Alexander Gietelink Oldenzeel, Daniel Murfet, and Stan Wingerden. Towards interppplennallnnteerrttilllllyy 2023. URL https: //www. wesswrong........... htt.swrong.oo////c/ posts/
Jennifer Hu and Roger Levy. Prompting is not substitute for probability measurements in large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5040-5060, Singapore, December 2023, Association for Computational Linguistics. doi: 10.18653/v1/ 2023. 100.eemllp-mai.33
URL https: https://aaclaathhology .orgooooogooooooooooooogooo wrg/orgg/33322 urgg/222.emmlpp-
306.
Jennifer Hu, Kyle Mahowald, Gary Lupyan, Anna Ivanova, and Roger Levy. Language models align with human judgments on key grammatical constructions, 2024. URL
01676
Anna A. Ivanova. Running cognitive evaluations on large language models: The do's and the don't ts, 2023. URL 2 01276.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William Sayed. Mistral 7B, 2023. URL https://arxxi..rr//bbbt
06825.
Roni Katzir. Why Large Language Models Are Poor Theories of Human Linguistic Cognition: A Reply to Piantadosi. Biolinguistics, 17, 2023. URL https: //ts:://doo// wrw.oorg///0 0tt///...599//// 5964/bioling. 13153.
Carina Kauf, Emmanuele Chersoni, Alessandro Lenci, Evelina Fedorenko, and Anna A. Ivanova. Comparing Plausibility Estimates in Base and Instruction-Tuned Large Language Models, 2024. URL https://arxix: org.org///b/.. abs 14859.
Rachel Keen. Representation of objects and events: Why do infants look smart and toddlers look so dumb? Current Directions in Psychological Science, 12(3):79-88) 83, 2003.
Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. Dynabench: Rethinking Benchmarking in NLP. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4110-4124, Online, June 2021. Association for Computational Linguistics. doi: 10.18653. /v0l55vll/ / 2002 naacl-main.33 URL https://aclanthollog org/2021 naacl-main. 324.
Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwa- sawa. Large Language Models are Zero-Shot Reasoners. In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neu- ral Information Processing Systems, volume 35, pp. 22199-22213. Curran Associates, Inc., 2022. URL https://procceddngg..o neurips wwtpseiccpippcccppeprrreeee #bbed291acd4accPe hac.ccaereee1111ggg006666--aaaa0
ccppaper_ppppepeepcpeeeeeeeee paape_fille Tilee//pper/2222/F112 pdf.
Andrew K Lampinen, Ishita Dasgupta, Stephanie C Chan, Hannah R Sheahan, Antonia Creswell, Dharshan Kumaran, James L McClelland, and Felix Hill. Language models, like humans, show content effects reasoning tasks. PNAS Nexus, 3(7):pgae233, July 2024. ISSN 2752-6542. doi: 10.1093 htt.0///paanexxu/pgepee pgae233. URL https: / //oii org/10. 1093 mamasnexus/pgae233.
13

---

Published as a conference paper at COLM 2024
Andrew Kyle Lampinen. Can language models handle recursively nested grammatical structures? case study on comparing models and humans, 2023. URL https://arxin: org/abs/2210. 15303.
Tal Linzen. How Can We Accelerate Progress Towards Human-like Linguistic Generaliza- tion? In Proceedings of the 58th Annual Meefing of the Association for Computational Linguis- fics, pp. 5210-5217, Online, July 2020. Association for Computational Linguistics. doi: URL htttp:://aclanthology.oomm
acl iell-main.455 465.
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. Assessing the Ability LSTMs Learn Syntax-Sensitive Dependencies. Transactions of the Association for Computational Linguistics, 4:521-535, 2016. doi: 10.1162 tattttlaaaaa a__0015.. URL https: ltttps://aclanthoooog.cooo org/Q16-1033
Place: Cambridge, MA Publisher: MIT Press.
Chenyang Lyu, Minghao Wu, and Alham Fikri Aji. Beyond Probabilities: Unveiling the Misalignment in Evaluating Large Language Models, 2024. URL https://arxiv.orr abs/2402. 13887.
Rebecca Marvin and Tal Linzen. Targeted Syntactic Evaluation of Language Models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Langwage Processing, pp. 1192-1202, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: 10.18053/
URL httttp://aclannthllgg. org/D18-115.
Kanishka Misra. minicons: Enabling Flexible Behavioral and Representational Analyses of Transformer Language Models, 2022. URL hatps://arxiic ortg 2203. 13112.
Ida Momennejad, Hosein Hasanbeig, Felipe Vieira Frujeri, Hiteshi Sharma, Nebojsa Jojic, Hamid Palangi, Robert Ness, and Jonathan Larson. Evaluating Cognitive Maps and Plan- ning in Large Language Models with CogEval. In seirty--eventh Conference on Neural In- formation Processing Systems, 2023. URL
net. meet//oorumiid-VVttGGGGee. http://pperreeeeeteeeeerrrrrrrme
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial NLE: A New Benchmark for Natural Language Understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4885 4901, Online, July 2020. Association for Computational Linguistics. doi: 10.18653 2088553vv/2022 acl-main.441. URL https: http:////caaahhoogyyyy. org/2020. acl-main main.441.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring broad discourse context. In Katrin Erk and Noah A. Smith (eds.), Proceedings of the 54th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 1525-1534 Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v (0.1653/v1/PP1--144 URL https: https://aclanntoooogy Beceey.rgg/PP11 org/P16-1144.
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets, 2022. URL https:
//arxiv. wtw..org/abb////g/
Deborah Raji, Emily Denton, Emily M. Bender, Alex Hanna, and Amanda- lynne Paullada. Al and the Everything in the Whole Wide World Bench- mark. In J. Vanschoren and S. Yeung (eds.), Proceedings of the Neural In- formation Processing Systems Track on Datasets and Benchmarks, volume 1. Cur- neurips. cc/paper_files/ pdf.
ran, 2021. URL
Paul Röttger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Rose Kirk, Hinrich Schütze, and Dirk Hovy. Political Compass Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models, 2024. URL
https://arxil: www.iv.org/bbb/////./ 16786.
14

---

Published as a conference paper at COLM 2024
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An Adversarial Winograd Schema Challenge at Scale. Commun. ACM, 64(9);99-106, August 2021. ISSN 0001-0782, doi: 10.1145/3474535 URL https://dof org/10. 1145/3443881 Place: New York, NY, USA Publisher: Association for Computing Machinery.
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are Emergent Abilities of Large Language Models a Mirage? In Thirly-seventh Conference on Neural Information Processing Systems, 2023. URL
David Schlangen. Language Tasks and Language Games: On Methodology in Current Natural Language Processing Research, 2019. URL https: wwtp:iwwwwggg///////// 1900b////98...1 10747.
James B. Simon, Maksis Knutins, Liu Ziyin, Daniel Geisz, Abraham J. Fetterman, and Joshua Albrecht. On the Stepwise Nature Self-Supervised Learning. In Proceedings of the 40th Internalional Conference on Machine Learning, 2023. URL https://proceediing.. http://roceednns..l....e...eee mlr. mll..ress////
20002//mmm22/////e simon22a..mmmmmmmmmmmmmmm...33.m ..df.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. lyer, Anders Johan Andreassen, Andrea Madotto, Andrea Santilli, An- dreas Stuhlmüller, Andrew M. Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakaş, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartlomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Howald, Bryan Orinion, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, Cesar Ferri, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burch, Christopher Waites, Chris- tian Voigt, Christopher D. Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, C. Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong- Ho Lee, Dylan Schrader, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodola, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A. Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Funice Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fate- meh Siar, Fernando Martinez-Plummed, Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Xinyue Wang, Gonzalo Jaimovitch-Lopez, Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannaneh Ha- jishirzi, Harsh Mehta, Hayden Bogar, Henry Francis Anthony Shevlin, Hinrich Schuetze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernández Fisac, James B. Simon, James Koppel, James Zheng, James Zou, Jan Kocon, Jana Thompson, Janelle Wingfield, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Batchelder, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez-Orallo, Josel Boudeman, Josephh Guerr, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh Dhole, Kevin Gimpel, Kevin Omondi, Kory Wallace Mathewson, Kristen Chiafullo, Kse- nia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao,
15

---

Published as a conference paper at COLM 2024
Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras-Ochando. Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros- Colón, Luke Metz, Lütfi Kerem Senel, Maarten Bosma, Maarten Sap, Maartje Ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramirez-Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L. Leavitt, Matthias Hagen, Mátyás Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael Andrew Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michal Swędrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suz- gun, Mitch Walker, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan Andrew Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S. lyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormollaashii, Peiyuan Liao, Percy Liang, Peter W. Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang, Piotr Milkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramon Risco, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan Le Bras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Russ Salakhutdinov, Ryan Andrew Chi, Seungjae Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel Stern Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepi- deh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshmiwal, Shyam Upadhyay, Shyamolima Shammie Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven Piantadosi, Stuart Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tat- sunori Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Venkatesh Ramasesh, vinay uday prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saun- ders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, Zirui Wang, and Ziyi Wu. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL www.nee///ee. peee//oormmidduyTlSSst
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremmy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open Foundation and Fine-Tuned Chat
16

---

Published as a conference paper at COLM 2024
Models, 2023. URL https::/arxxxv.oor////////////
Polina Tsvilodub, Hening Wang, Sharon Grosch, and Michael Franke. Predictions from language models for multiple-choice tasks are not robust under variation of scoring methods, 2024. URL https://arxivv http://wwwww//////////e/e httg/bbss22...../ 00998.
Esra Nur Turan- Tuçkn-Küçük and Melissa M Kibbe. Three-year-olds' ability to plan for mutually exclusive future possibilities is limited primarily by their representations of possible plans, not possible events. Cognition, 244: 244:105712, 2024.
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. BLiMP: The Benchmark of Linguistic Minimal Pairs for English. Transactions of the Association for Computational Linguistics, 8, 2020. URL https://dot: org/ 10. 1162//acl_a_0001111 00321.
Publisher: MIT Press.
Taylor Webb, Keith J. Holyoak, and Hongjing Lu. Emergent analogical reasoning in large lan- guage models. Nature Human Behaviour, 799:1122615541 September 2023. ISSN 2397-3374. doi: 10.1038/ 0 URL / 1058/ -1699w..
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent Abili- ties of Large Language Models. Transactions on Machine Learning Research, 2022a. URL https://openrevii/ wwtp:/nn//forunnidyyssssssseee
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain of Thought Prompting Elicits Reasoning in Large Language Models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022b. URL https://openreviee
Peter West, Ximing Lu, Nouha Dziri, Faeze Brahman, Linjie Li, Jena D. Hwang, Liwei Jiang, Jillian Fisher, Abhilasha Ravichander, Khyathi Chandu, Benjamin Newman, Pang Wei Koh, Allyson Ettinger, and Yejin Choi. The Generative Al Paradox: "What It Can Create, It May Not Understand". In The Twelfth International Conference on Learning Representations, 2024. URL https: http://ppppw/e.ee.eeeeeeee net www.:nne//ooommm CFCF 8H8MS5P8.
17


Reviews:
[{"review_id": "igBL6SSTKV", "reviewer": "Reviewer_qy4S", "summary": "This paper investigates how \"task demands\" in evaluating language models can mask their underlying cognitive capabilities. It argues that evaluations with greater auxiliary task demands beyond the targeted cognitive construct tend to yield lower performance, especially for weaker models. The authors adopt two evaluation contrasts to uncover the influence of task demands: 1) production vs forced choice, and 2) metalinguistic judgments vs direct probability measurements. These contrasts are applied to four test scenarios and five LLM families. The experimental results demonstrate that there is a \"demand gap\" where the higher-demand task results in lower performance compared to the lower-demand task, and this \"demand gap\" shrinks as model size/training increases.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your encouraging comments and helpful suggestions. \n\nRegarding your first point, we do acknowledge that the paper is currently a bit informal in its definition of high- and low-demand tasks. This informality follows the psychology literature, which does not provide clarity on what does and doesn’t count as a task demand. However, our exploration provides an opportunity for greater clarity, for example by defining task demands as directly related to a specific evaluation as opposed to transfer to other tasks. In fact, we view our work as providing potential value to psychologists by giving a tool for identifying task demands in future: task demands are evaluation-related demands that asymmetrically impact less capable agents (whether models or humans). \n\nRegarding your second point, we focused on these evaluation methods because they are relevant to many, if not the vast majority of current behavioral LM evaluations. Researchers have the choice to formulate their task as a prompt or a direct measurement of string probabilities, and independently, they also have the choice to evaluate the LM in a generative (production) or discriminative (forced choice) setting. In terms of cognitive constructs, we chose word prediction and grammaticality judgment because they are perhaps the clearest cases where metalinguistic prompting differs from clearly established ways of using direct measurements, but of course there are many other domains where this contrast is relevant and can lead to different results (e.g., semantic plausibility; see Kauf et al. 2024 or Hu & Levy 2023). \n\nFor the production versus forced choice contrast, we chose the analogical and reflective reasoning domains because they have been of great recent interest in LM evaluation (per recent high-profile papers on these topics that have received a great deal of interdisciplinary attention), and because each of these datasets has a well-motivated set of answer options to be used for forced-choice evaluation. We will elaborate on all these points in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper explores and verifies an important issue, the \"task demands\" in assessing LLMs. Its core idea is that different abilities should be disentangled, so that other basic abilities like \"interpreting questions\" do not influence the measurement of the target ability. Studying \"task demands\" is important, as it can help us better understand the true performance when benchmarking LLMs. I have learned valuable knowledge from this paper, and I believe the NLP community should be broadly interested in it.\n\n2. This paper is based on a concrete and traceable theory from developmental psychology, extending from assessing children to evaluating LLMs, which is reasonable and insightful. Considering that LLMs are currently widely used as human-like autonomous agents, this transferable research is worth encouraging.\n\n3. The authors select four test scenarios (cognitive constructs) and considered two sets of high- and low-demand evaluation contrasts. They conduct extensive experiments on five LLM families (a total of 13 LLMs), demonstrating the existence of \"demand gaps\" and their shrinkage as the model size/training increases. The main claims are well supported by the experimental results.\n\n4. The paper is very well-written and engaging.", "reasons_to_reject": "Overall, this paper is good, but it appears to be under-explored or arbitrary on the following two points:\n\n1. There is a lack of a clear definition and boundary of high-demand and low-demand tasks. What tasks are high-demand, and what are low-demand? Furthermore, what abilities are introduced by high-demand tasks that are not involved in low-demand tasks? Although the authors list some abilities like \"interpreting questions\", a complete and clear taxonomy or something similar is still missing. It is encouraged to supplement relevant psychological knowledge.\n\n2. The selected evaluation methods (production, forced choice, metalinguistic judgment, probability measurement) and cognitive constructs (analogical reasoning, reflective reasoning, word prediction, grammaticality judgment) lack motivational explanations. It is unclear whether these selections are representative or can be generalized to other scenarios.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GXILnVG1Ti", "reviewer": "Reviewer_11db", "summary": "The authors study how task demands impact language model evaluation. Their approach is motivated from a developmental psychology perspective where it is well established that children might fail certain tasks due to higher task demands and not because they lack the capacity of interest. Applying this framework to language model evaluation, the authors ask whether task demands have a bigger effect on smaller models, thereby masking their “true capabilities”. \n\nThe authors predict a demand gap, i.e., that less capable models (smaller models or those trained for less steps) suffer more from higher task demands, i.e., achieve lower performance, than more capable models. \n\nIn the experimental section, the authors evaluate 5 different model families on 4 different datasets while controlling for task demand. The experimental findings are mostly consistent with the demand gap prediction. \n\nFinally, the authors discuss the implications of their work for LM evaluation, the study of emergence, and even developmental psychology.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful feedback and suggestions. \n\nWe will discuss the behavior of the Pythia models in more detail in the revision. We only tested a single prompt for the metalinguistic evaluation tasks for reasons of space, and agree that the results would be stronger if multiple prompts were considered. We refer readers to Hu & Levy 2023, who tested models on 3 different metalinguistic prompts, in both English and Chinese, and found similar qualitative findings (prompting underperforms probability measurements).\n\nRegarding your questions, we are only measuring the logits assigned to answer options conditioned on the problem/prefix; the text of the options themselves are not part of the input during forced choice eval. We also appreciate your suggestion to discuss the limitations of the evaluation methods. We will add this to the revision as space allows.\n\nYou raise an interesting point about data contamination and LAMBADA. It is possible that in this setting, the low-task demand method might actually be measuring memorization rather than the target task of next-word prediction, especially for larger models with greater memorization capacities (Carlini et al., ICLR 2023). But memorization concerns also affect the high-task demand method: if a model has actually memorized the text of LAMBADA, then it should also be able to retrieve this text under the metalinguistic prompt evaluation. One could potentially argue that the presence of the prompt would make the context less similar to what has already been seen, so the memorization effect might be weaker. This is an important empirical question for future work (we would also appreciate references if this has already been studied!).\n\nFinally, we appreciate your suggestions about error analysis and answer ranks. We hope to explore these directions as space and time allow.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper studies an important problem and is exceptionally well written. The research is well motivated and the authors present a clear hypothesis for what they expect to find. \n\nThe experiments are well designed and sufficiently broad: testing 5 different model families, considering different sizes for every model (except Mistral), using 4 different datasets and 4 different ways of evaluation. \n\nThe discussions are extensive and the authors do a great job of connecting their work to the previous literature as well as discussing the implications of their findings. \n\nOverall, this is a great paper and I really enjoyed reading it.", "reasons_to_reject": "I only have two minor concerns and they are not strong enough to serve as reasons for rejection. I will still mention them here. \n\n- The Pythia models clearly behave as an outlier for 3 out of 4 evaluations. The authors could do a better job at discussing and investigating what is going on here (they briefly speculate about the number of training steps but I’m not fully convinced by that).\n\n- The meta linguistic evaluation considers only a single prompt (for each of the tasks where it is applied). It is however widely established that prompt engineering can have a dramatic influence on model performance. These results would be even more convincing if they would take different prompts into consideration.", "questions_to_authors": "**Evaluation methods**\n- Are the answer options part of the input during forced choice eval? Or are you only extracting the logits from the answer tokens? This was not entirely clear to me. \n- It would be interesting to discuss the limitations of each evaluation method a bit more and also talk about whether they can be applied to any task. As an example, direct evaluation assumes that we already know the answer and therefore might not be applicable at test time.  \n\n**Data contamination** \n- While I agree that performance per se is not so important here but rather the difference in performance, data contamination might impact the capabilities you are actually evaluating. If we assume that all models saw e.g., LAMBADA and bigger models are better on it, are we evaluating recall from memory or the actual task? I feel this could be discussed a bit more. \n\n**Error analysis for production and meta-linguistic (in the word prediction case)**\n- It would be interesting to look at the errors the models make in this case and see whether there are interesting patterns here. \nMaybe this could help to understand in which sense the model is misinterpreting or misunderstanding the task?\n\n**Figure 4**\n- It could be interesting to look at the rank of the correct answer (for the word prediction task) as a function of the training steps. \n\n**Pythia results**\n- The difference to other models might also be explained by the training data. I feel this aspect could in general be discussed a bit more.  \n- It would be interesting to see the checkpoint evaluation (Figure 4) for pythia models as well.", "extra_scores": {}}, {"review_id": "jlG7cwqwMo", "reviewer": "Reviewer_xeMC", "summary": "This paper makes an interesting contribution to the study of LMs by looking at task demands as a potential moderator variable for task performance. \nThe authors conduct a series of tests including varying the amount of training and find evidence that task demands do indeed pay a role.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the suggestion to add statistical testing. The key effect we would like to test for is the interaction between model size and task demands, and our prediction (visualized in Figure 1) is an interaction between these two, corresponding to the “demand gap” such that smaller models are differentially affected by task demands. \n\nA mixed effects model (in which effects can vary across groupings – in our case, model families) is most appropriate for our data. We fit the following model in R using lme4 (the standard package for mixed effects modeling) for each of the domains with binary responses (analogical reasoning, reflective reasoning, and grammaticality judgment):\n\n`glmer(correct ~ log_n_params * eval_method + (log_n_params * eval_method | model_family), family=binomial(link=\"logit\"))`\n\nAnd we fit the following model for the word prediction domain:\n\n`lmer(final_word_logprob ~ log_n_params * eval_method + (log_n_params * eval_method | model_family)`\n\nThe interaction between number of parameters (log_n_params) and evaluation method (eval_method) was significant for each domain except reflective reasoning.\n\nFor the training time analyses (for which there was no model grouping factor), we fit standard GLM models for the analogical reasoning and word prediction domains, respectively:\n\n`glm(correct ~ log_training_step * eval_method, family=binomial(link=\"logit\"))`\n\n`lm(final_word_logprob ~ log_training_step * eval_method)`\n\nIn both models, we found a significant interaction between the number of training steps (log_training_step) and evaluation method.\n\nWe will add these analyses and results to the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper of well written, clear and does well in connecting a well known and rather basic concept from developmental psychology with LMs. \nThe tests conducted are adequate and the conclusions floor logically from the findings.", "reasons_to_reject": "I only have one concern (which I think can and should be addressed). The analysis is incomplete as it is essentially a descriptive analysis without any formal statistical testing. \nThe authors could address this by looking at simple analyses such as an ANOVA or other models of the GLM family. \nThis kind of Testing is not only desirable but essential: looking at individual comparisons without statistical justification to do so is incorrect.", "questions_to_authors": "-", "extra_scores": {}}, {"review_id": "Wzc24uCXK7", "reviewer": "Reviewer_cyq6", "summary": "This paper looks at whether a \"demand gap\" exists in tasks designed for LMs, whether this gap is larger for \"simpler\" models, it draws on a cognitive analogy.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and detailed comments.\n\nThe first point concerns competence/performance distinctions. As originally conceptualized, C/P distinctions were a way to justify studying the latent construct of human linguistic representations, rather than the error-prone behaviors that people often produce. As you note, initial connectionist models were celebrated for their ability to reproduce the full distribution of observed behaviors, thus reducing the need to explain away specific *human* behaviors as purely performance-based. \n\nIn our work here, we apply this distinction to *models* instead of humans –  asking whether our evaluations of their manifest behaviors fail to reveal their latent knowledge (cf. Firestone 2020; Lampinen 2023). In other words, we were trying to use the C/P distinction as a way to understand the “construct validity” (to borrow a different psychological term) of LM evaluations. Perhaps it would be less confusing just to talk about observed behavior vs. latent constructs and not C/P, given the historical context you’ve brought up. \n\nSecond, as we state in Section 3.3, we don’t draw direct analogies between model size and child development. We use size/training as proxies for how LMs might gain resources that help with overcoming task demands. This may look very different for children vs LMs: e.g., children might acquire the ability to inhibit responses, and LMs might learn how to respond to prompts. Thus, the analogy between childhood and LM size is merely conceptual, but we can still make specific, testable predictions about patterns like the demand gap. We’ll clarify this in the revision. \n\nThird, we turn to the relationship between forced choice (FC) vs production. Conditioned on a context (e.g., analogical reasoning problem), the LM’s ability to solve the task given in the context is encoded in its distribution over subsequent tokens. This distribution is the same, whether the LM is evaluated through FC or production. The sampled response might not demonstrate the LM’s abilities because of auxiliary factors that affect decoding, whereas comparisons of answer options might reveal distinctions that are present in the LM’s distribution but would not be sampled. In this sense, production and FC can target the same cognitive construct, while imposing different auxiliary demands. We will elaborate on this as space allows.  \n\nFinally, we appreciate your suggestion about math word problems. We hope to try this at some point.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper raises an important point, that task effects may impact different models differently, and that there's a bias in how great this effect is given model size and training size. It reveals a strength of larger models (a robustness) or conversely, a weakness in the smaller models. Researchers should take this into account both for their evaluations and for deciding which models to deploy for their tasks.\n\nI appreciated the explicit paragraph on data contamination, and I which that were more standard that it is in NLP papers these days.\n\nThe 1st, 2nd, and 5th paragraphs in the discussion were cogent and really summarized the paper and its important conclusions well.", "reasons_to_reject": "I find it odd that the abstract assumes a competence-performance distinction in LMs. The distinction is widely but not universally accepted for humans in cognitive science, but it has been explicitly argued for decades that connectionist models (=> modern LMs) do not support such a distinction. Supporters of connectionist models of cognition saw/see this as a virtue. The consequence of this is that properties of the models relating to their architecture, eg number of parameters in this paper, and practical challenges emerging from those decisions, are fundamental to their \"competence.\" They cannot be disentangled. While this doesn't evaluate the paper, it does bring the general narrative, as well as analogy with humans, into question.\nTake a look at Schwarz (1992, Connection Science), and Allen & Seidenberg (2013, Emergentist Approaches to Language) for examples.\n\nBuried under 3.3, there's a short paragraph disavowing direct parallels between child development and model scaling. This is of course true. Children doesn't grow more neurons as they develop, let alone by a factor of 70. However, since model size is one of the two measures of model power that this paper looks at, this severely undermines the analogy between childhood development and model power that the authors make throughout this paper. At the very least, this caveat should be brought up into the introduction, where I first noticed this problem, but a better solution would be to downplay the cognitive analogies being made here. That would also help solve the (lack of) competence-performance distinction issue raised above.\n\nRegarding demonstrating the \"signature pattern,\" this relies both on the theoretical assumptions about competence-performance and also practical assumptions surrounding the demand gap. Is forced choice, which is often effectively classification, really the \"same\" cognitive construct as the production equivalent? They're certainly very different tasks from a traditional ML perspective. If they aren't, then what is being measured isn't really the \"demand gap\" as stated. They're two different competencies. \n\nRelated to whether or not they're the same task further assumes that the task setups are in fact evaluating the problem that the researchers think they are. This is particularly crucial in the forced choice setting, which the authors sort of get at in footnote 2, but they don't take it far enough. \n\nOne example of this that I found particularly compelling was the following, which looked at VQA with multiple choice answers. It turns out that their model achieved most of its performance without even being exposed to the image, which means that it was simply not actually doing VQA as intended. Instead it was picking up on biases in the Q and Q construction. It took aggressive debiasing of the data set to create an evaluation where the authors were confident that the model was actually performing VQA\nWei-Lun Chao, Hexiang Hu, and Fei Sha. 2018. \"Being negative but constructively: Lessons learnt from\ncreating better visual question answering datasets.\" NAACL\n\nThere was a more recent paper making the same point with regards to BLiMP. Because of the forced choice task setup and researcher-generated sentence pairs, models seem to be able to solve BLiMP without actually engaging in grammaticality. This is particularly relevant, since the authors rely on BLiMP for one of their evaluations.\nHéctor Vázquez Martínez, Annika Lea Heuser, Charles Yang, and Jordan Kodner. 2023. \"Evaluating Neural Language Models as Cognitive Models of Language Acquisition.\" GenBench\n\nA better \"demand gap\" set up might, for example, compare solving math word problems with a lot of filler text and complex scenarios on one hand and more straightforward word problems on the other hand. These are more clearly the same ML task.\n\nGiven all of the above, the 2nd past paragraph of the discussion comes off as probably nonsense. It should be removed.\n\nTypo at the bottom of page 5 \"prompted asked\"", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper argues that interpretation of LM performance should factor in task demands - factors extraneous to the ability being evaluated that might nonetheless influence model scores, such as the need to translate predictions across multiple different data formats. Specifically, the authors show that task demands have a particularly large influence in small LMs, potentially unfairly decreasing our impressions of how capable such systems are.
The reviewers noted many strengths:
- The paper shows how some ideas from psychology can be valuable for work about understanding LLMs. Reviewers noted that they found these ideas helpful and that others in the - NLP community would likely find them helpful too.
- The experiments are well-constructed and have a clear takeaway message (namely, that task demands have a particularly large influence in small models).
- The experiments are comprehensive
- The paper is well-written and enjoyable to read

Some important limitations were also noted:
- It is unclear what determines how high-demand a task framing is
- More exploration of prompting styles could have been helpful, given the sensitivity to prompts that LLMs display
- The competence/performance distinction has a complex history in connectionism that could have been helpful to discuss in more detail 
- The particular selection of task framings has some potential limitations - as noted by reviewer cyq6, some of the pairs that are studied might get at different competences from each other

All reviewers rated this paper above the acceptance threshold, most of them substantially so. I am recommending acceptance because this paper presents important ideas clearly and with convincing experimental support. The statistical test that the authors raised in a response would further strengthen the paper. To the extent that space permits, I encourage the authors to add to the paper the points they made in response to reviewer 11db about prompt engineering and in response to reviewer cyq6 about the competence/performance distinction and about the extent to which the different task framings target the same cognitive construct.
Decision: Accept