I am a Strange Dataset: Metalinguistic Tests for Language Models (2024)

Abstract: Statements involving metalinguistic selfreference (“This paper has six sections.”) are prevalent in many domains. Can current large language models (LLMs) handle such language? In this paper, we present “I am a Strange Dataset”, a new dataset for addressing this question. There are two subtasks: generation and verification. In generation, models continue statements like “The penultimate word in this sentence is” (where a correct continuation is “is”). In verification, models judge the truth of statements like “The penultimate word in this sentence is sentence.” (false). We also provide minimally different metalinguistic non-self-reference examples to complement the main dataset by probing for whether models can handle metalinguistic language at all. The dataset is hand-crafted by experts and validated by non-expert annotators. We test a variety of open-source LLMs (7B to 70B parameters) as well as closed-source LLMs through APIs. All models perform close to chance across both subtasks and even on the non-self-referential metalinguistic control data, though we find some steady improvement with model scale. GPT 4 is the only model to consistently do significantly better than chance, and it is still only in the 60% range, while our untrained human annotators score well in the 89–93% range. The dataset and evaluation toolkit are available at https://github.com/ TristanThrush/i-am-a-strange-dataset.

heading: 1 Introduction
text: Self-reference plays a crucial role in the way we think about mathematics (Gödel, 1931; Tarski, 1931), theoretical computer science (Church, 1936), recursive programming (Hofstadter, 1979), understanding complex cases in hate speech detection (Allan, 2017), aptitude tests (Propp, 1993), and comedy (Hofstadter, 1985). Some positions in the philosophy of mind consider self-referential capabilities to be a key aspect of higher intelligence
or even consciousness (Hofstadter, 2007; Baars, 1993). Of course, self-reference is also pervasive in how we communicate: at least one paper you read today is bound to contain “In this paper” (Thrush et al., 2024).
In this paper, we focus on metalinguistic selfreference, the complex kind of self-reference in which language is used to make claims about itself, as in “This sentence has five words” and “This paper has six sections”.1 Using such language involves reasoning about metalinguistic properties (counting words, naming parts of speech, etc.) and resolving self-reference. Humans generally have no trouble with such language, and may even enjoy its playful and sometimes paradoxical nature (Hofstadter, 1979, 1985, 2007).
Recently, Large Language Models (LLMs) have demonstrated striking cognitive capabilities (Radford et al., 2019; Brown et al., 2020; OpenAI, 2022, 2023; Anthropic, 2023; Touvron et al., 2023; Jiang et al., 2023; Zhu et al., 2023). But do they have the same mastery over metalinguistic self-reference as we do? See Figure 1 for an example of the issue that LLMs face. To help address this question, we present a new task and dataset called “I am a Strange Dataset”. We are inspired by Douglas Hofstadter’s explorations of self-reference in language (Hofstadter, 1979, 1985, 2007), and borrow part of the name from one of his books: “I am a Strange Loop” (Hofstadter, 2007).
1Sentences like “I am Douglas Hofstadter” are selfreferential but not metalinguistic in the sense of interest here.
8888
An example in “I am a Strange Dataset” is comprised of two self-referential statements that begin in the same way but have different endings (Figure 2). One is true and one is false. Crucially, the ending flips the truth value of the overall statement. There are two subtasks: generation and verification. In generation, the model must generate the true statement and reject the false one. In verification, models judge the truth of completed statements. To complement the main self-referential data, the dataset also contains metalinguistic nonself-reference examples. These are minimally different from the main examples and serve as controls to assess whether models can reliably handle metalinguistic statements in the absence of selfreference. In addition, all the examples in the dataset are tagged by expert annotators to further aid in error analysis.
“I am a Strange Dataset” is validated by nonexpert annotators. As a group, they have agreement rates in the 89–93% range, depending on which metric we use, as compared to chance rates at 50%. This further supports the claim that metalinguistic self-reference is relatively easy for humans. LLMs, by contrast, struggle: “I am a Strange Dataset” turns out to be so difficult that models are generally near chance both in generation and verification, and do not even succeed in the prerequisite metalinguistic non-self-reference case. That said, we do find some limited evidence that GPT 4 is getting some traction on the dataset: it is significantly above chance on all tested metrics (and seems to struggle especially with the self-referential data as compared to the non-self-referential controls). However, overall, it seems safe to say that “I am a Strange Dataset” poses a serious challenge for even the best present-day models.

heading: 2 Related Work
text: AI Challenges. We present our dataset as a challenge for the AI community. There are a range of AI stress tests and probes that use schemas targeting coreference resolution (Levesque et al., 2012; Sakaguchi et al., 2020), pronoun resolution (Rudinger et al., 2018), word order (Sinha et al., 2021; Thrush et al., 2022; Yuksekgonul et al., 2023), syntax (Linzen et al., 2016; Gulordava et al., 2018; Gauthier et al., 2020a; Hu et al., 2020), and interactions between syntax and semantics (Kann et al., 2019; Thrush et al., 2020). Although the schema for these tests can be simple to describe,
the knowledge required to solve the problems need not be. “I am a Strange Dataset” follows a simple schema that requires self-referential language, and consequently tests an array of metalinguistic capabilities. As far as we know, this is the first AI challenge dataset targeting metalinguistic selfreference, although there are metalinguistic evaluations (Hu and Levy, 2023; Behzad et al., 2023) and corpuses (Wilson, 2012; Kranzlein et al., 2024).
Self-reference. Ideas involving self-reference have been used to boost LLM performance. LLMs can verify their own outputs either via extra passes of natural language generation (Weng et al., 2023; Huang et al., 2023) or by writing code (Zhou et al., 2023). LLMs can also enhance their own inference code to some degree (Zelikman et al., 2023). Much of the previous work on self-reference with LLMs is about a model improving on itself or its outputs. Our dataset is not about that – it is complementary work about a model’s ability to generate and in-
terpret metalinguistic and self-referential language. Still, a statement that refers to itself could be about a model that generated it, too: “This sentence was generated by me, HAL9000”.

heading: 3 I am a Strange Dataset
text: In this section, we describe how the dataset is constructed and how we measure model performance.

heading: 3.1 Dataset
text: We aim to test whether a language model can produce and understand self-referential statements, and has the required metalinguistic capabilities. For example, consider this incomplete statement:
The penultimate word in this sentence is . . .
If we did not understand metalinguistic selfreference, we might complete the sentence with the word “sentence”. It is true that “sentence” is the penultimate word before adding more text, but by writing “sentence”, we have just changed the penultimate word! Here, a correct way to complete the statement is by inserting “is”. Completing statements is an established task format for language models (Paperno et al., 2016), but as far as we know, we are the first to apply it to metalinguistic tasks. Concretely, the schema for examples in our dataset is as follows:
• There is a self-referential statement which must be completed by adding text to the end.
• There are two candidate strings, with the same number of words, that can be used to complete the statement:
1. One of the candidate strings would make the statement true if the statement refers to itself before the addition of the string, but false if it refers to itself after adding the string. An example is the answer “sentence” above.
2. The other candidate string would make the statement true if the statement refers to itself after the addition of the string, but false if it refers to itself before the addition of the string. An example is the answer “is” above.
The dataset was created by four expert annotators each with several years of experience in computer science, linguistics, and/or cognitive science and all living in the United States. Each of the experts were given the schema and encouraged to be as creative as possible. Overall, the dataset is
comprised of 208 examples, and split into 200 examples for the evaluation set, 3 examples for few shot prompts, and 5 examples for use in an onboarding task for non-expert human validators.
There are 10 additional examples that are completely separate from these 208 examples, which we call “I am an Impossible Dataset”. They left even expert annotators stumped until they were given an explanation. We provide examples and GPT 4 responses in Appendix B and leave it as inspiration for a future challenge.

heading: 3.2 Tags
text: After the dataset was created, an expert annotator came up with a set of 10 tags with which to categorize all of the examples. By using this set, we 1) ensured that there are at least 20 examples for each tag, and 2) captured aspects of the mental facilities that an expert annotator noticed when they tried solving the problems. We show the example counts for each tag in Table 1, along with representative examples from the dataset. Each example can have more than one tag. Notice that the Sensory tag example in Table 1 is also Hypothetical, the example for the Existence of Element tag is also a Grammaticality example, and so on. Below, we describe the knowledge categories for each tag.
1. Negation & Scope. Understanding of words such as all, some, most, none.
2. Numerical Operations. Arithmetic (e.g. multiplication, addition, counting, subtraction). It is used only if arithmetic is explicitly mentioned.
3. Location of Element. Where items are located in a sentence relative to everything else.
4. Sub-Word. Understanding of characters, morphemes, syllables, and other word components.
5. Sensory. Perceptual knowledge about how emojis look, how words are arranged visually, how words sound, how something might taste, etc.
6. Existence of Element. Whether an element is present in a statement.
7. Grammaticality. Knowing grammar terms. 8. Multi-Channel. Knowledge of at least two
mediums. A medium might be Python code, English, Hebrew, C code, internet slang, etc.
9. Hypothetical. Reasoning about hypotheticals. 10. Question. A question is involved.

heading: 3.3 Metrics
text: We want to test whether models can generate and understand self-referential and metalinguistic statements. To this end, we present several metrics.

heading: 3.3.1 Generation
text: The primary capability that we want to test (and seemingly, the hardest) is whether language models generate true self-referential statements with greater likelihood than false ones. To test for this, we take an example from the dataset and compare the losses of the continuation that makes the overall statement true versus the continuation that makes the overall statement false. If the loss of the correct continuation is lower, then the model is said to have gotten that example correct, otherwise it is incorrect. Comparing a language model’s surprisal of an incorrect continuation versus a correct continuation is a common method used to test for syntax-comprehension (Linzen et al., 2016; Gauthier et al., 2020b) and reasoning (Gao et al., 2021; McKenzie et al., 2022). Surprisal is generally proportional to the loss, L, of the language model in our case. So, we define the generation score for an example’s beginning b, true ending et, and false ending ef , as given by Eq. 1.
g(b, et, ef ) = { 1 if L(et|b) < L(ef |b) 0 otherwise (1)
The generation metric does not use a prompt. It is based on the loss that a model assigns to continuations, given only the beginning of a statement.

heading: 3.3.2 Validation
text: A secondary capability that we want to test is whether a language model can at least correctly
judge a given self-referential statement as true or false. To test for this, we include the self-referential statement in a prompt along with instructions that tell the model to answer whether the statement is true or not. In principle, the instructions could be anything. For our experiments, we write a standard zero-shot (ZS), few shot (FS), and chain of thought (CoT) (Wei et al., 2022) prompt. We provide the full prompts in Appendix A.
For the ZS and FS prompts, we use the method established for the Generation metric above, except this time we compare the loss of “False” to the loss of “True”. Overall, the FS and ZS validation score for an example’s true prompt pt (i.e. the true full sentence plus any instructions), and false prompt pf , is given by Eq. 2. The blue parts are associated with correct model judgements and the red parts are associated with incorrect ones.
v(pt, pf ) =    1 if L(“True”|pt) < L(“False”|pt) and L(“True”|pf ) > L(“False”|pf ) 1 2 if L(“True”|pt) < L(“False”|pt) and L(“True”|pf ) ≤ L(“False”|pf ) 1 2 if L(“True”|pt) ≥ L(“False”|pt) and L(“True”|pf ) > L(“False”|pf )
0 if L(“True”|pt) ≥ L(“False”|pt) and L(“True”|pf ) ≤ L(“False”|pf )
(2)
We can compute the FS and ZS validation scores differently. Above, we compare the loss of “False”
versus “True”, given one context at a time. We can also compare the ratios of the “True” and “False” loss, in the false versus true contexts. We call this the relative validation score because it compares a model’s judgement for the truth of one sentence in an example relative to the truth of the sister sentence. This metric is given by Eq. 3.
vr(pt, pf ) =
{ 1 if L(“True”|pt)L(“False”|pt) < L(“True”|pf ) L(“False”|pf )
0 otherwise (3)
For the CoT metric, the model is prompted to output its reasoning steps as text. We do string matching to determine the answer. Eq. 4 gives us the validation CoT score, where G is the function that gives the model’s generated text after lowercasing. Instead of string matching, we could also insert a follow-up question after the model’s generation that requires a “True” or “False” and then compare log probabilities (henceforth “logprobs”). But it is useful to have a metric in our repository that does not use logprobs, which model APIs do not always provide.
vc(pt, pf ) =    1 if “true” ∈ G(pt), “false” /∈ G(pt) and “true” /∈ G(pf ), “false” ∈ G(pf ) 1 2 if “true” ∈ G(pt), “false” /∈ G(pt) and ¬(“true” /∈ G(pf ), “false” ∈ G(pf )) 1 2 if ¬(“true” ∈ G(pt), “false” /∈ G(pt)) and “true” /∈ G(pf ), “false” ∈ G(pf )
0 otherwise (4)

heading: 3.4 Non-Self-Referential Control
text: Is the self-referent part of self-referential statements (e.g. “this sentence ...”) the “hard” part of metalinguistic self-reference? There are metalinguistic problem categories that are not exclusive to self-referential language: recursive phrase counting, character-level manipulation, understanding hypothetical sentence-editing scenarios, etc.
Instead of giving a language model a sentence that refers to itself, we could give it an equivalent sentence that refers to that self-referential sentence. This way, the language model would not have to know whether a self-referential sentence is true. It would only have to know whether a sentence that refers to another sentence (which happens to
be self-referential) is true. This new task is still metalinguistic, but not self-referential.
It turns out that, for 97 of the sentence beginnings in “I am a Strange Dataset”, we can replace “this sentence” with “the following sentence”, and then copy the original self-referential statement below the new sentence. We can then test models for their ability to judge whether this non-self-referent version of the same statement is true. We use 2 of these 97 examples in the few shot and CoT prompts, because 2 of the examples in the original prompts cannot be turned into the non self-referent format. See Appendix A for the non-self-referent prompts. This leaves us with 95 non-self-referent examples and 95 original examples with which to compare results. GPT 4’s response to a statement from the main dataset is shown in Figure 3, along with its response to the analagous non-self-referential control statement in Figure 4. The responses happen to be the same in this case.

heading: 4 Human Experiment Details
text: To get a human baseline for our main task, we show each of the 400 self-referential statements (2 from each of the 200 examples) to at most 10 Mechanical Turk (Amazon, Retrieved 2023) workers. We separate statements from the same pair into differ-
ent experiment batches. As instructions, we give the annotators the few shot prompt in Appendix A, plus an extra paragraph: Do not use any AI assistants such as ChatGPT to help you; AI assistants perform very poorly at this task and so will get many of the answers wrong. Although, you can otherwise feel free to search online for any information that would help you answer confidently. For example, a few statements may contain a language besides English. So, you can feel free to use Google Translate. You can also search for the definitions of words that you are unfamiliar with.
To ensure a validator quality baseline, we require that all turkers are “master” annotators, are in the US only, have had 1000 or more previous HITs approved, have a 95% or higher HIT approval rate, and pass a custom qualification test which we release along with the dataset. The test involves correctly answering “True” or “False” to five statements from the “I am a Strange Dataset” example distribution. These five statements are not used in the official evaluation set of 200 examples. The qualification test involves an acknowledgement that the annotator is familiar with basic Python programming or can search online to answer basic questions about it, because there are a few examples that require a limited understanding of code. We include a screenshot of the Mechanical Turk annotator interface in Appendix C.
Because there are up to 10 human judgements for each statement, we get an analog to the “loss” for humans, and compute our metrics for humans in an analogous way to models. We can do this by com-
puting ratios, e.g. #responded true10 and #responded false
10 . The human scores are 89.25 and 93.00 on the Val FS and Valrel FS metrics, as shown in Table 2. Compared to the highest scoring model, the performance difference is 29 and 27, respectively. The human instructions are nearly identical to the few shot prompt, so the human responses are most comparable to the models’ few shot validation responses.

heading: 5 Results
text: Table 2 showcases our results on a variety of opensource (Touvron et al., 2023; Jiang et al., 2023; Zhu et al., 2023; Jiang et al., 2024) and closedsource (Brown et al., 2020; OpenAI, 2023; Anthropic, 2023) models. Overall, the models perform close to the level of chance. The only model to achieve scores significantly above random on all metrics tested is GPT 4, and even so, the perfor-
mance is well below the non-expert human scores. Results in this paper for Claude 2 (Anthropic,
2023), GPT 3.5 Turbo (Brown et al., 2020), and GPT 4 (OpenAI, 2023) were collected through the gpt-4, gpt-3.5-turbo, and claude-2 endpoints on their respective APIs on Jan 7, 2024 (unless otherwise stated). Note that the Claude 2 API does not support logprobs, so the only metric that we report for it is the text-based CoT validation metric. The OpenAI API supports top-5 logprobs access, and it turns out that this is enough to get logprobs for “True” vs “False” in our experiments. It is not enough to get the multi-token logprobs required for the generation metric, though.
The CoT metric gives us extra insight into the limitations of models because we can see if their reasoning aligns with their final answer. In many cases, they make fairly obvious mistakes as seen in Figure 3. In some cases, the models choose the correct answer even though their reasoning is flawed, as seen in Figure 5.
It is important to note that there are limitations with the CoT validation metric, and the ZS and FS non-relative validation metrics too. These metrics are about judging a statement as true or false,
independent of the statement with the alternative continuation. The dataset schema calls for sentences which are either true or false, but in practice, language can be ambiguous. Figure 6 illustrates. A fairer gauge of model capability may be to compare the continuations with each other, as the generation and relative validation metrics do.
Tags. We present the average logprobs-based model scores for the tags in Table 3. We opt to leave CoT scores out due to their high variance (many of the open-source models do not even return either “true” or “false” in their answers). Full tables for the tags are in Appendix E.
Non-Self-Referential Control. We provide the difference in scores between the non-self-referent examples and analogous original examples in Table 4. The significant values in the CoT column are not particularly compelling because the open-source models are below chance regardless of whether there is self-reference. For this metric, the open-source models are extremely sensitive to prompting – many of them do not even output “true” or “false” for every example. Although stronger models do so reliably - only 7 out of all of GPT 4’s responses contain both a “true” and a
“false”, or neither. We find that our dataset is so hard due to non-
self-referential metalinguistic aspects that models score around chance here too, with the exception of GPT 4. It is the only model which is strong enough to perform significantly above chance for every metric tested on the main dataset in Table 2, and to have all positive values in Table 4 for every metric tested, including logprobs-based metrics (meaning that the self-referential version was harder for it). Although, only the Val ZS value for GPT 4 is well outside of the 95% confidence interval, and GPT 4 is also still not particularly good at the non-self-referential version. Figure 4 shows that GPT 4 struggles with the non-self-referent version of the Figure 3 example. There is some signal that the challenge posed by self-referents will remain as LLMs gain competence at other metalinguistic problems, but the dataset is so hard that we do not have overwhelming evidence.
Model Scale. If we exclude the high-variance CoT metric, we see a clear scaling trend that models with more parameters score higher on the test. See Figure 7. Will this trend continue? For additional discussion, see Appendix D.

heading: 6 Conclusion
text: A grasp of self-reference is important in a variety of domains, and is a notable aspect of human intelligence. We introduced a novel task and dataset to assess the capacity of models to generate and
understand self-referential statements, and understand the prerequisite metalinguistic reasoning. All models that we tested perform fairly close to the level of chance. GPT 4 is the only model to score significantly above chance on all of the metrics tested, and still it is not by much. The poor performance may be indicative of a larger issue about the limitations of even today’s best causal language models. Even though the task is straightforward for people, we find evidence that scale beyond 70B parameters may be needed for the emergence of comparable performance from models.
Our results indicate that this dataset is hard not only due to the self-referent part of a self-referential statement. The challenge also comes from other metalinguistic aspects, such as recursively applying arithmetic operations on sentences. Still, there is some limited evidence that GPT 4 struggles more with self-referential metalinguistic problems than analogous non-self-referential problems.

heading: 7 Dataset Release Strategy
text: We release the dataset on GitHub. at https://github.com/TristanThrush/ i-am-a-strange-dataset. The data is encrypted, but the decryption script is provided. Our
goal is not to hide the dataset from people, but to hide the dataset from any processes that scrape training data from the web. We encourage the rest of the community to take up this practice when releasing evaluation datasets in a public repository.

heading: 8 Limitations
text: It is possible that the self-reference aspect of “I am a Strange Dataset” will turn out to be the bottleneck for many models, but it is also true that models are largely failing at the purely metalinguistic aspect. Although the schema targets metalinguistic selfreference, it is difficult to make a specific claim about why models fail without running more experiments and without waiting until models become more competent.

heading: 9 Ethical Considerations
text: We aimed to pay crowdworkers 15 USD hourly based on an estimated task completion time.

heading: A Model Prompts
text: In this section, we provide the full model prompts that we used for the validation metrics. The generation metric does not use a prompt (generation is conditioned on the beginning of a statement). For each prompt, $example is replaced by one of the 400 full statements before being given to a model. The few-shot examples were selected to match the author’s judgements about what best represents the task as a whole; they were not selected randomly.
A.1 Zero Shot Overview:
Your task is to say whether a self-referential statement is true or false. In cases where you think that the answer is ambiguous, you should use your best judgement and still say either 'True' or 'False'.
Example:
$example
Answer:
A.2 Few Shot Overview:
Your task is to say whether a self-referential statement is true or false. In cases where you think that the answer is ambiguous, you should use your best judgement and still say either 'True' or 'False'. Make sure that you understand the given examples before continuing.
Example:
I am the text you are reading. My last line has how many words? Two words.
Answer: True
Example:
I am the text you are reading. My last line has how many words? Seven words.
Answer: False
Example:
this sentence l o o o k s a bit like the l e
t t e r "p"
Answer: True
Example:
this sentence l o o o k s a bit like the l e t t e r "o"
Answer: False
Example:
here in reversed is order word the.
Answer: True
Example:
here in fine is order word the.
Answer: False
Now it is your turn to complete an example. Fill in the answer.
Example:
$example
Answer:
A.3 Chain of Thought Overview:
Your task is to say whether a self-referential statement is true or false. In cases where you think that the answer is ambiguous, you should use your best judgement and still say either 'True' or 'False'. Make sure that you understand the given examples before continuing.
Example:
I am the text you are reading. My last line has how many words? Two words.
Answer: The text has 3 lines. The last line is "Two words". The number of words in "Two words"
is 2. So the statement is true.
Example:
I am the text you are reading. My last line has how many words? Seven words.
Answer: The text has 3 lines. The last line is "Seven words". The number of words in "Seven words" is 2. So the statement is false.
Example:
this sentence l o o o k s a bit like the l e t t e r "p"
Answer: Visually, the words and letters of the sentence are arranged to look like a "p" on the page. So, the statement is true.
Example:
this sentence l o o o k s a bit like the l e t t e r "o"
Answer: Visually, the words and letters of the sentence are arranged to look like a "p" on the page, not an "o". So, the statement is false.
Example:
here in reversed is order word the.
Answer: We have to notice that, if the word order is reversed, the statement reads: "the word order is reversed in here". In the original reversed statement, that is true. So the statement is true.
Example:
here in fine is order word the.
Answer: We have to notice that, if the word order is reversed, the statement reads: "the word order is
fine in here". In the original reversed statement, the word order is not fine. So the statement is false.
Now it is your turn to complete an example. Fill in the answer.
Example:
$example
Answer:
A.4 Zero Shot (Non-Self-Referent) Overview:
Your task is to say whether a statement about a sentence is true or false. In cases where you think that the answer is ambiguous, you should use your best judgement and still say either 'True' or 'False'.
Example:
$example
Answer:
A.5 Few Shot (Non-Self-Referent) Overview:
Your task is to say whether a statement about a sentence is true or false. In cases where you think that the answer is ambiguous, you should use your best judgement and still say either 'True' or 'False'. Make sure that you understand the given examples before continuing.
Example:
The number of words in the following sentence is nine.
The number of words in this sentence is nine.
Answer: True
Example:
The number of words in the following sentence is eight.
The number of words in this sentence is eight.
Answer: False
Example:
the following sentence l o o o k s a bit like the l
e t t e r "p"
this sentence l o o o k s a bit like the l e t t e r "p"
Answer: True
Example:
the following sentence l o o o k s a bit like the l e t t e r "o"
this sentence l o o o k s a bit like the l e t t e r "o"
Answer: False
Example:
The food words that the following sentence contains are guava and cake and also carrot.
The food words that this sentence contains are guava and cake and also carrot.
Answer: True
Example:
The food words that the following sentence contains are guava and cake but not carrot.
The food words that this sentence contains are guava and cake but not carrot.
Answer: False
Now it is your turn to complete an example. Fill in the answer.
Example:
$example
Answer:
A.6 Chain of Thought (Non-Self-Referent) Overview:
Your task is to say whether a statement about a sentence is true or false. In cases where you think that the answer is ambiguous, you should use your best judgement and still say either 'True' or 'False'. Make sure that you understand the given examples before continuing.
Example:
The number of words in the following sentence is nine.
The number of words in this sentence is nine.
Answer: Counting the number of words in the sentence, we see that there are nine words. So, the statement is true.
Example:
The number of words in the following sentence is eight.
The number of words in this sentence is eight.
Answer: Counting the number of words in the sentence, we see that there are nine words, not eight. So, the statement is false.
Example:
the following sentence l o o o k s a bit like the l e t t e r "p"
this sentence l o o o k s
a bit like the l e t t e r "p"
Answer: Visually, the words and letters of the sentence are arranged to look like a "p" on the page. So, the statement is true.
Example:
the following sentence l o o o k s a bit like the l e t t e r "o"
this sentence l o o o k s a bit like the l e t t e r "o"
Answer: Visually, the words and letters of the sentence are arranged to look like a "p" on the page, not an "o". So, the statement is false.
Example:
The food words that the following sentence contains are guava and cake and also carrot.
The food words that this sentence contains are guava and cake and also carrot.
Answer: The food words mentioned are indeed guava, cake, and carrot. So, the statement is true.
Example:
The food words that the following sentence contains are guava and cake but not carrot.
The food words that this sentence contains are guava and cake but not carrot.
Answer: The food words mentioned in the sentence are guava, cake, and carrot. It is not true that carrot is not in the sentence. So, the statement
is false.
Now it is your turn to complete an example. Fill in the answer.
Example:
$example
Answer:
B I am an Impossible Dataset
The schema of “I am a Strange Dataset” does not place any limits on the complexity of an example. One of the expert annotators in particular took full advantage of this while they were creating examples. They generated 10 examples that another expert annotator could not understand without an explanation. These examples are not in our main dataset, where we verify that non-expert people can perform well. We release these 10 examples separately to showcase just how hard examples fitting this self-referential schema can get. We also hope that when language models improve, these 10 examples can be a useful initial probe into their reasoning capabilities. We provide two examples from this seemingly “Impossible” set in Figure 8. We also provide GPT 4’s output on another example from this dataset in Figure 9 below.

heading: C Mechanical Turk Annotator Interface
text: Here, we show a screenshot of the interface used by Mechanical Turk workers in Figure 10.

heading: D Supplemental Discussion
text: Here, we present supplemental discussion about why models are showing poor performance on “I am a Strange Dataset”.
D.1 A Test of the Tokenizer? Our tests are related to whether a model truly “sees” text in the same way as people. Metalinguistic statements may refer to the number of characters that they have, how text is arranged on the page, capitalization of certain letters, and relative positions of words. A human can easily count the characters that they see in a sentence, but models tend to encode text in tokens, not characters. We do not provide tests to disentangle the impact of different tokenizers, so this section is speculative.
D.2 Training Data Limitations Practically speaking, it is unlikely that there are many examples of metalinguistic statements in training datasets. They are incredibly timeintensive to generate, even if they are easy to verify. Yet people, who have almost surely seen even fewer examples, can do much better at this task than models. This “hard-to-create, easy-toverify” feature of hard evaluation datasets is true of Winoground (Thrush et al., 2022) too, which is a vision and language evaluation dataset that has remained unsaturated for well over a year. This goes to show that our models still have the wrong biases - will they change simply with model scale?

heading: E Results by Tag
text: In this section, we provide results in a table for each of the 10 tags.

heading: Future Work
text: We provide examples and GPT 4 responses in Appendix B and leave it as inspiration for a future challenge