Title: CUTE: Measuring LLMsâ€™ Understanding of Their Tokens
Benchmark: CUTE
Summary: The paper introduces CUTE, a benchmark designed to test the orthographic knowledge of large language models (LLMs), specifically their understanding of the character composition of tokens. It evaluates multiple LLMs on tasks requiring spelling, character-level similarity, and text manipulation.
Contribution: It also includes a Russian version (CUTE-Rus).
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Orthographic knowledge
Phenomenon defined by authors: No
Phenomenon definition: 1. Do LLMs know which characters make up
their tokens?
2. Do LLMs understand the difference between
semantic and orthographic similarity?
3. Can LLMs manipulate text at the character
level?
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: The task is to evaluate whether LLMs can identify, compare, and manipulate the character-level structure of their tokens through a series of prompts.
Task item definition: Prompt and expected answer.
Task source: Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: Synthetic datasets derived from high-frequency English words (sourced from Google Web Trillion Word Corpus) and simple sentences (from TinyStories).
Task ecology: Low ecology, humans wouldn’t usually ask LLMs to do these tasks.
Dataset size: 14,000 (not specified in paper, found on Hugging Face)
Dataset metadata available: No
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Simple mean
Metric subscores: Yes
Metric subscores detail: By task category (composition, similarity, manipulation) and granularity (e.g., character-level vs. word-level), and by language (English, Russian).
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: NLP / 
New taxonomy: General Capability / Compositional Reasoning
Procedural notes: Available on Hugging Face: https://huggingface.co/datasets/leukas/cute