Title: Connecting the Dots: Evaluating Abstract Reasoning Capabilities of LLMs Using the New York Times Connections Word Game
Benchmark: NYT Connections
Summary: The paper introduces a dataset of New York Times Connections puzzles with custom metrics to evaluate top language models against human players of varying skill levels. Results show that the best models significantly underperform both novice and expert humans. Based on the empirical analysis, the work develops a knowledge taxonomy to analyze model limitations in word categorization tasks.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: (Latent) reasoning, Lexical categorization in LLMs
Phenomenon defined by authors: Yes
Phenomenon definition: Abstract reasoning represents a person’s ability to solve problems, identify patterns, and work with logical systems. We propose the NYT Connections Game as a test bed for investigating the abstract reasoning capabilities of both humans and large language models (LLMs). (Section 1)
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Each example supplies a list of 16 unordered words taken from a single NYT Connections puzzle. The model is expected to partition them into four disjoint clusters of four words and name the underlying category for each cluster, producing its solution in one shot without feedback or retries. ​
Task item definition: A single Connections puzzle = a list of 16 words + hidden gold groupings + category labels.
Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions)
Task source detail: An archival site consisting of all possible answer choices and
their corresponding categorizations. As the NYT does not maintain an archive of NYT Connections puzzles, we resorted to an external, third-party site for data collection. Our data spans daily problems from the conception of NYT Connections in June 2023 to August 2024. (Section 3.1.)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: Models/humans see identical word grids, single‑shot constraint for both.
Dataset size: 438 puzzles
Dataset size detail: 438 puzzles = 7,008 word instances; 1,752 category instances
Dataset metadata available: Yes
Dataset metadata detail: Category difficulty, taxonomy label for each knowledge grouping.
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Structured response (e.g. valid JSON, API call alone)
Response format detail: model outputs category name + four‑word list per line
Metric definition: Exact Match (accuracy, F1, precision, recall), Distribution (perplexity, calibration, correlation)
Metric aggregation: Simple Mean
Metric statistics: simple mean, weighted and unweighted clustering scores, frequency counts, Fleiss Kappa
Metric subscores: Yes
Metric subscores detail: Distribution for each metric, success rates per reasoning type
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: Yes
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Success requires broad types of knowledge
Taxonomy: Reasoning / 
New taxonomy: General Capability / Logical Reasoning
Procedural notes: Human study used volunteer peers (14‑60 yrs).