Title: SETLEXSEM CHALLENGE: Using Set Operations to Evaluate the Lexical and Semantic Robustness of Language Models
Benchmark: SetLexSem Challenge
Summary: This paper introduces a synthetic benchmark designed to evaluate the robustness of large language models (LLMs) in performing set operations under lexical and semantic variation.
The benchmark systematically alters input features like token type, length, frequency, and semantic similarity to test LLMs' ability to generalize across incidental variations, i.e. their System 2 robustness.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Robustness
Phenomenon defined by authors: Yes
Phenomenon definition: "Robustness in this context is System 2 robustness and requires that a perfect intelligent system exhibit no variance in task performance as incidental aspects of the input vary."
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: The task involves computing set operations (union, intersection, difference, symmetric difference) given two sets, with variations in operand size, token type (numbers or words), token length, token frequency, semantic similarity, prompting method, demonstration phrasing, and number of in-context demonstrations.
Task item definition: A prompt with this template: set construction, task definition, demonstrations, and final instructions.
Task source: Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Low ecology
Dataset size: Not directly reported, calculated: 12,800 configurations × 50 samples per configuration = 640,000 prompts generated
Dataset metadata available: No
Sampling method: Random sample (creators defined a task space and sampled from it), Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code), Structured response (e.g. valid JSON, API call alone)
Response format detail: The expected response format is one of the hyperparameters of the generated prompts.
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Mean and standard deviation

Metric subscores: Yes
Metric subscores detail: Scores by prompt generation hyperparameter: set operations, operand sizes, token types, token length, semantic grouping, prompt style and number of demostrations.
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Industry
Benchmark availability: The code to generate it is available
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results realism: No
Authors discuss validity: Yes
Author validity detail: The authors explicitly discuss the construct validity of the SetLexSem benchmark, particularly in relation to its goal of measuring System 2 robustness, defined as invariance to incidental task features. They argue that the benchmark validly captures this construct by systematically manipulating those incidental features and observing variance in performance.
Taxonomy: Language Modelling / Robustness
New taxonomy: General Capability / Reliability