Title: Weak-eval-Strong: Evaluating and Eliciting Lateral Thinking of LLMs with Situation Puzzles
Benchmark: SPLAT
Summary: Multi-turn puzzle game in which an agent in given a crazy scenario "The man's car lights were broken, and the fox was in the middle of the road, but he didn't hit him" and has to work on a reasonable explanation for why the situation isn't in fact, crazy. The agent gets to ask yes/no questions to an LLM overseeer, before submitting a guess at the final answer. 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Lateral thinking
Phenomenon defined by authors: No
Phenomenon definition: Vertical and lateral thinking are two essential styles that play critical roles in human cognition and
decision-making [42 ]. As noted in [ 20 ], vertical thinking, characterised by its logical and structured
nature, involves a systematic, step-by-step approach to problem-solving where each step logically
follows the previous one. In contrast, lateral thinking is about creativity and viewing problems from
multiple angles. It involves breaking away from traditional thought patterns to generate new ideas,
and embracing a more playful and imaginative problem-solving approach.
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: We propose the
exploration of lateral thinking in LLMs by situation puzzles as a primary research tool. A situation
puzzle, often referred to as a lateral thinking puzzle, involves a scenario, usually presented as an
unusual situation, and the goal is to figure out what is going on. Players ask yes-or-no questions
to gather more information and solve the puzzle.
Task item definition: Story: Matthew keeps reading a bedtime story to his son despite the blackout. Why?
Reference Answer: Matthew was blind, and he usually read bedtime stories to his son from a braille
book. That night there was a blackout, but this did not stop him from finishing the story.
Task source: Expert-crafted task examples (e.g. hand-written examples)
Task source detail: Scraped from public websites of situation puzzles
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: This is a highly fabricated task. Lateral thinking in the wild isn't so easily measured. 
Dataset size: 975
Dataset size detail: They train on the test set in order to evaluate downstream impact on other lateral thinking datasets
Dataset metadata available: Yes
Dataset metadata detail: 50k examples of humans taking the puzzles
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Extended interaction (e.g. conversation, calling an API and processing the response)
Response format detail: Responses are multi-turn 
Metric definition: Exact Match (accuracy, F1, precision, recall), Define 2 new metrics, RND and OCC which handle intricacies of the mutli-turn evaluation
Metric definition detail: Requires LLM-as-judge
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: By difficulty
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: No
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Authors discuss validity: They acknowledge the lateral thinking is hard to measure: "In this paper, we seek to explore and elicit the lateral thinking ability of LLMs. However, accurately evaluating this capability poses significant challenges due to the complexity of measuring creative thinking [29 , 19 ] and the difficulty of obtaining relevant data. The generation of novel ideas is inherently non-trivial, even for humans [13 , 14 ]. Considering these challenges, we propose the exploration of lateral thinking in LLMs by situation puzzles as a primary research tool"
Author validity detail: They acknowledge the lateral thinking is hard to measure: "In this paper, we seek to explore and elicit the lateral thinking ability of LLMs. However, accurately evaluating this capability poses significant challenges due to the complexity of measuring creative thinking [29 , 19 ] and the difficulty of obtaining relevant data. The generation of novel ideas is inherently non-trivial, even for humans [13 , 14 ]. Considering these challenges, we propose the exploration of lateral thinking in LLMs by situation puzzles as a primary research tool"
Taxonomy: Reasoning / 
New taxonomy: General Capability / Logical Reasoning
Procedural notes: Stored as excel file!