Weak-eval-Strong: Evaluating and Eliciting Lateral Thinking of LLMs with Situation Puzzles
Authors: Qi Chen
Bowen Zhang
Gang Wang
Qi Wu
Venue: NeurIPS 2024 poster (2024)
Keywords: large language models
lateral thinking
situation puzzles
Primary area: natural_language_processing

Abstract: While advancements in NLP have significantly improved the performance of Large Language Models (LLMs) on tasks requiring vertical thinking, their lateral thinking capabilities remain under-explored and challenging to measure due to the complexity of assessing creative thought processes and the scarcity of relevant data. To address these challenges, we introduce SPLAT, a benchmark leveraging Situation Puzzles to evaluate and elicit LAteral Thinking of LLMs. This benchmark, containing 975 graded situation puzzles across three difficulty levels, employs a new multi-turn player-judge framework instead of the traditional model-based evaluation, which often necessitates a stronger evaluation model. This framework simulates an interactive game where the model (player) asks the evaluation model (judge) questions about an incomplete story to infer the full scenario. The judge answers based on a detailed reference scenario or evaluates if the player's predictions align with the reference one. This approach lessens dependence on more robust evaluation models, enabling the assessment of state-of-the-art LLMs. The experiments demonstrate that a robust evaluation model, such as WizardLM-2, closely matches human judgements in both intermediate question-answering and final scenario accuracy, achieving over 80% agreement--similar to the agreement levels among humans. Furthermore, applying data and reasoning processes from our benchmark to other lateral thinking-related benchmarks, e.g., RiddleSense and BrainTeaser, leads to performance enhancements. This suggests that our benchmark effectively evaluates and elicits the lateral thinking abilities of LLMs.

Weak-eval-Strong: Evaluating and Eliciting Lateral Thinking of LLMs with Situation Puzzles
Chen Bowen Zhang Gang Wang Qi Wu Australian Institute for Machine Learning, University of Adelaide chen04, b. b.znang, qi www.waaeeeeadddee..mmommoaadaae edu. au, gangütheb..
Abstract
While advancements in NLP have significantly improved the performance Large Language Models (LLMs) on tasks requiring vertical thinking, their lateral thinking capabilities remain under-explored and challenging to measure due the com- plexity of assessing creative thought processes and the scarcity of relevant data. To address these challenges, we introduce SPLAT, benchmark leveraging Situ- ation Puzzles to evaluate and elicit LAteral Thinking of LLMs. This benchmark, containing 975 graded situation puzzles across three difficulty levels, employs multi-turn player-judge framework instead of the traditional model-based evaluation, which often necessitates a stronger evaluation model. This framework simulates interactive game where the model (player) asks the evaluation model (judge) questions about an incomplete story to infer the full scenario. The judge answers based on detailed reference scenario evaluates if the player's predic- tions align with the reference one. This approach lessens dependence on more robust evaluation models, enabling the assessment of state-of-the-art LLMs. The experiments demonstrate that a robust evaluation model, such as WizardLM-2. closely matches human judgements in both intermediate question-answering and final scenario accuracy, achieving over 80% agreement-similar to the agreement levels among humans. Furthermore, applying data and reasoning processes from our benchmark to other lateral thinking-related benchmarks, e.g., RiddleSense and BrainTeaser, leads performance enhancements. This suggests that our bench- mark effectively evaluates and elicits the lateral thinking abilities of LLMs. Code is available at: https:/ggihub..cm/chnnnnoo wwtp.//o/o/occo/..aaaaa..a. wat.ll////aaaaaaaaaaaaaaa Thtnkalliinning
Introduction
Vertical and lateral thinking decision-making [42). As noted in (20), vertical thinking, characterised by its logical and structured nature, involves systematic, step-by-step approach problem-solving where each step logically follows the previous one. In contrast, lateral thinking is about creativity and viewing problems from multiple angles. It involves breaking away from traditional thought patterns to generate new ideas, and embracing a more playful and imaginative mobllem-solving approach.
essential styles that play critical roles in human cognition and
The evolution of natural language processing (NLP) models, particularly large language models (LLMMs) 10 15 41 has seen significant advancements tasks that require vertical thinking, such complex reasoning [5) [4] and commonsense inference 6 38). Despite these achievements, the development and evaluation these models have primarily focused on vertical thinking capabilities [7] 43], often neglecting lateral thinking, which essential for creatively solving novel problems. Current benchmarks
(35 36 frequently dismiss creative thinking as irrelevant, focusing only on
Corresponniig author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Easy
Question/Story: black cat walking through the middle of read where thene broken streetlamp. A man driving his vehicle with huodlights that do not work, Despite all of this, he is uble to avoid harning the cut. Reference nswer: The man avaided
because was daylight.
Medium
Question/Story: bunter aimed his gun carefully and fired. Seconds later, he realized his mistake Minutes later, was dead. Reference nswer: He hunted in snow-capped mountains. The shot provoked an avalanche, which covered the man. He died of strangulation.
Hard
Q QuestionStory: Two men are beoting each other up and both of them suddenly fall to the floor: Reference A nswer: The daughter of one of boxers had been kidnapped and she would be killed her father won fight. At the end of the fight, the boxer's opponent fainted and the other boxer- in order not to win the match- - threw himsedf to the floor.
Figure 1: Examples from SPLAT benchmark, which are categorised into three ascending levels of difficulty, i.e., Easy, Medium, and Hard. The puzzles that are medium to hard in difliculty usually require guidance from the judge to be solved. If puzzle can be solved without the judge's ' guidance, it typically resembles a regular puzzle requiring specific knowledge more than lateral thinking skills.
problems solvable through conventional commonsense reasoning, indicating a gap in traditional LLMs' handling of unconventional thinking tasks, such as puzzles (20).
To fill this gap, benchmarks increasingly aim for tasks challenging even for humans, with outputs that are typically open-ended g 27 However, evaluating such open-ended generation often relies on human or model-based assessments 477. As tasks grow in complexity, such as longer output requirements, human evaluations become less feasible due to the extensive time and effort required to accurately judge long, complex scenarios created by LLMs. Moreover, model-based evaluations typically depend on more advanced models, thereby restricting their capacity to evaluate newer, state-of-the-arr models. This reliance may also introduce biases, such as preference for the first choice [47]. Consequently, there is a pressing need to rethink the evaluation framework of the benchmark for assessing emerging LLMs.
In this paper, we seek to explore and elicit the lateral thinking ability of LLMs. However, accurately evaluating this capability poses significant challenges due to the complexity of measuring creative thinking [29] 19] and the difficulty of obtaining relevant data. The generation of novel ideas is inherently non-trivial, even for humans [33 [4]. Considering these challenges, we propose the exploration of lateral thinking in LLMs by situation puzzles as a primary research tool. A situation puzzle, often referred to as a lateral thinking puzzle, involves scenario, usually presented as an unusual situation, and the goal is to figure out what is going on. Players ask yes-or-no questions to gather more information and solve the puzzle. These puzzles challenge players to think outside the conventional bounds of the scenario's 's information, using creativity and indirect reasoning to uncover the underlying explanation for the situation described. In this context, introduce SPLAT, a benchmark that leverages Situation Puzzles for evaluating and eliciting LAteral Thinking of L.MMs, which contains 975 high-quality situation puzzle pairs. As shown in Figure 1 for better assessment, we categorise these puzzles into three difficulty levels-Ear Easy, Medium, and Hard- each annotated by human evaluators. We obtain 54,463 annotations of difficulty
average of ~55 per puzzle.
To assess the performance of LLMs in our benehmark with an open-ended setting, we initially aim to apply model-based evaluation paradigm. However, this paradigm (e.g., 47)) typically relies on using an evaluation model that is stronger than the models being tested. This creates a challenge in accurately evaluating models that may be superior the evaluator itself. Therefore, there is need for alternative methods that can effectively assess the advanced models without this limitation. To this end, inspired by the situation puzzle game, we propose multi-turn player-judge framework. This framework emulates an interactive game where the model (player) queries the evaluation model (judge) about given incomplete story, striving to collect information to unravel the intricate scenario it implies. Conversely, the evaluation model (judge) is tasked solely with responding to questions based on the detailed reference scenario or evaluating whether the scenario deduced by the player aligns semantically with the reference one In this way. this framework reduces the reliance on more advanced evaluation models, and thereby can be used to evaluate the state-of-the-art LLMs.
The experimental results indicate that a relatively strong evaluation model, e.g., WizardLMM22 can align closely with human judgements in both intermediate crestion-answering and final an- swer/scenario accuracy, achieving over 80% coreemment--mmprraabe to agreement level between
Note that the ground truth reference answer/scenario only be seen by the judge.

---

humans. Moreover, applying the data and reasoning processes from our benchmark to other lateral thinking-relevant benchmarks like RiddleSense [22] shows performance improvement (e.g., the accuracy of Llaam3-70B []] increase from 83.34% to 85.21%). It indicates that our benchmark can not only evaluate but also elicit the lateral thinking capabilities of LLMss. Finally, we evaluate several advanced LLMs such as GPT-4 and GPT-4 Turbo on our benchmark. The results highlight the ongoing challenges and the need for improved lateral thinking in LLMs.
We summarise our contributions as follows:
We introduce benchmark, called SPLAT, evaluate the lateral thinking capacity of LLMs with situation puzzles. In this benchmark, we construct a new situation puzzles dataset, containing 975 unique puzzles that are divided into three difficulty levels.
We design a multi-turn player-judge evaluation framework to move beyond the existing model- based evaluation paradigm in open-ended settings. This new framework reduces reliance on a stronger evaluation model typically stronger than the models being assessed.
The dataset and framework are designed not only to evaluate but also to actively elicit the lateral thinking of LLMs. Experiments show that using data and reasoning processes from our framework leads to improved performance of LLMs, even when applied to other lateral thinking benchmarks.
Related Works
Lateral Thinking for LLMs. The realm of lateral thinking and computational creativity spans various
tasks [50 26 48
39,
18, 32). such as pun detection [50] and humour recognition [26] 48), which
assess different cognitive capabilities. Notable contributions in this field include BrainTeaase (20), which evaluates a broad spectrum of human intelligence attributes such as strategy and creativity. Another significant effort is RiddleSense (22). collection of riddles from public websites aimed testing model abilities. In contrast, our work enhances this area by introducing scenario-level lateral thinking puzzles, i.e., situation puzzles, constructing benchmark to evaluate and elicit the lateral thinking ability of Large Language Models (LLMs).
Benchmarks for LLMs. Large language models (LLMs) 2
3 37 are increasingly showing capabilities for various tasks ranging from writing and coding to engaging in multi-turn dialogues. However, assessing their extensive capabilities presents new challenges. Existing benchmarks, as categorised by [47]. predominantly fall into three types. korellnow benchmarks [17 46 [1] [[4
9 49
assess LLMs through shoo and few-shot tasks, requiring concise, specific answers to questions that are easily validated. Instruction-follovin
benchmarks [24] (8) handle more open- ended questions and broader range of tasks, which evaluate LLMs post-instructional fine-tuning. Lastly, conversational benchmarks (30 16 21 designed for dialogue-bar based tasks, still lack the diversity and complexity needed to fully challenge advanced chatbots.
For more challenging benchmarks, there a trend toward including tasks that are even more difficult for humans, often requiring open-ended outputs [ 27. Evaluating such open-ended tasks typically depends on either human or model-based assessments. As these tasks increase in complexity, human evaluations become impractical due to the substantial time and effort needed to accurately assess the lengthy and intricate scenarios. Existing model-based evaluations [47] inherently rely on stronger models, which limits their ability assess new state-of-the-arr models. In this paper, introduce a multi-turn player-judge framework to improve upon traditional model-based evaluations in open- ended settings. It reduces the need for stronger evaluation model typically used in such assessments.
3 Task and Dataset
3.1 Task Definition of Situation Puzzle Game
The situation puzzle game involves a player P and a judge J. Firstly, the judge provides an incomplete story Sp to the player. Then, the player poses a set of questions Q = (q1- 92..... 9: ......) gradually for the judge to gather information about the unknown and detailed scenario/answer Š behind the initial story Sp- The judge responds with answers A {a,, .2,...,,,..................,.... ..,...,, where each answer a, E (yes, no, irrelevant). The objective is to deduce the hidden scenario/answer Š that semantically aligns with all preceding yes-or-no question-answertee answering pairs while fitting the given story Sp.
3

---

Table 1: Comparison between our SPLAT and related benchmarks, including both vertical (Ver.) and lateral (Lat.) thinking. For a comprehensive comparison, we evaluate the benchmarks across four dimensions: the target task, the fashion of answers, the evaluation paradigm, and the statistics of samples. 'Ref. A' indicates whether the question or dialogue comes with a reference answer. 'Open-E.' denotes whether the response is open-ended. 'Ref.-G'' and 'Mod.-B." refer to whether the result evaluation is reference-guided or model-based, respectively. 'Assessment' denotes the specific evaluation pattern used to assess the quality of the predicted results. 'Avg. Q' and 'Avg. A' are the average number of tokens in questions (or input dialogues) and reference answers, respectively.
Anewer Ref. A Open-E. Ref-GG
Evaluation
Statisric
Benchmark
Task
Mod.BB
Assessment
Number
Avg. Q
Avg.
GALA 2
Anesticoo-Answering
Exact Match
466
65,85
2.03
Chaibot Arena
Dialogues
Battle
30,000
/
MTT-Bench (Ist
Muhi-tum QA
Grading or Bartle
80
64.78
MT. ATT-Bench (2nd Turn)
Multi-um QA
Grading or Bartle
80
22.76
Oogini (T2T, Eng.1
F8
Multiple-choice QA
Ranking
6,433
12.46
10.38
RidkleSSnne
Multiple-dhoice
Ranking
5,733
26.21
2.87
BrainTesssr (8.)
20
Multiple-choiie QA
Ranking
627
34,88
9.11
BrainTraser (W)
M0
Multiple-choiee QA
Ranking
492
10:65
3.00
SPLAT (Ours)
Sccnario Deduction
Semantic Judge
975
79.74
45.53
3.2 Data Construction
Data Collection, To collect the situation puzzle data, we follow the steps from 20) Specifically, we first gather over one thousand situation puzzles and their answers from various public websites using web crawlers After that, we merge the data from different sources and then eliminate the duplicates based on sentence similarity [ Then, we correct typographical errors using the Auto Correct library followed by human review process to ensure the puzzles maintain their intended meanings. This approach combines automated corrections with manual oversight to preserve the quality of the puzzles. Finally, we obtain collection of 975 unique situation puzzles.
Data Annotation and Difficulty. Inspired by the organisation of GAIA (27). we categorise our collected situation puzzles into various distinct levels of difficulty, ranging from (easiest) to 9 (hardest). This classification is done through crowd-sourced annotation involving average of approximately 55 annotators per puzzle to ensure the reliability of the difficulty rating, Then, these levels are further grouped into three broader categories: Easy (I~3), Medium (4~6), and Hard (7~9).
3.3 SPLAT Benchmark (Ours) vs. Related Benchmarks
We compare benchmark with the most relevant ones w.E. both vertical and lateral thinking. Specifically, for vertical thinking datasets, we consider GAIA (27) and LLMM-as-a-Judge [47). where the former is the most recent benchmark for general AI assistants, and the latter is the widely used benchmark to evaluate LLMs. The LLM-as-a-Judge 47 contains two different benchmark datasets, i.e., MT-bench and Chatbot Arena. The MT-bench is set of challenging multi-turn open-ended questions designed to evaluate chat assistants. Chatbot Arena is a benchmark platform designed for evaluating LLMs through interactive, randomised battles in crowd-sourced manner. allows for the assessment of LLMs in real-time dialogues with human participants.
Regarding lateral thinking benchmarks, we compare Oogiri 488 RiddleSense (22 and Brain- Teaser 20) As the Oogiri benchmark (48) is multi-modal (text and image) and multi-lingual (English, Chinese, and Japanese), for fair comparison,
consider Text-to-Texs (T2T) English (Eng.) samples only. RiddleSense [22] is a benchmark dataset designed for evaluating complex commonsense reasoning within the framework of riddle-style questions. It presents unique multiple- choice question-ans wering task that requires an understanding of figurative language, counterfactual reasoning, and other advanced natural language understanding skills. The BrainTeaser dataset 20 is a versatile and challenging collection for evaluating models on their ability to engage in lateral think- ing and solve intricate puzzles. It features two main types of puzzles: Sentence (S.) puzzles and Word (W.) puzzles. The sentence puzzles require solvers to interpret or decipher complex sentences, often involving multiple meanings or requiring the solver to think beyond straightforward interpretation.
2 Upon publication, we will release our data, as all the puzzles are sourced from publicly accessible websites. bbttss/giituub.c///ppaappggld

---

An Overview Puzzle
Easy
Medium
Questions
#Samples
50
Average Tolens per Sample
1:00
200
Time (Minues)
of Semple
80
150
30
60
100
20
40
0
50
100
150
200
Difficulty
Number of Tokkns (Answor)
(a) Sample Distribution
(b) Time Cost vs. Length of Answer
Figure 2: We show (a) the distribution of sample counts across different difficulty levels, and the average number of tokens per sample. We also exhibit (b) the distribution of time cost for human players and the number of tokens for each reference answer (200 samples in total) difficulty levels.
The word puzzles focus wordplay semantic puzzles, where the challenge lies in understanding and manipulating words based on how they are presented or their semantic relationships.
In Table we compare our SPLAT benchmark with existing ones across various dimensions such as the type of target task, the fashion of answers, the evaluation paradigm, and the statistics of samples. Here's s detailed discussion of how our benchmark, focused on situation puzzles, stands out:
Task Uniqueness and Open-ended Responses. Unlike traditional QA and dialogue-based tasks in [27 [7]. our task requires models to infer the scenario from given incomplete story, necessitating deep engagement with both context and creativity. The process of accurately inferring the scenario involves multiple steps and is challenging to brute force due to their diversity [27]. as well as difficult to evaluate. Our benchmark supports the verification of reasoning traces and predicted answers. Moreover, unlike the multiple-choic formats seen in tasks like Oogiri [48) and RiddleSense [22). our open-ended format lessens the likelihood of inadvertently leading to the correct answer.
Reference-guided Semantic-aware Evaluation. Unlike the exact match or pairwise battle ap- proaches used in vertical thinking benchmarks 47 277, SPLAT employs 'Semantic Judge' evaluation. Analogous to Natural Language Inference (NLI) (25), we assess the semantic align- ment between the predicted output and hunan-geeerated reference answer using LLM-based judge. This approach ensures accurate evaluation, even when the formats of the predicted outputs significantly differ from those of the reference answers.
Answer. Our dataset features 975 samples with relatively high averages in question and answer lengths (79.74 and 45.53 tokens respectively). This indicates a depth of content and complexity in our scenarios, surpassing most lateral thinking datasets such RiddleSense (22) or BrainTeaser (20). where the questions answers are typically much shorter and less detailed. Moreover, by employing model-based and inferrnnce-gidder evaluation paradigm, we reduce the dependence on more robust evaluation model, different from [47], which typically needs a stronger model to assess. Our approach only necessitates that the judge model determines if the predicted answer is semantically aligned with the reference answer. Once verify that the used judge model can reliably perform this task, the relative strength of the models being evaluated compared to the judge model becomes irrelevant.
Challenging Question with Judgement-friendee
3.4 Statistics on SPLAT Benchmark
Situation puzzles on the proposed SPLAT can be classified into three ascending levels of difficulty, based on three criteria. 1) Time to Solve: How long does it typically take for player to arrive at correct answer? Shorter times indicate easier puzzles, while longer time suggests higher complexity. 2) Length of Reference Answer/Scenario: Simpler puzzles may have shorter, more straightforward answers, whereas more complex ones might require lengthy explanations involve intricate scenarios.
5

---

3) Subjective Complexity of the Scenario: This involves subjective evaluation of how mentally challenging the scenario for the human players. Thus, the puzzle in each level can be defined as:
Easy Puzzles (1~3): These puzzles typically require minimal time to solve, with reference answers that are short and direct, involving straightforward solutions. These scenarios simple and can usually be resolved quickly, often within a few minutes, and do not necessitate extensive reasoning.
Medium Puzzles (4~6); These are moderately complex puzzles that demand more time and deeper analysis. The reference answers are longer, providing more details that players must consider. Solving these puzzles generally takes moderate amount of time, with players needing to make several logical deductions and possibly sift through some distractors.
Hard Puzzles (7~9): These are the most challenging puzzles, designed for advanced players. They require a significant amount of time to unravel, with lengthy and complex scenarios that may include extensive details or embedded subtleties. Players must engage in high degree of reasoning, with an array of logical deductions and the potential for numerous misleading clues. These puzzles often demand an arbitrary number of steps, deep thinking, and the ability to focus over prolonged periods.
Figure aa suggests a correlation between difficulty level and the average number of tokens, partieu- larly for answers, which tend to increase in length as the difficulty level rises. This could indicate that more complex puzzles require longer, more detailed answers. In Figure (b). the clusters by difficulty level also show that harder puzzles tend to have both longer answers and higher time costs, aligning with intuitive expectations about puzzle solving.
4 Multi-turn Player-Judge Framework
We design such a framework (Figure B for three main reasons. Firstly, the complexity of scenarios in situation puzzles makes it difficult for a model to accurately predict the final answer in just one attempt (i.e., in a single turn). Secondly, we aim to ensure that the reasoning process remains interpretable and assessable, even in the absence of pre-defined reference reasoning steps. This framework allows for a more thorough evaluation of the model's ability to solve complex puzzles step by step. Thirdly, we hope that the framework or the intermediate reasoning processes derived from this framework can help elicit the lateral thinking ability of LLMs even on other lateral thinking rellvant datasets.
Characters. The multi-turn player-judge framework of situation puzzle game has two primary characters: a player P and a judge J. The player P is a participant (e.g., LLMs we need to evaluate), who attempts solve the puzzle by asking questions. The judge J is the entity, either a human an AI model, that knows the solution to the puzzle and provides responses to the player's 's questions. Due to the page limit, we put the detailed character guidelines in the supplementary.
Interaction Dynamics. The interaction between players and the judge driven by a series of questions Q {91. 92, ... ............................. ......} and answers A = {a1, 02,.... ay, ag...................... .). Specifically, at the start
of the game, the player is presented with an incomplete story, denoted as Sy. The player must read this story and initially decide if the hidden scenario or answer á can be inferred from the present information (i.e. solely Su). If the scenario remains unclear, the player will then pose yes-or-no question to obtain more details about the puzzle.
The judge provides a response E {yes, no, irrelevant) based on the incomplete story So and the hidden scenario $ i.e., 01 J(n, Sp, S). Upon receiving the response a1, the player will either attempt to deduce the final answer or, if still uncertain, will ask a subsequent question (i.e., 42). This process continues iteratively, with the player refining the understanding based on the judge's answers.
Information Sets and Game State. At any turn in the game, the information set I, represents all the questions asked and answers received up to that time:
I {(gi, a1), (2),0))...... (q., az))
The game state I, includes the incomplete story So. the information set Ie and any inferred knowledge K, about the situation puzzle:
r (So, Iz, Ki), where K, P(So, Iz).
'Note that the judge knows the detailed scenario. i.e. the reference answer.

---

Chararter Initialisation (fudge):
You are judge, and your role to challenge user with a given puzzle the form of short, logically self-consistent story. The story are complete but may intentionally omit some details that the user needs to deduce or inquire about.
Here are the guidelines for your role: 1. 4.
Now, the system will give your first puzzle. The game starts when the indicates they ready.
Enjoy the challenge of testing and expanding the user's deductive reasoning!
Puzzle Title: Sweet D Drems
Story: Matthew keeps reading bedtime story to his son despite the blackout. Why?
Reference Answer: Matthew was blind, and he usually read bedtime stories to his son from a braille
book. That night there was blackout, but this did not stop him from finishing the story.
Tudge
fan Congratulations! 14. Chinkk have Your woreect
4x-11 Is Ca.: Yes
<Story> Matthow koups reading bedtime story
Qa:
bedtime
story
also blind
Matthew
enough information. Matthew blind
and read the book by touch
son despitu the
being read from
or visually through Braille, and the blackout
blackout Why?
physical book?
impaired!
ddoes not nol abilitt to read...
Player
Character Initialisation (Player)):
You are a player and going to play interesting situation puzzle game.
The rules of the game Are you ready? Let's begin! User will give you the first puzzle.
follows: 1. 2. 3. 4. 6.
Figure 3: Overall of the multi-tune turni player-judge framework. The player begins with given story and poses yes/no questions to uncover hidden details. The judge, informed by a reference answer, responds to guide the player toward the correct reasoning. The player's goal is to deduce the scenario based on the judge's feedback and the initial story input. The game continues with questions until the player deduces the correct answer, at which point the judge confirms with a congratulatory response.
Here, K2 is the knowledge inferred by the player based on the story S. and information set I,. The player determines the next question based on the current game state by
g++1 P(T,).
The objective of the player is to deduce the scenario/answer S that satisfies all given constraints based on the responses from the judge. Mathematically, the goal is to find the hypothesised solution S that maximises the posterior probability given the game state Iy:
Objective.
$ <- arg max Pr(S TT).
8
5 Experiments
5.1 Agreement Evaluation
Inspired by to assess the alignment of judge's responses with human preferences, we consider an agreement evaluation among them w.r.f. judgements for both final predicted answers and the intermediate reasoning processes. We randomly select 20 puzzles from each difficulty level, which is 60 puzzles in total. For each puzzle, we ask three different individuals to assess the final answer agreement and the reasoning process agreement, respectively. For a fair comparison, we use the questions generated by taking llama3 (70B-Instruct) I as the player and WizardL (8x22B) 45 as the judge. Besides humans, we also consider Llama3 Instructt as another judge model to test the agreement among humans and different judge models. More details are in the supplementary.
Final Answer Agreement. To quantitatively evaluate the semantic alignment between the final predicted answers and reference answers, we first ask humans to assess the semantic alignment between each predicted answer and the corresponding reference answer/scenario. Subsequently, we measure the agreement performance between the human judgement and the evaluations from the judge model. We totally obtain 156 judgements from humans. In Table the high level of agreement between the LLMM-bseed judges and humans (e.g.. WizardL achieves 100%, 87.50% 88. 24.2%% agreements with humans in three difficulties, respectively) indicates that L.LMs can well align with human standards of reasoning and judgement within this specific context of lateral thinking puzzles.
7

---

Besides, in the open-ended setting, the player model may deduce other reasonable scenarios that satisfy both the given incomplete story and the responses from the judge model. In this situation, it is hard to judge whether these deduced scenarios are correct or not due to the diversity. Inspired by the evaluation of image captioning on MSCOCO (23), which uses multiple reference answers for each image and computes metrics based on the closest reference answer, we consider a similar approach. However, generating multiple reference scenarios/answers for situation puzzles is challenging. As an alternative, we repeatedly run each situation puzzle multiple times (denoted as R). and then choose one (maybe r-th) that is closest to the reference scenario from these iterations as the final result. This Table 2: Final answer agreement among three types of judges on SPLAT benchmark.
Easy
Medium
Hard
Judge Model
Llama3- 70B
Human
I Llama3-708
Human
Llama3-70B
Human
WizardLM-2
93.21%
92.49%
87.50%
88.2496
LLamm3-708
54.56%
80 61
83.175
Human
100%
87.505
100%:
Table 3: Reasoning process agreement among three types of judges on SPLAT benchmark.
Easy
Medium
Hard
Judge Model
Llama3-708
Human
LAura3-708
Hurmam
LAura3-708
Human
M--
82.15%
64.18 56
78 82%
89 86/6
72. 144
90. 10.15%%
II 11mm3-7708
N0.22%
77.83%
71.36%
Human
92.933
94.13%
93.11%
Reasoning Process Agreement. To assess the agreement reasoning processes, particularly for intermediate questions where no predefined reference answers exist, we use a method where humans asked to respond to these questions based on the context provided by the given story and the final reference answer. Their responses are categorised as [yes, no, irrelevant). We then evaluate the alignment of these human responses with those from the judge model. We obtain 2,271 judgements from humans. Table 3 shows that LLMs, e.g., WizardL.M- and Llama3-70B, demonstrate good alignment with human reasoning in simpler puzzles. As the difficulty level rises, WizardLM-22 keeps a relatively high agreement with human judgements, while the alignment between Llama3-70B and humans decreases dramatically. The results verify that our choice of WizardLM-22 as the judge model aligns well with human judgement across all difficulty levels, maintaining over 80% agreement.
5.2 Performance of LLMs on SPLAT Benchmark
Evaluation Metrics. To assess the performance of LLMs using the proposed SPLAT benchmark, employ two distinct metrics: Accuracy (Acc, %6), which measures the correctness of the scenarios de- duced by LLMs against the reference scenarios/answers, and Average Round (Rnd), which quantifies the average number of interaction rounds required for LLMs to reach the reference scenarios. Note that if the LLMs fail deduce the correct scenario before a pre-defined max round, the round count is the same as the max one, and the accuracy for that particular puzzle is recorded as O. Conversely, the correct scenario is successfully predicted, the accuracy is to and the number of rounds taken to reach the correct answer is recorded. Finally, we define an OverAll (O/A) evaluation metric as
1 N I((samlle,) O/A
N Σ Rnd,
100,
(5)
where II()) is an indicator that returns 1 if the scenario deduced by LLMs matches the reference one semantically, and otherwise. Given that [(ssmpll;;) takes values in (0, 1) and Rnd, ranges from I to maximum, the O/A metric spans from 0 to 100, with higher values indicating better performance.
approach not only maintains the diversity of potential predictions but also reduces the impact of hallucination from LLMs by subjecting each puzzle to several evaluations.
Results. We compare the performance of various LLMs on our SPLAT benchmark, including Lama3 (8B-Instruct & -n [I]. Qwen1.5 (32B-Chat & 10BB-hha (OB-Chat) 4. WizardLM-2 (8x22B) [45] GPT-4 and GPT-4 Turbo 2 Here, we take these LLMS as players and use WizardLM- (8x22B) as a judge in all these performance comparison experiments. We set R R==1 1 and the max round is 15. Table GPT-4 and its Turbo variant, along with the Ilama3 (70B-Instruct) and WizardI.M-2
(8x22B), show robust capabilities, achieving relatively higher accuracy. Conversely, models like
8

---

Table 4: Performance of different LLMs on the proposed SPLAT benchmark.
Easy
Medium
[aad
Averape
Acc (T. 5)
Rnd (1)
O/A (?)
Acc E, (3)
Rnd (1)
0/A(11 (1)
Arc (1, (8)
Rnd (.)
0VA (1)
Are (1, ())
Rad (.)
OVA (t)
Liama3
31.79
13.32
3.65
14.81
14.29
1.55
4.35
14,80
0.50
17.05
14.13
1.91
Lamas-208
64.05
10.75
8.67
28.54
13.37
3.54
10.91
14.31
1.36
14.90
12.81
4.52
Qwenl 5-128
29.55
12.22
6.39
15.5%
14.11
2.47
10.00
14.22
2.01
18.51
13.68
5.62
Qweni 5-1108
46.08
11.28
9.71
24.22
13.48
3.74
17.27
14,01
2.16
29. 19
12.92
5.20
W
58.52
10.49
11,97
31.48
13.12
4,67
16.36
13.96
2.74
35.45
12.52
0.46
GPT4 Turho
60.36
9.59
13.22
28. 39
12.87
5,86
10.00
14.37
1.35
32.91
12.27
6:54
GPT4
70.96
8.93
15.25
36,88
12.41
6.71
17.27
13.99
2,52
41.70
11.77
8.16
Qwen1.5 (32B- (2B-hhat) and Qwen1.5 1110--hhat show lower accuracy, which might reflect challenges adapting their chat-based configurations to the lateral thinking required on the SPLAT benchmark.
Performance Bias. In the SPLAT benchmark, concerns about bias when a model like WizardI.Mr (8x22B) evaluates its own performance mitigated through several key design elements. First, the SPLAT benchmark is designed with a specific framework that focuses on how well each L.M,, serving as a player, can reason through situation puzzles to arrive correct scenarios/answers. The judge's role is tasked solely with responding to questions based on the detailed reference scenario or evaluating whether the scenario deduced by the player aligns semantically with the reference one, which is relatively simple and objective. Besides, the evaluation metrics used (Accuracy and Rounds needed) are also objective, reducing the potential for subjective bias or preference.
From Table 4 WizardLM-2 (8x22B) performs well but not exceptionally in every category. It has competitive but not maximal accuracy rates, and its average round count is not the lowest. It suggests that the WizardL (8x22B) we use has strong enough capabilities to serve as a judge model. Besides, under benchmark, the used judge model does not have obvious preference for itself.
5.3 Eliciting Lateral Thinking of LLMs
Models and Benchmarks. As previously discussed, our benchmark serves to evaluate the lateral thinking abilities of LLMs and to elicit these capabilities actively. To verify this, seek to enhance LLMs with the data and reasoning processes from our benchmark, and then evaluate them on other lateral think- ing benchmarks (i.e., RiddleSense (22 and BrainTeaasr (00) to observe performance en- hancements across various LLMs. Our anal- ysis includes both open-source and closed- source LLMs, specifically Llama3 (Instruct version) and GPT-4, with additional consid- erations for different model sizes, such as Llama3-8B and Llama3-70B.
Accuracy Various LLMs
90
85
80
75
70
65
60
58
BEFT base
Largo
RoaeiiTe Large
ALBERT-3N X2L
Kaotet (R8-j)
MNHRR (R8 4)
MNNGHN LAB-XX
UndfeedCA (75-U)
UnafeedgA ITS:
Liama Lama3-o0 ap
Liama 708
Unma3- Trg-
GPT
Implementations. In the zero-shot setting, LLMs are challenged with multiple-chote choice riddles from these benchmarks, where they must choose answers based solely on the rid- dles presented. The accuracy of their re- sponses is calculated by comparing them to the correct answers. To effectively inte- grate our benchmark's 's data and reasoning processes from the proposed player-judge framework, we treat the question-answe pairs generated during the benchmark as auxiliary prompts. These are seamlessly incorporated into the LLMs' reasoning processes to enhance their thinking ability.
Figure 4: Performance of various LLMS on Riddle- Sense (dev set). Llama3 (8B & 70B) and GPT-4 are in the zero-shot setting, while others are trained on the training set of RiddleSense and CSQA 33 means models with our auxiliary reasoning prompts.
To implement this, we set up the game where one of the LLMs (e.g., Llama3-70B) acts as the player with WizardI.Mr continuing as the judge. We randomly select m situation puzzles (we set = 1 our experiment) from our benchmark dataset and engage both the player and the judge these selected puzzles using multi-turn player-judge framework. From each puzzle, gather the first

---

Table 5: Performance of LLMs on BrainTeaser. Models with mean with our auxiliary reasoning prompts. 'Ori', 'Sem', and 'Con' are 'Original', 'Semantic', and 'Context', respectively. 'OS' means "Ori&Sem' while 'OSC' is 'Ori&Sem&Con'. 'Overall' is the average of 'Ori', 'Sem', and 'Con'.
BrainTeaser (Sentence)
BrainTeaser (Word)
Instruction hased
Group-hased
Instruction hased
Group-hased
Ori
Scm
Con
OSC
Overall
Ori
Scm
Con
OSC
Overall
LLama3 8B
70.23
63.09
69.64
52,09
38.32
67.65
47.72
44.69
46.21
31,06
17.42
46.20
72.18
65.47
72.02
45.23
54.54
54.54
39.39
29.54
LLama3 70B
89.34
87.57
86.39
84.61
75.73
87.76
71.96
71.96
69.69
62.87
49.24
71.20
LLama3-7UB*
93.49
90.53
81.06
81.81
75.75
59.09
GPT-4
93.49
89.94
83.43
88.75
75.14
88.95
71.21
65.91
56.06
$9.09
41.66
64.39
GPTF4*
95.26
91.71
91.71
82.24
74.24
72.72
64.39
question-answer pairs to form an auxiliary reasoning prompt set U, comprising a total of 5 m pairs. This auxiliary prompt set U is then integrated into the original prompts P used by various LLMs. is important to note that the same auxiliary prompts are used across different LLMs during their evaluations on these two benchmarks. This keeps consistency in the supplementary data provided to each model, ensuring a fair comparison of their performance.
Results on RiddleSense and Brain Teaser Benchmarks. On the RiddleSense (dev set), Figure shows that with our reasoning prompts, the accuracy of various LLMs consistently improves (Llama3- 8B: 70.51 % - 72.18% Llama3-70B: 83.34% - 85.2 %, GPT-4: 88.83 - 89 89.23%) on the RiddleSense benchmark. The results demonstrate that the data and framework in our benchmark can be used to elicit the lateral thinking ability of LLMs when handling other lateral thinking tasks.
BrainTeaser features two main types of puzzles: sen- tence puzzles and word puzzles. Following its official settings, we assess model performance using two accu- racy metrics: 1) Instance-batt Accuracy: This metric individually evaluates each question-wweer original or reconstructed version. We present instance- based accuracy for both the original puzzles and their se- mantic and contextual reconstructions. 2) Group-based Accurucy: Under this metric, each original puzzle and its variants are treated as group. The model earns score of 1 only if it correctly solves all three puzzles within group. If it fails to do so, the score awarded is From Table 5 in the zero-shot setting, the accuracy of various LLMs consistently improves
Table 6: Impact of our data and reasoning processes. Models with 't' mean using our data only while with ** mean using both data and reasoning processes. 'RS' refers to the results on RiddleSense (Dev). 'BT (S.)' and 'BT' (W.)' are the overall re- sults on Teaser (Sentence) and Brain- Teaser (Word), respectively.
it the
(S.)
(W.)
Avg.
Llama3-8B
70.51
67.65
46,20
61.45
Llama3-88 8B"
70.32
69.62
52.28
64.07
BrainTeasee These demonstrate that the data and framework of our benchmark effectively elicit lateral thinking capabilities of LLMs when applied to various lateral thinking tasks.
IJama3
72.18
58.07
66.71
Llama3-70B
83.34
87.76
71.20
80.76
Llama3-70B!
82.95
91.12
77.27
83.78
1amma3-788
85.21
91.51
79.54
85.42
Ablation Study. We further explore the influence of our data and reasoning processes. In Table 6 incorporating our data into base LLMS leads to an increase average accuracy (Llama3-8B: 61.45 -+ 64.07; Llama3-70B: 80.76 - 83.78). This improvement is further enhanced to 66.71 and 85.42, respectively, when we integrate the reasoning processes. These results further demonstrate that both our data and reasoning processes can effectively enhance the lateral thinking capabilities of LLMs.
6 Conclusion
In this paper, we address the gap in the lateral thinking capabilities of LLMs despite significant advances vertical thinking skills. We propose SPLAT, a benchmark using Situation Puzzles to evaluate and elicit LLMs' LAteral Thinking by multi-turn player-judge framework, which departs from traditional model-based evaluations. This new framework evaluates the LLMs by interacting with judge model to solve puzzles, reducing the reliance on more robust evaluation models. Our experiments with SPLAT, including robust evaluation model like WizardL.M-2. demonstrate over 80% agreement with human judgements and show improvements when applied to another lateral thinking benchmark-Riddlese RiddleSense. This highlights SPLAT's effectiveness in evaluating and eliciting lateral thinking in LLMs, suggesting its potential broader application in Al research and development.
10

---

Acknowledgements This work was supported by the Centre for Augmented Reasoning, an initiative by the Department of Education, Australian Government.
References
[1] https:/llama..tta.cmm/llaaa 2024.
[2] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, L. Aleman, D. Almeida, J. Altenschmidt, Gpt-4 technical report. arXiv preprint arXiv:2303.081 2023.
S. Altman, S. Anadkat,
[3] R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, R. Taropa, P. Bailey, Z. Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305. 10403, 2023.
[4] J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang. X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, B. Hui, I. Ji, M. Li, Lin, R. Lin, D. Liu, G. Liu, Lu, K. Lu, J. Ma, R. Men, X. Ren, X. Ren, C. Tan, S. Tan, Tu, P. Wang, S. Wang, W. Wang. S. Wu, B. Xu, Xu, A. Yang. H. Yang. J. Yang. Yang. Y. Yao, B. Yu, Yuan.
Z. Yuan, J. Zhang, X. Zhang, Y. Zhang. Z. Zhang, C. Zhou, J. Zhou, X. Zhou, and T. Zhu. Qwen technical report, arXiv preprint M0ix:2339.160000 2023.
[5] Y. Bisk, R. Zellers, Gao, Choi, et al. Piqa: Reasoning about physical commonsense natural language.
In AAAI, pages 7432-7439, 2020.
[6] A. Bosselut, H. Rashkin, M. Sap, Malaviya, A. Celikyilmaz, and Y. Choi. Comet: Commonsense transformers for automatic knowledge graph construction. ACL, 2019.
[7] S. Bubeck, Chandrasekaran, R. Eldan, J. Gehrke, Horvitz, E. Kamar, Lee. Y. T. Lee, Y. Li. S. Lamdbberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.1277 2023.
[8] C. Burns, P. Izmailov, J. H. Kirchner, Baker, L. Gao, L. Aschenbrenner, Chen, A. Ecoffet, M. Joglekar, Leike, et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint XXXi:2312.093900 2023.
[9] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
[10] S. Chen, Y. Wang, Y--F Wu, Q-G. Chen, Z. Xu, W. Luo, K. Zhang, and L. Zhang. Advancing tool- augmented large language models: Integrating insights from emors in inference trees. Adv. Neural Inform. Process. Syst., 2024.
[11] P. Clark, 1. Cowhey, O. Etzioni. T. Khot, A. Sabharwal, C. Schoenick. and O. Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint arXiv: 180300000. 2018.
[12] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. TTTorek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 10.1:22110.14168, 2021.
[13] K. Daniel. Thinking, fast and slow. 2017.
[14] E. De Bono and Zimbalist. Lateral thinking. Penguin London, 1970.
[15] J. Devlin, M.-W. Chang, K. Lec, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. NAACL, 2019.
[16] J. Feng, Q. Sun, C. Xu, P. Zhao, Y. Yang, C. Tho, D. Zhao, and Q. Lin. Mmdialog: A large-scale multi-tum dialogue dataset towards multi-modal open-domain conversation. arXiv preprint arXiv:2211.05719. 2022.
[17] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding, arXiv preprint arXiv:2009.03300 2020.
[18] S. Huang, S. Ma. Y. Li, M. Huang. W. Zou, W. Zhang, and H.-T. Zheng. Lateval: An interactive lims evaluation benchmark with incomplete information from lateral thinking puzzles. arXiv preprint arXiv:2308. 10855, 2023.
[19] H. Jiang and Q.pp Zhang. Development and validation of team creativity measures: A complex systems perspective. Creativity and Innovation Management, (3(3):264-275. 2014.
[20] Y. Jiang, F. Ilievski, and K. Ma. Brainteaser: Lateral thinking puzzles for large language model. EMNLP. 2023.

---

[21] A. Köpf. Y. Kilcher. D. von Rütte, S. Anagnostidis, Z. R. Tam, K. Stevens, A. Barhoum, D. Nguyen.
O. Stanley, R. Nagyli, et al. Openassistant conversations-ddmeraaiiza large language model alignment. Adv. Neural Inform. Process. Syst., 36, 2023.
[22] B. Y. Lin, Z. Wu, Y. Yang. D.-H. Lee, and X. Ren. Riddlesense: Reasoning about riddle questions featuring linguistic creativity and commonsense knowledge. ACL, 2021.
[23] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L.. Zitnick. Microsoft Common objects context. In Eur. Conf. Comput. Vis., 2014.
coco:
[24] S. Longpre, L. Hou, T. Vu, A. Webson, H. W. Chung, Y. Tay, D. Zhou, Q. V. Le, B. Zoph, J. Wei, al. The flan collection: Designing data and methods for effective instruction tuning. In ICML, pages 22631-22648. 2023.
[25] B. MacCartney. Natural language inference. Stanford University, 2009.
[26] J. Meaney, S. Wilson, L. Chiruzzo, A. Lopez, and W. Magdy. Semeval 2021 task 7: Hahackathon, detecting and rating humor and offense. Proceedings of the 15th International Workshop on Semantic Evaluation (S0mEval-2022), pages 105-119, 2021.
[27] G. Mialon, C. Fourrier, C. Swift, Wolf, Y. LcCun, and T. Scialom. Gaia: a henchmark for general ai assistants. Int. Conf. Learn. Represent., 2024.
[28] S. Mishra, D. Khashabi, C. Baral, and H. Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. ACL. 2022.
[29] M. Mölle, L. Marshall, B. Wolf, H. L. Fehm, and J. Born. Eeg complexity and performance measures of creative thinking. Psychophysiology. 36(1):95- 1999.
[30] S. Reddy, D. Chen, and C. D. Manning. Coqa: conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249-266, 2019.
[31] N. Reimers and Gurevych. Sentence-bert: Sentenee embeddings using siamese nert-networks. EMNLP, 2019.
[32] P. Sadeghi. A. Abaskohi, and Y. Yaghoobzadeh. utebc-nlp at semeval-2024 task 9: Can Ilms be lateral thinkers? arXiv preprint arXiv:2404. 02474, 2024.
[33] A. Saha, V. Pahuja, M. Khapra, K. Sankaranarayanan, and S. Chandar. Complex sequential question answering: Towards learning to converse over linked question answer pairs with knowledge graph. In AAAI, volume 32, 2018.
[34] K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-100) 106, 2021.
[35] M. Sap. R. Le Bras, E. Allaway, C. Bhagavatula, N. Lourie, H. Rashkin, B. Roof, N. A. Smith, and Y. Choi. Atomic: An atlas of machine commonsense for if-then reasoning. In AAAI, volume 33, pages 3027-3035. 2019.
[36] R. Speer, J. Chin, and C. Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge. In AAAI, volume 31, 2017.
[37] A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta,
A. Garriga-Alonso, al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint 1511:5200600611, 2022.
[38] A. Talmor, Herzig, Lourie, and Berant. Commonsensega: question answering challenge targeting commonsense knowledge. NAACL-HLT, 2019.
[39] G. Todd, Merino. S. Earle, and J. Togelius. Missed connections: Lateral thinking puzzles for large language models. arXiv preprint arXiv:2404.1177 2024.
[40] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
[41] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei. N. Bashlykov, S. Batra, P. Bhargava,
S. Bhosale, et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint 2rXiv:2307.09288 2023.
12

---

[42] S. Waks. Lateral thinking and technology education. Journal of Science Education and Technology. 6:245-255, 1997.
[43] J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, et al. Emergent abilities of large language models. arXiv preprini arXiv:2206.07666 07620227222, 2022.
[44] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Adv. Neural Inform. Process. Syst., 35:24824-24837, 2022.
[45] C. Xu, Sun, K. Zheng, X. Geng, Zhao, Feng, C. Tao, and D. Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.1222 2023.
[46] R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint (ef5::19000008850 2019.
[47] L. Zheng, W.-L. Chiang. Y. Sheng. S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, al. Judging llm--a---judge with mt-bench and chatbot arena. Ady. Neural Inform. Process. Syst., 36, 2023.
[48] S. Zhong, Z. Huang, S. Gao, W. Wen, L. Lin, M. Zitnik, and P. Zhou. Let's think outside the box: Exploring leap-of-thought large language models with creative humor generation. IEEE Conf. Comput. Vis. Puttern Recog., 2024.
[49] Zhong, R. Cui, Guo, Liang, Lu, Y. Wang, A. Saied, Chen, and Duan. Agieval: human-centric benchmark for evaluating foundation models. arXiv preprint 1eXXiv:3344063360 2023.
[50] Y. Zou W. Lu. Joint detection and location of english puns. NAACL, 2019.
13

---

A Appendix supplemental material
A.1 Limitations and Social Impacts
The nature of lateral thinking tasks often encourages non-linear and novel thought processes, which could lead to unpredictable or unexpected outputs from LLMs. While creativity is desired, these outputs might sometimes include harmful, misleading, information, especially if not properly supervised. To alleviate this, we will engage in robust testing across diverse scenarios to understand the boundaries and limitations of the model's creative outputs. Testing should aim to uncover any potential for generating harmful or inappropriate content.
Limitations
dangerously incorrect
Broader Impacts Our work's advancement in lateral thinking capabilities of Large Language Models (LLMs) could significantly benefit society by enhancing problem-solving abilities across various domains, such as education, engineering, and the arts. By enabling AI to approach problems with level of creativity akin to humans, these models could provide innovative solutions that defy conventional thinking patterns.
However, the enhancement of LLMs in lateral thinking also brings potential risks, such as the misuse of technology for malicious purposes like fraud misinformation, job displacement in creative and problem-solving professions, and increased societal dependence on technology for innovative thinking. To counter these risks, it essential to establish robust ethical guidelines and transparent Al usage policies. We should seek to promote a culture of ethical AI use combined with ongoing education about Al's societal impacts. This will help ensure that the benefits of advanced Al are realised while minimising its potential harms.
A.2 Character Initialisation
Judge: The guidelines for a judge include:
Scenario Understanding: Read the given short story and the corresponding answer. Make
sure you can understand the both story and answer, and their logical relationships.
Response: Respond to the user's questions with only "yes", "no", "irrelevant". Your answers should help guide the user towards understanding the full context of the puzzle.
Evaluation: Evaluate the user's final answer carefully. If the user's answer is correct, confirm their success and say "Congratulations", If incorrect, allow them continue asking questions or provide subtle hints to steer them in the right direction.
Question Handling: If there are multiple questions, respond to the first question and ignore others.
Player:
The guidelines for player include:
Judge will give you puzzle in the form of short story that is logically self-consistent. The story will contain all the information needed to solve the puzzle, but some details may be omitted.
Your task is ask questions gather more information and help you find the solution to the puzzle. You can ask any yes/no questions, and the judge will answer truthfully with yes/no/irrelevant
During the questioning process, please carefully analyse the known information and clarify the logical relationships between the clues. Don't tt overlook any seemingly trivial details.
When you believe you have collected enough information, please provide your answer and explain how you reasoned the conclusion based on the known clues.
If your answer correct, the game ends; it's incorrect, you can continue asking questions until you find the correct answer.
You will never ask questions with answers other than yes/no/irrelevant. Remember, the key to the puzzle to collect clues through reasonable questions and use logical reasoning to draw conclusions.
14

---

A.3 Agreement Evaluation
Inspired by to assess the alignment of judge's responses with human preferences, we consider an agreement evaluation among them w.r.f. judgements for both final predicted answers and the intermediate reasoning processes. We randomly select 20 puzzles from each difficulty level, which is 60 puzzles in total. For each puzzle, we ask 3 different individuals to assess the final answer agreement and the reasoning process agreement. We finally obtain 156 judgements from humans for the final answer and 2,271 judgements for the reasoning process.
Final Answer Agreement To quantitatively evaluate the semantic alignment between the final predicted answers and reference answers, we first ask humans to assess the semantic similarity between each predicted answer and the corresponding reference answer. Subsequently, we measure the agreement performance between the human judgement and the evaluations from the judge model. For example, if three humans vote 'matched', 'matched', and 'unmatched" for puzzle, respectively, the agreement among them, noted as Nuuman-human is only 5. as there are three pairs (matched, matched), (matched, unmatched), and (matched, unmatched). If the judge model vote 'matched', the agreement between humans and the judge model is 2
This process can be mathematically structured as follows:
Define an alignment metric M that quantifies the agreement between the judge's judgement and human assessments. The metric can be expressed as
M
m Σ 8(1,h;)) hz)
where ö is an indication function that equals if the judgement j; E (matched, unmatched} from the judge model matches the human judgement húp and 0 otherwise. Here, m is the total number of comparisons made between the judgements from the judge model and humans, respectively. A higher value of M indicates a stronger alignment of the decisions of the judge model with human preferences, suggesting that the judgements rendered by the automated system closely mirror those of human evaluators terms of understanding and evaluating the semantic content of the answers.
Reasoning Process Agreement To assess the agreement on the reasoning process, particularly for intermediate questions where no predefined reference answers exist, we employ a methodology where humans are asked to respond to these questions based on the context provided by the given story and the final reference answer. Their responses are categorised (yes, no, irrelevant). We then evaluate the alignment of these human responses with those from the judge model. This process can be mathematically described as follows:
Define a reasoning agreement metric R to quantify the alignment between the human responses and the judgements provided by the judge model for intermediate questions:
R
=
8(a;, uz)
1=1
where ó is an indication function that equals if the human response to the intermediate question matches the judge model's 's response ag, and otherwise. Here, represents the total number of intermediate questions evaluated. A higher value of R indicates strong alignment between the human responses and the judge model's 's decisions regarding the intermediate reasoning steps.
Instruction for Human Figures 5 and 6 provide the instruction shown for humans and ask them to write the judgement for the final answer and for the reasoning process, respectively.
A.4 Examples
We present several examples illustrating the process of situation puzzle games. In these examples, Llama3
[I serves as the player M-2 (8x22B) 45 functions as the judge.
Puzzle The title of the puzzle Sweet Dreams. Matthew keeps reading bedtime story to his son despite the blackout. Why?
15

---

Instruction for Human Evalustors
Objective: The goal to quanitativeyy evaluate the semaantcc alignment between the final predicted answers provided by the LLMs and the reference answers for each scenario.
Task Deseription:
1. Review the Materials: You will be provided with set of predicted answers generated by the LLMs and the sorresponding reference answers each scenario.
2. Assess Semantic Similrity. For each (predicted answer and reference answer), assess the semantic similarity. Consider how
predicted ansever captures the essence, meaning, and nuances reference ansmer. Evaluate whether the predicted answer logically and contestually aligned with the reference answer.
3. Scoring: predicted
matches the reference If the predicted does match the reference semantically.
terms semantic content and context,
o.
4. Submission: Once have completed assessment for al provided pairs. submit
Purpose Evaluation:
The results
judgments and This understanding how judge evalustions perceptions semantic aligmment
will
the performance
evaiuations
Additional
Take your
thorough understanding both your evaluation onttrraa scanarios maintain reliabliiy
Thank contribution this ovaluation task. Your insights crucial performance relibiltyy understanding generating contestually appropriate responses.
Figure 5: Instructions ask humans to write the judgement for the final answer.
Instruction for Human Evaluators:
The goal is assess the agreement reasoning processes, particularly for intermediate questions in contest sitution puzzles, where no predefined reference answers exist.
Objective:
Task Deseription:
Review the Contest and Questions: You will be provided with series of intermediate questions that arise during the evalustion situation puzzles. For each question, you will also receive the associated story context and the final reference answer.
Respond Questions: Based on the information provided the story and the contect of the final fefernnee anower, you are required to respond each intermediate question. Your task is determine the most appropriate response to the question given the overall contast the story and the and goal of the puzzle.
Categorize Your Responses: Yes: Choose this if the information the story and the reference answer supports
posiive response to the question No: Choose this if the information contradicts does not support the question inelevant Choose this if question does not pertain to the contest the story or the reference answer, or if cannot be answered defintively based on the provided information.
Documentation Record your response for each question the provided format Ensure maintain consistency and objectivity your responses. basing each the given story contest and the final reference answer.
Submission: Once you have completed your responses for all the provided questions, submit your answers as per the instructions given.
Purpose of the Evaluation:
Your resporses are crucia for measuring how closely the judge model's reasoning aligns with human reasoning. enhancing our understanding and development of the model's capabilites.
Additional Notes:
Take adequate time understand the context and the implications of each question within the narrative of the story. Strive consistency in how apply the onteria for Yes, No: and Tirelevant across different questions to ensure reliability the assesoment
Thank you for partccpppp participation and mediclous attention to detail this assessment Your contributions are invaluable improving the interpretabilly fit and accuracy of models simulating human-lts lik reasoning processes.
Figure 6: Instructions to ask humans to write the judgement for the reasoning process.
16

---

Interaction Rounds
Background: Read and fully understand the provided short story but do not show the answer to the user.
Round 1: Player: Let's start. Here's 's my first question:
Is the bedtime story being read from physical book? Judge: Yes.
Round 2: Player: That helps! Here's 's my next question:
Is the blackout caused by power outage? Judge: Yes.
Round 3: Player: Got it! Next question:
Matthew reading the story by candlelight or some other alternative light source? Judge: No.
Round 4: Player: That helps to rule out some possibilities. Here's ' my next question:
Had Matthew already started reading the story before the blackout occurred? Judge: Yes.
Round 5: Player: That's useful clue. Here's my next question:
Does Matthew already know the story by heart?
Judge: No.
Round 6: Player: That helps to rule out another possibility. Here's my next question:
the room where Matthew reading the story already familiar to him, i.c., he has been in the room before the blackout?
Judge: Yes.
Round 7: Player: That's helpful. Here's my next question:
Is Matthew's already asleep least lying in bed when Matthew continues reading the story despite the blackout? Judge: No.
Here's my next question: Is Matthew's son visually impaired or blind? Judge: No.
Round 8: Player:
Round 9: Player: That helps rule out another possibility. Here's my next question:
Is the story being read from a book with raised print or Braille, allowing Matthew to read it by touch? Judge: Yes.
Round 10: Player: think I'm getting close to solving the puzzle. Here's my next question: Is Matthew also blind or visually impaired? Judge: Yes.
Round 11: Player: think have enough information to solve the puzzle. Here's my answer: Matthew keeps reading the bedtime story to his son despite the blackout because he is blind and can read the book by touch through Braille, and the blackout does not affect his ability to read. Judge: Congratulations.
Puzzle The title of the puzzle is Fatal Shot. hunter aimed his gun carefully and fired. Seconds later, he realized his mistake. Minutes later, he was dead. Why?
Interaction Rounds
Background: Read and fully understand the provided short story but do not show the answer to the user.
Round 1: Player: Let's start by asking some questions to gather more information. Here's my first
question:
Was the hunter aiming at an animal when he fired his gun? Judge: Yes.
17

---

Round 2: Player: That helps to narrow it down bit. Here's 's my next question: Was the animal he was aiming a large game animal, such as a bear or deer? Judge: Irrelevant.
Round 3: Player: That helps to rule out some possibilities. Here's my next question: Did the hunter die as direct result of the bullet he fired? Judge: No.
Round 4: Player: That's interesting. So the hunter's death wasn't directly caused by the bullet he
fired. Here's s my next question:
Was the hunter a location that was somehow hazardous or dangerous when he fired the gun?
Judge: Yes.
Round 5: Player: That's good clue! With that in mind, here's 's my next question:
Was the hunter in a location where the sound of the gunshot could cause a problem, such as near a cliff, on a boat, or in confined space? Judge: Yes.
Round 6: Player: think I'm starting to get an idea. Here's my next question:
Was the hunter cliff or mountain when he fired the gun, and did the sound of the gunshot cause rockfall or avalanche that killed him?
Judge: Yes.
Round 7: Player: was hoping that might be the case. Well, I think have enough information to solve the puzzle now.
Here's s my answer:
The hunter aimed his gun carefully and fired, but seconds later, he realized his mistake. The sound of the gunshot caused rockfall or avalanche, which killed him minutes later.
I reasoned this out by combining the clues that the hunter was aiming at an animal, that he died as an indirect result of firing the gun, and that he was in hazardous location where the sound of the gunshot could cause a problem. Putting all those clues together led me the conclusion that the sound of the gunshot triggered rockfall or avalanche that killed the hunter.
Judge: Your answer on the right track, but is not fully correct. The key detail you are missing is the specific cause of death. Please continue to ask questions or provide another answer focusing on exact cause of the hunter's death. Would you like to ask any specific questions to help refine your answer?
Round 8: Player: Thank you for the feedback. I'm glad I was on the right track, but I'd like to refine my answer get the exact cause of death.
Here's question to help me refine my answer: Did the hunter die from being crushed hit by the falling rocks or debris, or was the cause of death something else related to the rockfall or avalanche?
Judge: No.
Round 9: Player: So the hunter didn't i die from being crushed hit by the falling rocks or debris. Here's ' another question to help me refine my answer: Did the hunter die from a fall, such falling off cliff or down slope, result of the rockfall or avalanche?
Judge: No.
Round 10: Player: So the hunter didn't die from a fall either. Here's another question to help
refine my answer:
Was the hunter's death related a lack of oxygen, such being buried under snow or trapped in small space, as a result of the rockfall or avalanche? Judge: Yes.
Round 11: Player: I think have it now!
Here's ' my revised answer:
The hunter aimed his gun carefully and fired, but seconds later, he realized his mistake. The sound of the gunshot triggered an avalanche, which buried him under the snow. Minutes later, he died from asphyxiation due lack of oxygen.
I'm confident that this is the correct answer. The hunter's death was an indirect result of firing the gun, and the specific cause of death was asphyxiation due being buried under the snow. Judge: Congratulations.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the contributions and match the experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of our work and propose directions for future research in the supplementary.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
19

---

Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: In the method and experiment sections.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
20

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We offer a comprehensive explanation of our dataset in the paper and supple- mentary. The code and dataset will be made available upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ https/ccppplliigggdeeeeeeeeeeeee Coaee//dddeSebmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the details are provided in the paper and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: this paper, we primarily introduce benchmark for evaluating and eliciting the lateral thinking ability of L.MMs. Statistical significance is not emphasised.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
21

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [No]
Justification: As we use the APIs for all the LLMs employed in this work, any standard personal computer can handle it.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics sectps://neuripss https///cccppcccccccccccceeeeee.
Answer: [Yes]
Justification:
We have reviewed and agree with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We thoroughly discuss both the positive and negative potential societal impacts of our work in the supplementary.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
22

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification:
We check the scraped data by humans one by one.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
23

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the datasct/code/modell as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification:
We include the full text of instructions given to participants in the supplemen-
tary.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.


Reviews:
[{"review_id": "hDslRv79cd", "reviewer": "Reviewer_s9Dk", "summary": "This paper introduces SPLAT, a novel benchmark for evaluating and eliciting lateral thinking abilities in Large Language Models (LLMs) using situation puzzles. The key contributions include: A new dataset of 975 graded situation puzzles across three difficulty levels.\nA multi-turn player-judge evaluation framework that reduces reliance on stronger evaluation models. Demonstration that the benchmark can both evaluate and elicit lateral thinking in LLMs. Experimental results showing that a robust evaluation model (WizardLM-2) closely matches human judgments. Evidence that applying SPLAT's data and reasoning processes to other lateral thinking benchmarks leads to performance improvements in LLMs.", "questions": "1. How does the performance of LLMs on SPLAT correlate with their performance on other, more traditional NLP tasks? Is there evidence that lateral thinking ability is distinct from other language model capabilities?\n2. Have you considered ways to automatically generate new situation puzzles to expand the dataset and improve scalability?\n3. How sensitive is the performance of LLMs on SPLAT to the specific prompts or instructions given? Could you elaborate on the prompting strategy used?", "limitations": "None.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The paper presents a novel approach to assessing lateral thinking in LLMs, an area that has been underexplored compared to vertical thinking. The use of situation puzzles and the multi-turn player-judge framework are creative solutions to the challenges of evaluating open-ended, creative thought processes.\n2. The methodology appears rigorous, with careful data collection, annotation, and difficulty categorization. The multi-turn player-judge framework is well-designed to overcome limitations of traditional model-based evaluations. The experimental results, including comparisons with human judgments and applications to other benchmarks, provide strong evidence for the effectiveness of the approach.\n3. The paper is well-structured and clearly written. The task definition, data construction process, and evaluation framework are explained in detail. Figures and tables effectively illustrate key concepts and results.\n4. This work addresses an important gap in the evaluation of LLMs by focusing on lateral thinking, which is crucial for creative problem-solving and innovation. The benchmark has potential applications beyond evaluation, as demonstrated by its ability to improve LLM performance on other lateral thinking tasks. This could lead to advancements in developing more creative and flexible AI systems.", "weaknesses": "1. While the data collection process is described, there's minimal discussion of potential biases in the dataset, such as cultural specificity of the puzzles or biases introduced during the human annotation process.\n2. The paper would benefit from ablation studies to isolate the impact of different components of the SPLAT framework, such as the difficulty categorization or the multi-turn questioning process.\n3. The paper lacks a discussion of potential ethical implications or misuse of improved lateral thinking capabilities in LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1. There's minimal discussion of potential biases in the dataset.**\n\nDuring our data collection process, we also track user preferences for each situation puzzle, measured by a preference score (\\%). We notice a pattern where puzzles (about 1\\% of our dataset) with lower preference rates (30\\%-40\\%) show greater variation in the time taken to solve them, ranging from 5 to 26 minutes for puzzles categorised as \"Medium\". In contrast, puzzles with higher preference rates (above 80\\%) exhibit less time variation, with solving times ranging from 9 to 17 minutes for \"Medium\" puzzles. This suggests that annotations may introduce more noise when users engage with puzzles they find less appealing.\n\nTo avoid this bias, we will enhance the diversity of the puzzles to ensure a broader appeal across different user groups, thereby reducing the likelihood of low preference scores. Besides, we plan to implement a dynamic refinement method during data collection to enhance puzzle engagement. If a puzzle receives a low preference score, we will revise it based on user feedback and re-release it to assess improvements in user preference.\n\n**W2. Ablation studies to isolate the impact of different components of the SPLAT framework.**\n\nWe conduct an ablation study to explore the influence of our data and reasoning processes. In addition to the RiddleSense, we include the BrainTeaser benchmark, which features both word and sentence puzzles. As shown in the table below, incorporating our data into base LLMs leads to an increase in average accuracy (Llama3-8B: 61.45 -> 64.07; Llama3-70B: 80.76 -> 83.78). This improvement is further enhanced to 66.71 and 85.42, respectively, when we integrate the reasoning processes. These results further demonstrate that both our data and reasoning processes can effectively enhance the lateral thinking capabilities of LLMs.\n\n| Model | RiddleSense (Dev) | BrainTeaser (Sentence, overall) | BrainTeaser (Word, overall) | Average  |\n| ------- | ------- | ------- | ------- | ------- |\n| Llama3-8B (base) | 70.51 | 67.65    | 46.2 | 61.45 |\n| Llama3-8B (data)| 70.32 | 69.62    | 52.28| 64.07 |\n| Llama3-8B (data+reasoning)  | 72.18 | 69.89    | 58.07| 66.71 |\n| Llama3-70B (base)| 83.34 | 87.76    | 71.2 | 80.76 |\n| Llama3-70B (data) | 82.95 | 91.12    | 77.27| 83.78 |\n| Llama3-70B (data+reasoning) | 85.21 | 91.51    | 79.54| 85.42 |\n\n**W3. Potential ethical implications or misuse.**\n\nThank you for the kind reminder. We have discussed these in Section A.1 of our supplementary. Briefly, while enhancing LLMs in lateral thinking offers numerous benefits, it also introduces potential risks such as the misuse of technology for fraud or misinformation. Thus, we emphasise the importance of building robust ethical guidelines and transparent AI usage policies.\n\n**Q1. How does the performance of LLMs on SPLAT correlate with their performance on other, more traditional NLP tasks? Is there evidence that lateral thinking ability is distinct from other language model capabilities?**\n\nWhile there is some overlap in the skills required for SPLAT and traditional NLP tasks, they are not exactly the same.\n\nModels (e.g., GPT-4 and GPT-4 Turbo) that perform well on standard vertical thinking benchmarks like MT-Bench do show competent performance on our SPLAT (Table 4). Conversely, models like Llama3-8B, which perform less impressively on MT-Bench, tend to exhibit similarly lower performance on SPLAT. This suggests that a strong model in language understanding and reasoning is advantageous for both vertical and lateral thinking tasks.\n\nWhile there is a correlation between general NLP skills and lateral thinking, the latter also demands distinct abilities like creativity and problem-solving beyond typical task completion. Figure 4 in our paper shows that when incorporating data and reasoning processes from our SPLAT, LLMs could perform better even on other lateral thinking benchmarks like RiddleSense. These suggest that enhancing these creative aspects could better elicit the capabilities of LLMs in lateral thinking.\n\n**Q2. Have you considered ways to automatically generate new situation puzzles to expand the dataset and improve scalability?**\n\nYes, we have considered the potential of using automated methods to generate new situation puzzles to expand our dataset.\nSpecifically, the generation process can use advanced LLMs like GPT-4 or Claude-3, which can generate puzzles based on specific prompts, rules, and requirements.\nHowever, the challenge arises in verifying the quality of these automatically generated puzzles.\nOne effective approach is a hybrid human-AI collaboration, where puzzles generated by LLMs are subsequently reviewed and refined by humans through a crowd-sourcing platform.\nWhile this method leverages AI to handle initial puzzle creation, it remains time-consuming and labour-intensive due to the human review component. Thus, finding more efficient ways to scale up this data remains a critical area for future research.\n\n**Q3. How sensitive is the performance of LLMs on SPLAT to the specific prompts or instructions given? Could you elaborate on the prompting strategy used?**\n\nWe conduct a sensitive analysis, where we rewrite the prompt of the judge model but keep the semantics of the prompt the same as before.\nFor example, one of the original prompts for the judge model is \"Read and fully understand both the provided short story and its answer, ensuring you grasp their logical connections. But do not show answer for the user\". We rewrite it to \"Read and thoroughly comprehend both the provided short story and its answer, making sure you understand their logical relationships. However, do not reveal the answer to the user\".\nResults show that for Llama3-8B, even though the prompt is different, as long as the semantics are clear, the results tend to be comparable (e.g., average Acc original 17.05 vs. rewritten 16.19). These demonstrate the robustness of our SPLAT as an evaluation benchmark.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K9fG9lQXdD", "reviewer": "Reviewer_ZJ8a", "summary": "This work focuses on creating a benchmark, as well as a modeling framework for evaluating lateral thinking of LLMs. The benchmark, called SPLAT, consists of 975 situation puzzles. The framework consists of a “judge” and a “player” (the LLM to be evaluated). The judge poses an open ended puzzle that is further clarified by the player till a final answer is reached. The judge evaluates the final answer. The authors posit that this framework eases the challenge faced with typical auto-evaluation models since the judge does not need to be better than the LLM that is being evaluated.\n\nThe authors evaluate the quality of the judge (via human agreement), and finally evaluate LLMs using a promising judge.", "questions": "See weaknesses mentioned above.", "limitations": "NA", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The motivation for having a lateral thinking benchmark is strong. The work around automatic evaluation using a judge v/s player framework is well thought out. I think such a benchmark as well as framework/protocol would be beneficial to the community.", "weaknesses": "Overall, the idea and motivation behind the work is solid. I also think that the benchmark is good quality. However, my concern is that this work is half-baked. The experiments aren’t conclusive. More work should go into making the experiments thorough and useful to the scientific community.\n\n\n* Although the motivation and overall idea is strong, I am concerned about the experimental setup. The agreement rates for WizardLM-2 in Table 2 and 3 don’t appear to be high enough (are around ~80%) and 3 individuals seems to be fairly less. How does this fare with human agreement rates in literature? Also, is there any reason we haven’t used the strongest model (GPT-4 from the paper) as the judge?\n\n* I also am concerned about Table 4. The authors should consider using strong base models for evaluation (that are closer to GPT-4 on public benchmarks)\n\n* It would be good to introduce modeling methods to improve upon the metrics that are obtained for models on SPLAT. This will give more insight into how this benchmark is valuable to the community.\n\n* nit: I see mentions of “using data and reasoning processes from our framework” consistently throughout the paper. This should be framed better and be more precise.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1. Although the motivation and overall idea are strong, I am concerned about the experimental setup. The agreement rates for WizardLM-2 in Tables 2 and 3 don’t appear to be high enough (are around ~80\\%) and 3 individuals seem to be fairly less. How does this fare with human agreement rates in literature?**\n\nThank you for pointing out this. Due to the strict calculation method of our agreement, the agreement between humans cannot always reach 100\\%.\nSpecifically, as mentioned in the supplementary. if three humans vote 'matched', 'matched', and 'unmatched' for a puzzle, respectively, the agreement among them, noted as 'human-human', is only 1/3, as there are three pairs: (matched, matched), (matched, unmatched), and (matched, unmatched). If the judge model votes `matched', the agreement between humans and the judge model is 2/3.\n\nBut even under such a strict evaluation metric, the judge model WizardLM-2 (8x22B) aligns closely with human judgements, achieving over 80\\% agreement on both final answers and reasoning processes. Specifically, as shown in Table 2, the final answer agreements between WizardLM-2 (8x22B) and human evaluations are 100\\%, 87.50\\%, and 100\\% for easy, medium, and hard puzzles, respectively. Similarly, the agreement on reasoning processes (Table 3) shows robust results with 84.18\\%, 89.86\\%, and 90.15\\% for each difficulty level, respectively.\n\n**W2. Also, is there any reason we haven’t used the strongest model (GPT-4 from the paper) as the judge?\nI also am concerned about Table 4. The authors should consider using strong base models for evaluation (that are closer to GPT-4 on public benchmarks).**\n\nBefore the submission deadline, data from its official website [1] indicates that WizardLM-2 (8x22b) performs competitively on benchmarks like MT-Bench, comparable to leading models such as GPT-4, i.e., WizardLM-2 (8x22b): 9.12 vs. GPT-4-0314: 8.96 vs. GPT-4-1106-Preview: 9.32. Besides, we also conduct an experiment that employ GPT-4 as the judge model, which yields results comparable to WizardLM-2 in both final answer agreement and reasoning process agreement. For instance, on \"Hard\" puzzles, GPT-4 achieved an 89.85\\% final answer agreement rate, closely matching WizardLM-2's 88.24\\%.\n\n[1] https://wizardlm.github.io/WizardLM2/\n\n**W3. It would be good to introduce modeling methods to improve upon the metrics that are obtained for models on SPLAT. This will give more insight into how this benchmark is valuable to the community.**\n\nTo enhance the alignment/agreement between the judge model and human assessments, we plan to focus on two approaches in future work: 1) Employing in-context learning methods such as Chain-of-Thought (CoT) [2] or Tree of Thought (ToT) [3], which follow human-like reasoning patterns and thus can improve alignment; 2) Utilising training-based methods like Supervised Fine-Tuning (SFT) and Direct Preference Optimisation (DPO) [4], which involve training models directly on lateral thinking puzzles to enhance their accuracy in evaluating such scenarios. These methods, while promising, are beyond the scope of this paper. We regard them as our future work.\n\n[2] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS, 2022.\n\n[3] Tree of Thoughts: Deliberate Problem Solving with Large Language Models. NeurIPS, 2023.\n\n[4] Direct Preference Optimisation: Your Language Model is Secretly a Reward Model. NeurIPS, 2023.\n\n**W4. nit: I see mentions of \"using data and reasoning processes from our framework\" consistently throughout the paper. This should be framed better and be more precise.**\n\nThis refers to 'using the question-answer pairs generated during our SPLAT's player-judge interactions as auxiliary prompts'. We will revise this thoroughly and make it clearer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SKF4mZviyQ", "reviewer": "Reviewer_6oC8", "summary": "This paper focuses on lateral thinking, which is about creativity and viewing problems from multiple angles. To sovle the challenge that the complexity of assessing creative thought processes and the scarcity of relevant data, this paper introduces SPLAT, a benchmark leveraging Situation Puzzles to evaluate and elicit LAteral Thinking of LLMs. By employing a new multi-turn player-judge framework instead of the traditional model-based evaluation, the proposed method lessens dependence on more robust evaluation models, enabling the assessment of state-of-the-art LLMs. Moreover, through applying data and reasoning processes from our benchmark to another lateral thinking-related benchmark, like RiddleSense, leads to performance enhancements.", "questions": "1. Why not provide more than one answer for each question (I do not think it is challenging, there are multiple answers to a question on the internet such as Quora)? Since one such question could have multiple answers, more reference answer might also be helpful for evaluation.\n2. A overall evaluation metric is needed, some model might have a high acc with a high average round, you should define a overall metric to combine these two metrics for a clear comparison of existing LLMs.\n3. It is better to adopt more benchmarks for validating the effectiveness of the proposed dataset (more than just RiddleSense).", "limitations": "None", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper focuses on the lateral thinking problem, which is important to creativity and viewing problems from multiple angles.\n2. The proposed dataset is open-ended and the evaluation method is good.\n3. The paper is well-written.", "weaknesses": "1. The size of the dataset is samll.\n2. The reference answer is limited to 1, while the questions are one-to-many questions.\n3. The in-depth analysis is not enough", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1. The size of the dataset is small.**\n\nPlease refer to General Response **G2**.\n\n**W2+Q1. The reference answer is limited to 1, while the questions are one-to-many questions. Why not provide more than one answer for each question (I do not think it is challenging, there are multiple answers to a question on the internet such as Quora)? Since one such question could have multiple answers, more reference answers might also be helpful for evaluation.**\n\nWe have tried to directly gather questions and answers from question-and-answer websites like Quora, where we observed that situation puzzle queries often ask \"How to build a situation puzzle?\" with responses typically listing several one-to-one puzzle pairs rather than multiple answers to a single question.\n\nThus, as discussed in the paper, we originally wanted to evaluate it like in MS COCO, where multiple reference answers are provided for each image, and metrics are calculated based on the closest reference answer. However, due to the difficulty in collecting multiple reference scenarios for situation puzzles, we adapt our approach. Instead, we run each puzzle multiple times (denoted as $R$) and select the closest match to the reference scenario from these iterations as the final result. This method preserves the diversity of potential answers and mitigates the effects of hallucinations in LLMs by allowing multiple evaluations per puzzle.\n\nIn the future, we seek to employ crowd-sourcing platforms like Amazon Mechanical Turk to collect diverse answers to the same puzzle. Besides, we will provide contributors with guidelines that encourage diverse thinking, asking them to provide unique answers or perspectives on the same puzzle. \n\n**W3+Q3. The in-depth analysis is not enough. It is better to adopt more benchmarks for validating the effectiveness of the proposed dataset (more than just RiddleSense).**\n\nPlease refer to General Response **G1**.\n\n**Q2. An overall evaluation metric is needed, some models might have a high acc with a high average round, you should define an overall metric to combine these two metrics for a clear comparison of existing LLMs.**\n\nThank you for your constructive suggestion. Based on the calculation of Accuracy (Acc, \\%) and Average Round (Rnd) in our paper, we further define an overall evaluation metric as Overall = $1/N\\sum_{i=1}^N\\mathbb{I}$(sample$_i$) / Rnd$_i$, where $\\mathbb{I}$ is an indicator function that returns 1 if the scenario deduced by LLMs matches the reference scenario/answer semantically, and 0 otherwise. Given that $\\mathbb{I}$(sample$_i$) takes values in \\{0, 1\\} and Rnd$_i$ ranges from 1 to a maximum (maximum = 15 in our paper), the Overall metric spans from 0 to 1, with higher values indicating better performance.\n\nThe table below reflects trends similar to those discussed in our paper, where GPT-4 and its Turbo variant, as well as WizardLM-2 (8x22B) show strong performance and outperform other models. GPT-4 leads with an Overall score of 8.16 (Avg, x100), indicating high capability. However, it is still far from saturation, which suggests room for further improvement.\n\n| Model | Overall (Easy, x100) | Overall (Medium, x100) | Overall (Hard, x100) | Overall (Avg, x100) |\n| --- | --- | --- | --- | --- |\n| LLama3-8B | 3.65| 1.58 | 0.50 | 1.91 |\n| LLama3-70B  | 8.67  | 3.54    | 1.36  | 4.52 |\n| Qwen1.5-32B| 6.39  | 2.47    | 2.01  | 3.62 |\n| Qwen1.5-110B | 9.71  | 3.74    | 2.16  | 5.20  |\n| WizardLM-2-8x22B | 11.97 | 4.67    | 2.74  | 6.46 |\n| GPT-4 Turbo | 13.22 | 5.06    | 1.35  | 6.54 |\n| GPT-4  | 15.25 | 6.71    | 2.52  | 8.16 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4MgVVV3JZH", "reviewer": "Reviewer_eDsP", "summary": "The paper constructs the benchmark SPLAT to evaluate and elicit lateral thinking abilities in LLMs. It includes questions similar to brain teasers, which describe scenarios and ask non-obvious questions for guessing. \n\nExample: \n\n__Question/Story__: A hunter aimed his gun carefully and fired. Seconds later, he realized his mistake. Minutes later, he was dead. \n__Reference Answer__: He hunted in snow-capped mountains. The shot provoked an avalanche, which covered the man. He died of strangulation.\n\nThey also propose a multi-turn player-judge evaluation framework that reduces reliance on stronger evaluation models.", "questions": "1. Similar datasets have existed before, as mentioned in the paper, such as RiddleSense and Oogiri. Here's an example from RiddleSense: \n\nMy life can be measured in hours. I serve by being devoured. Thin, I am quick; Fat, I am slow. Wind is my foe. What am I? (A) paper (B) candle (C) lamp (D) clock (E) worm\n\nIn comparison, SPLAT has longer questions and answers. Also, the reference answers for SPLAT are open-ended, while previous ones are all multiple-choice, which is claimed as a contribution. However, this also makes the evaluation more challenging. What makes you think open-ended is a much better option that makes it a contribution?\n\n2. Author claim that their methods reduces reliance on stronger evaluation models, but it seems they are still using WizardLM-2 as a component (Judge Model) of their framework?", "limitations": "The size for this benckmark is small compared with previous similar datasets that have existed before, which include many more puzzles compared to the proposed SPLAT.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The SPLAT dataset is different from previous datasets, with graded difficulty levels and harder questions.\n2. the multi-turn player-judge framework are claimed to be able to  reduces reliance on stronger evaluation models. (I acutally feel a bit confused for this part, see qustions)\n3. The authors conduct experiments to validate their approach, demonstration that SPLAT can improve LLM performance on other lateral thinking tasks suggests broader applicability.", "weaknesses": "1. \"The dataset and framework are designed not only to evaluate but also to actively elicit lateral thinking in LLMs. Experiments show that using data and reasoning processes from our framework leads to improved performance of LLMs, even when applied to other lateral thinking benchmarks.\" This part really needs ablation studies. It's unclear which parts are effective. For example, are the reasoning processes, the entire dataset, or similar questions in previous data responsible for the improvement? Did this data improve the model's ability to think about these types of questions, or is the improvement possibly due to some tricks? Moreover, why was testing only done on RiddleSense when many other benchmarks were reported earlier? Is this cherry-picking?\n\n2. A player-judge evaluation framework based on WizardLM-2 is proposed as an alternative to human judges to evaluate the answers, but it shows much worse agreement on hard questions compared to inter-human agreements. However, harder questions are actually proposed as a contribution of this work. As the answers for such puzzles can be \"off by a hair but miss by a mile,\" it might require human evaluation for the hard puzzles. It is also a bit confusing why the human agreement on Medium questions is much lower than on hard and easy ones.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1-1. \"The dataset and framework are designed not only to evaluate but also to actively elicit lateral thinking in LLMs. Experiments show that using data and reasoning processes from our framework leads to improved performance of LLMs, even when applied to other lateral thinking benchmarks.\" This part really needs ablation studies. It's unclear which parts are effective. For example, are the reasoning processes, the entire dataset, or similar questions in previous data responsible for the improvement? Did this data improve the model's ability to think about these types of questions, or is the improvement possibly due to some tricks?**\n\nWe conduct an ablation study to examine the impact of our data and reasoning processes on model performance. Besides RiddleSense, we incorporate another lateral thinking benchmark (i.e., BrainTeaser), which includes word and sentence puzzles. The results, as shown in the table below, indicate that incorporating our dataset improves average accuracy (Llama3-8B: from 61.45 to 64.07; Llama3-70B: from 80.76 to 83.78). When we integrate the reasoning processes, these scores further increase to 66.71 and 85.42, respectively.\nThese results further demonstrate that our data and reasoning processes can effectively boost LLMs' ability in handling lateral thinking tasks rather than tricks.\n\n| Model | RiddleSense (Dev) | BrainTeaser (Sentence, overall) | BrainTeaser (Word, overall) | Average  |\n| ------- | ------- | ------- | ------- | ------- |\n| Llama3-8B (base) | 70.51 | 67.65    | 46.2 | 61.45 |\n| Llama3-8B (data)| 70.32 | 69.62    | 52.28| 64.07 |\n| Llama3-8B (data+reasoning)  | 72.18 | 69.89    | 58.07| 66.71 |\n| Llama3-70B (base)| 83.34 | 87.76    | 71.2 | 80.76 |\n| Llama3-70B (data) | 82.95 | 91.12    | 77.27| 83.78 |\n| Llama3-70B (data+reasoning) | 85.21 | 91.51    | 79.54| 85.42 |\n\n**W1-2. Why was testing only done on RiddleSense when many other benchmarks were reported earlier? Is this cherry-picking?**\n\nPlease refer to General Response **G1**.\n\n\n**W2. A player-judge evaluation framework based on WizardLM-2 is proposed as an alternative to human judges to evaluate the answers, but it shows much worse agreement on hard questions compared to inter-human agreements. However, harder questions are actually proposed as a contribution of this work. As the answers for such puzzles can be ``off by a hair but miss by a mile'', it might require human evaluation for the hard puzzles. \nIt is also a bit confusing why the human agreement on Medium questions is much lower than on hard and easy ones.**\n\nThe human-human agreement rates, sometimes as low as 87.5\\% as seen in Table 2, reflect the strictness of our agreement metric. For instance, if judgements from three humans result in two 'matched' and one 'unmatched,' the agreement is considered 1/3. If the judge model agree with the 'matched' responses, its agreement with humans is 2/3. Despite this rigorous evaluation, our method still achieves a high agreement rate of 88.24\\% on hard puzzles. It shows the diversity of acceptable answers and the challenging nature of our benchmark.\n\n\n**Q1. Similar datasets have existed before, as mentioned in the paper, such as RiddleSense and Oogiri. Here's an example from RiddleSense:\nMy life can be measured in hours. I serve by being devoured. Thin, I am quick; Fat, I am slow. Wind is my foe. What am I? (A) paper (B) candle (C) lamp (D) clock (E) worm.\nIn comparison, SPLAT has longer questions and answers. Also, the reference answers for SPLAT are open-ended, while the previous ones are all multiple-choice, which is claimed as a contribution. However, this also makes the evaluation more challenging. What makes you think open-ended is a much better option that makes it a contribution?**\n\nNot all problems are suited to multiple-choice formats, such as puzzles. While multiple-choice questions are easier to evaluate, they often simplify the problem. Besides, an open-ended format lessens the likelihood of inadvertently leading to the correct answer.\nHowever, the evaluation of open-ended answers is much more challenging.\nTo this end, we develop a multi-turn framework that leverages LLMs as judges to address open-ended evaluation tasks. The robustness of this framework represents one of the key contributions of our work.\n\n\n**Q2. Authors claim that their methods reduce reliance on stronger evaluation models, but it seems they are still using WizardLM-2 as a component (Judge Model) of their framework?**\n\nBy \"reduce reliance on stronger evaluation models\", we mean that the judge model in our framework does not necessarily need to be more powerful than the player model. However, it still needs to be sufficiently robust, achieving a high level of consistency with human judgements, as demonstrated in Tables 2 and 3. In traditional model-based evaluation methods (e.g., [1]), the evaluation model typically needs to be more capable than the model being evaluated since it directly grades the responses.\nThis requirement often constrains the ability to evaluate newer, more advanced models. Our approach, by contrast, allows for the use of robust but not necessarily superior models as evaluators, facilitating a broader assessment of SoTA models.\n\n[1] Judging llm-as-a-judge with mt-bench and chatbot arena. NeurIPS, 2023.\n\n\n**L1. The size for this benchmark is small compared with previous similar datasets that have existed before.**\n\nPlease refer to General Response **G2**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**General Response**

**G1. Results on more benchmarks than just RiddleSense.**

Besides the results on the RiddleSense (Section 5.3 in the submitted paper), we provide several results on another lateral thinking-focused benchmark, i.e., BrainTeaser [A], which features two main types of puzzles: Sentence (S.) puzzles and Word (W.) puzzles.
Following its official settings, we assess model performance using two accuracy metrics:
(1) Instance-based Accuracy: This metric individually evaluates each question—whether it is the original or a reconstructed version. We present instance-based accuracy for both the original puzzles and their semantic and contextual reconstructions (provided by the official dataset).
(2) Group-based Accuracy: Under this metric, each original puzzle and its variants are treated as a group. The model earns a score of 1 only if it correctly solves all three puzzles within a group. If it fails to do so, the score awarded is 0.

As in our submitted paper, we consider both open-source and closed-source LLMs, specifically Llama3 (in its Instruct versions) and GPT-4, with additional considerations for different model sizes, such as Llama3-8B and Llama3-70B.
As shown in the following tables, in the zero-shot setting, the accuracy of various LLMs consistently improves on the BrainTeaser benchmark (both sentence and word puzzles). The results further demonstrate that the data and framework of our benchmark effectively elicit lateral thinking capabilities in LLMs when applied to various lateral thinking tasks.

[A] Brainteaser: Lateral thinking puzzles for large language model. EMNLP, 2023.

BrainTeaser (Sentence)

| Model| Original | Semantic| Context | Ori \& Sem | Ori \& Sem \& Con | Overall |
| --- | --- | --- | --- | --- | --- | --- |
| Llama3-8B| 70.23 | 63.09 | 69.64 | 52.09 | 38.32 | 67.65 |
| Llama3-8B*| 72.18 | 65.47 | 72.02 | 58.92 | 45.23 | 69.89 |
| Llama3-70B | 89.34 | 87.57 | 86.39 | 84.61 | 75.73 | 87.76 |
| Llama3-70B*| 93.49 | 90.53 | 90.53 | 88.75 | 82.84 | 91.51 |
| GPT-4| 93.49 | 89.94 | 83.43 | 88.75 | 75.14 | 88.95 |
| GPT-4*| 95.26 | 91.71 | 88.69 | 91.71 | 82.24 | 91.88 |

BrainTeaser (Word) 

| Model| Original | Semantic| Context | Ori \& Sem | Ori \& Sem \& Con | Overall |
| --- | --- | --- | --- | --- | --- | --- |
| Llama3-8B| 47.72 | 44.69 | 46.21 | 31.06 | 17.42 | 46.20 |
| Llama3-8B*| 54.54 | 54.54 | 65.15 | 39.39 | 29.54 | 58.07 |
| Llama3-70B| 71.96 | 71.96 | 69.69 | 62.87 | 49.24 | 71.20 |
| Llama3-70B*| 81.06 | 81.81 | 75.75 | 74.24 | 59.09 | 79.54 |
| GPT-4| 71.21 | 65.91 | 56.06 | 59.09 | 41.66 | 64.39 |
| GPT-4*| 74.24 | 72.72 | 62.12 | 64.39 | 45.45 | 69.69 |


**G2. The size of the dataset.**

Our dataset contains 975 puzzles, where the number of puzzles is more than that in BrainTeaser (Sentence) and BrainTeaser (Word), which are 627 and 492, respectively. Besides, as each of our puzzles requires multi-turn conversations (always more than 5) to solve, it is comparable to the dataset like RiddleSense or Oogiri (T2T, Eng.) in terms of inference volume, where both contain about 5,000 puzzles. Moreover, as a benchmark, our priority is to ensure diversity and robustness to produce a stable evaluation framework. We believe the current quantity of puzzles is sufficient for this purpose. We also acknowledge the need for a larger dataset and plan to expand SPLAT in future versions.

Meta-review: This paper introduces a novel benchmark for measuring the strength of LLMs. In particular the new benchmark captures the "lateral thinking" which I think is similar to some form of common sense reasoning that is not very vertical specific like current benchmarks on QA, Math abilities etc. The evaluation proposes a new multi-turn player-judge framework instead of the traditional model-based evaluation. The proposed method lessens dependence on more robust evaluation models, enabling the assessment of state-of-the-art LLMs. The paper is well-written and presents a novel way of doing evaluation of LLMs. The authors also show that improving the model in a few shot setting to perform better on their benchmark also improves the performance of the model on another related benchmarks like RiddleSense displaying the generalizability of the model properties while hilclimbing on SPLAT.
Decision: Accept (poster)