Weak-eval-Strong: Evaluating and Eliciting Lateral Thinking of LLMs with Situation Puzzles

Authors: qi  chen
bowen  zhang
gang  wang
qi  wu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 14:41:26

Abstract: While advancements in NLP have significantly improved the performance of Large Language Models (LLMs) on tasks requiring vertical thinking, their lateral thinking capabilities remain under-explored and challenging to measure due to the complexity of assessing creative thought processes and the scarcity of relevant data. To address these challenges, we introduce SPLAT, a benchmark leveraging Situation Puzzles to evaluate and elicit LAteral Thinking of LLMs. This benchmark, containing 975 graded situation puzzles across three difficulty levels, employs a new multi-turn player-judge framework instead of the traditional model-based evaluation, which often necessitates a stronger evaluation model. This framework simulates an interactive game where the model (player) asks the evaluation model (judge) questions about an incomplete story to infer the full scenario. The judge answers based on a detailed reference scenario or evaluates if the player's predictions align with the reference one. This approach lessens dependence on more robust evaluation models, enabling the assessment of state-of-the-art LLMs. The experiments demonstrate that a robust evaluation model, such as WizardLM-2, closely matches human judgements in both intermediate question-answering and final scenario accuracy, achieving over 80% agreement--similar to the agreement levels among humans. Furthermore, applying data and reasoning processes from our benchmark to other lateral thinking-related benchmarks, e.g., RiddleSense and BrainTeaser, leads to performance enhancements. This suggests that our benchmark effectively evaluates and elicits the lateral thinking abilities of LLMs.

Main paper: 
 introduction: 
 Vertical and lateral thinking are two essential styles that play critical roles in human cognition and decision-making [41] . As noted in [19] , vertical thinking, characterised by its logical and structured nature, involves a systematic, step-by-step approach to problem-solving where each step logically follows the previous one. In contrast, lateral thinking is about creativity and viewing problems from multiple angles. It involves breaking away from traditional thought patterns to generate new ideas, and embracing a more playful and imaginative problem-solving approach.The evolution of natural language processing (NLP) models, particularly large language models (LLMs) [14, 39, 40] , has seen significant advancements in tasks that require vertical thinking, such as complex reasoning [5, 43] and commonsense inference [6, 37] . Despite these achievements, the development and evaluation of these models have primarily focused on vertical thinking capabilities [7, 42] , often neglecting lateral thinking, which is essential for creatively solving novel problems. Current benchmarks [34, 35] frequently dismiss creative thinking as irrelevant, focusing only on Easy Medium Hard Question/Story: A black cat is walking through the middle of a road where there is a broken streetlamp. A man is driving his vehicle with headlights that do not work. Despite all of this, he is able to avoid harming the cat. Reference Answer: The man avoided the cat because it was daylight.Question/Story: A hunter aimed his gun carefully and fired. Seconds later, he realized his mistake. Minutes later, he was dead. Reference Answer: He hunted in snow-capped mountains. The shot provoked an avalanche, which covered the man. He died of strangulation.Question/Story: Two men are beating each other up and both of them suddenly fall to the floor. Reference Answer: The daughter of one of the boxers had been kidnapped and she would be killed if her father won the fight. At the end of the fight, the boxer's opponent fainted and the other boxer -in order not to win the match -threw himself to the floor.Figure 1 : Examples from SPLAT benchmark, which are categorised into three ascending levels of difficulty, i.e., Easy, Medium, and Hard. The puzzles that are medium to hard in difficulty usually require guidance from the judge to be solved. If a puzzle can be solved without the judge's guidance, it typically resembles a regular puzzle requiring specific knowledge more than lateral thinking skills.problems solvable through conventional commonsense reasoning, indicating a gap in traditional LLMs' handling of unconventional thinking tasks, such as puzzles [19] .To fill this gap, benchmarks increasingly aim for tasks challenging even for humans, with outputs that are typically open-ended [8, 26] . However, evaluating such open-ended generation often relies on human or model-based assessments [46] . As tasks grow in complexity, such as longer output requirements, human evaluations become less feasible due to the extensive time and effort required to accurately judge long, complex scenarios created by LLMs. Moreover, model-based evaluations typically depend on more advanced models, thereby restricting their capacity to evaluate newer, state-of-the-art models. This reliance may also introduce biases, such as a preference for the first choice [46] . Consequently, there is a pressing need to rethink the evaluation framework of the benchmark for assessing emerging LLMs.In this paper, we seek to explore and elicit the lateral thinking ability of LLMs. However, accurately evaluating this capability poses significant challenges due to the complexity of measuring creative thinking [28, 18] and the difficulty of obtaining relevant data. The generation of novel ideas is inherently non-trivial, even for humans [12, 13] . Considering these challenges, we propose the exploration of lateral thinking in LLMs by situation puzzles as a primary research tool. A situation puzzle, often referred to as a lateral thinking puzzle, involves a scenario, usually presented as an unusual situation, and the goal is to figure out what is going on. Players ask yes-or-no questions to gather more information and solve the puzzle. These puzzles challenge players to think outside the conventional bounds of the scenario's information, using creativity and indirect reasoning to uncover the underlying explanation for the situation described. In this context, we introduce SPLAT, a benchmark that leverages Situation Puzzles for evaluating and eliciting LAteral Thinking of LLMs, which contains 975 high-quality situation puzzle pairs. As shown in Figure 1 , for better assessment, we categorise these puzzles into three difficulty levels-Easy, Medium, and Hard-each annotated by human evaluators. We obtain 54,463 annotations of difficulty on an average of ∼55 per puzzle.To assess the performance of LLMs in our benchmark with an open-ended setting, we initially aim to apply a model-based evaluation paradigm. However, this paradigm (e.g., [46] ) typically relies on using an evaluation model that is stronger than the models being tested. This creates a challenge in accurately evaluating models that may be superior to the evaluator itself. Therefore, there is a need for alternative methods that can effectively assess the advanced models without this limitation. To this end, inspired by the situation puzzle game, we propose a multi-turn player-judge framework. This framework emulates an interactive game where the model (player) queries the evaluation model (judge) about a given incomplete story, striving to collect information to unravel the intricate scenario it implies. Conversely, the evaluation model (judge) is tasked solely with responding to questions based on the detailed reference scenario or evaluating whether the scenario deduced by the player aligns semantically with the reference one1 . In this way, this framework reduces the reliance on more advanced evaluation models, and thereby can be used to evaluate the state-of-the-art LLMs.The experimental results indicate that a relatively strong evaluation model, e.g., WizardLM-2, can align closely with human judgements in both intermediate question-answering and final answer/scenario accuracy, achieving over 80% agreement-comparable to agreement level between humans. Moreover, applying the data and reasoning processes from our benchmark to other lateral thinking-relevant benchmarks like RiddleSense [21] shows a performance improvement (e.g., the accuracy of Llama3-70B [1] increase from 83.34% to 85.21%). It indicates that our benchmark can not only evaluate but also elicit the lateral thinking capabilities of LLMs. Finally, we evaluate several advanced LLMs such as GPT-4 and GPT-4 Turbo on our benchmark. The results highlight the ongoing challenges and the need for improved lateral thinking in LLMs.We summarise our contributions as follows:• We introduce a benchmark, called SPLAT, to evaluate the lateral thinking capacity of LLMs with situation puzzles. In this benchmark, we construct a new situation puzzles dataset, containing 975 unique puzzles that are divided into three difficulty levels. • We design a multi-turn player-judge evaluation framework to move beyond the existing modelbased evaluation paradigm in open-ended settings. This new framework reduces reliance on a stronger evaluation model -typically stronger than the models being assessed. • The dataset and framework are designed not only to evaluate but also to actively elicit the lateral thinking of LLMs. Experiments show that using data and reasoning processes from our framework leads to improved performance of LLMs, even when applied to other lateral thinking benchmarks. 
related works: 
 Lateral Thinking for LLMs. The realm of lateral thinking and computational creativity spans various tasks [49, 25, 47, 38, 17, 31] , such as pun detection [49] and humour recognition [25, 47] , which assess different cognitive capabilities. Notable contributions in this field include BrainTeaser [19] , which evaluates a broad spectrum of human intelligence attributes such as strategy and creativity. Another significant effort is RiddleSense [21] , a collection of riddles from public websites aimed at testing model abilities. In contrast, our work enhances this area by introducing scenario-level lateral thinking puzzles, i.e., situation puzzles, constructing a benchmark to evaluate and elicit the lateral thinking ability of Large Language Models (LLMs).Benchmarks for LLMs. Large language models (LLMs) [2, 3, 7, 36] are increasingly showing capabilities for various tasks ranging from writing and coding to engaging in multi-turn dialogues. However, assessing their extensive capabilities presents new challenges. Existing benchmarks, as categorised by [46] , predominantly fall into three types. Core-knowledge benchmarks [16, 45, 10, 33, 9, 11, 48] assess LLMs through zero-shot and few-shot tasks, requiring concise, specific answers to questions that are easily validated. Instruction-following benchmarks [23, 27] handle more openended questions and a broader range of tasks, which evaluate LLMs post-instructional fine-tuning. Lastly, conversational benchmarks [29, 15, 20] , designed for dialogue-based tasks, still lack the diversity and complexity needed to fully challenge advanced chatbots.For more challenging benchmarks, there is a trend toward including tasks that are even more difficult for humans, often requiring open-ended outputs [8, 26] . Evaluating such open-ended tasks typically depends on either human or model-based assessments. As these tasks increase in complexity, human evaluations become impractical due to the substantial time and effort needed to accurately assess the lengthy and intricate scenarios. Existing model-based evaluations [46] inherently rely on stronger models, which limits their ability to assess new state-of-the-art models. In this paper, we introduce a multi-turn player-judge framework to improve upon traditional model-based evaluations in openended settings. It reduces the need for a stronger evaluation model typically used in such assessments.3 Task and DatasetQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss the limitations of our work and propose directions for future research in the supplementary.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
task definition of situation puzzle game: 
 The situation puzzle game involves a player P and a judge J. Firstly, the judge provides an incomplete story S 0 to the player. Then, the player poses a set of questions Q = {q 1 , q 2 , . . . , q i , . . .} gradually for the judge to gather information about the unknown and detailed scenario/answer Ŝ behind the initial story S 0 . The judge responds with answers A = {a 1 , a 2 , . . . , a i , . . .}, where each answer a i ∈ {yes, no, irrelevant}. The objective is to deduce the hidden scenario/answer Ŝ that semantically aligns with all preceding yes-or-no question-answering pairs while fitting the given story S 0 . Data Collection. To collect the situation puzzle data, we follow the steps from [19] . Specifically, we first gather over one thousand situation puzzles and their answers from various public websites using web crawlers2 . After that, we merge the data from different sources and then eliminate the duplicates based on sentence similarity [30] . Then, we correct typographical errors using the Auto Correct library3 , followed by a human review process to ensure the puzzles maintain their intended meanings. This approach combines automated corrections with manual oversight to preserve the quality of the puzzles. Finally, we obtain a collection of 975 unique situation puzzles.Data Annotation and Difficulty. Inspired by the organisation of GAIA [26] , we categorise our collected situation puzzles into various distinct levels of difficulty, ranging from 1 (easiest) to 9 (hardest). This classification is done through a crowd-sourced annotation involving an average of approximately 55 annotators per puzzle to ensure the reliability of the difficulty rating. Then, these levels are further grouped into three broader categories: Easy (1∼3), Medium (4∼6), and Hard (7∼9).We compare our benchmark with the most relevant ones w.r.t. both vertical and lateral thinking. Specifically, for vertical thinking datasets, we consider GAIA [26] and LLM-as-a-Judge [46] , where the former is the most recent benchmark for general AI assistants, and the latter is the widely used benchmark to evaluate LLMs. The LLM-as-a-Judge [46] contains two different benchmark datasets, i.e., MT-bench and Chatbot Arena. The MT-bench is a set of challenging multi-turn open-ended questions designed to evaluate chat assistants. Chatbot Arena is a benchmark platform designed for evaluating LLMs through interactive, randomised battles in a crowd-sourced manner. It allows for the assessment of LLMs in real-time dialogues with human participants.Regarding lateral thinking benchmarks, we compare Oogiri [47] , RiddleSense [21] , and Brain-Teaser [19] . As the Oogiri benchmark [47] is multi-modal (text and image) and multi-lingual (English, Chinese, and Japanese), for a fair comparison, we consider Text-to-Text (T2T) English (Eng.) samples only. RiddleSense [21] is a benchmark dataset designed for evaluating complex commonsense reasoning within the framework of riddle-style questions. It presents a unique multiplechoice question-answering task that requires an understanding of figurative language, counterfactual reasoning, and other advanced natural language understanding skills. The BrainTeaser dataset [19] is a versatile and challenging collection for evaluating models on their ability to engage in lateral thinking and solve intricate puzzles. It features two main types of puzzles: Sentence (S.) puzzles and Word (W.) puzzles. The sentence puzzles require solvers to interpret or decipher complex sentences, often involving multiple meanings or requiring the solver to think beyond straightforward interpretation. The word puzzles focus on wordplay or semantic puzzles, where the challenge lies in understanding and manipulating words based on how they are presented or their semantic relationships.In Table 1 , we compare our SPLAT benchmark with existing ones across various dimensions such as the type of target task, the fashion of answers, the evaluation paradigm, and the statistics of samples. Here's a detailed discussion of how our benchmark, focused on situation puzzles, stands out:Task Uniqueness and Open-ended Responses. Unlike traditional QA and dialogue-based tasks in [26, 46] , our task requires models to infer the scenario from a given incomplete story, necessitating deep engagement with both context and creativity. The process of accurately inferring the scenario involves multiple steps and is challenging to brute force due to their diversity [26] , as well as difficult to evaluate. Our benchmark supports the verification of reasoning traces and predicted answers. Moreover, unlike the multiple-choice formats seen in tasks like Oogiri [47] and RiddleSense [21] , our open-ended format lessens the likelihood of inadvertently leading to the correct answer.Reference-guided Semantic-aware Evaluation. Unlike the exact match or pairwise battle approaches used in vertical thinking benchmarks [46, 26] , our SPLAT employs a 'Semantic Judge' evaluation. Analogous to Natural Language Inference (NLI) [24] , we assess the semantic alignment between the predicted output and a human-generated reference answer using an LLM-based judge. This approach ensures accurate evaluation, even when the formats of the predicted outputs significantly differ from those of the reference answers.Challenging Question with Judgement-friendly Answer. Our dataset features 975 samples with relatively high averages in question and answer lengths (79.74 and 45.53 tokens respectively). This indicates a depth of content and complexity in our scenarios, surpassing most lateral thinking datasets such as RiddleSense [21] or BrainTeaser [19] , where the questions or answers are typically much shorter and less detailed. Moreover, by employing a model-based and reference-guided evaluation paradigm, we reduce the dependence on a more robust evaluation model, different from [46] , which typically needs a stronger model to assess. Our approach only necessitates that the judge model determines if the predicted answer is semantically aligned with the reference answer. Once we verify that the used judge model can reliably perform this task, the relative strength of the models being evaluated compared to the judge model becomes irrelevant.Situation puzzles on the proposed SPLAT can be classified into three ascending levels of difficulty, based on three criteria. 1) Time to Solve: How long does it typically take for a player to arrive at a correct answer? Shorter times indicate easier puzzles, while longer time suggests higher complexity.2) Length of Reference Answer/Scenario: Simpler puzzles may have shorter, more straightforward answers, whereas more complex ones might require lengthy explanations or involve intricate scenarios.3) Subjective Complexity of the Scenario: This involves a subjective evaluation of how mentally challenging the scenario is for the human players. Thus, the puzzle in each level can be defined as:Easy Puzzles (1∼3): These puzzles typically require minimal time to solve, with reference answers that are short and direct, involving straightforward solutions. These scenarios are simple and can usually be resolved quickly, often within a few minutes, and do not necessitate extensive reasoning.Medium Puzzles (4∼6): These are moderately complex puzzles that demand more time and deeper analysis. The reference answers are longer, providing more details that players must consider. Solving these puzzles generally takes a moderate amount of time, with players needing to make several logical deductions and possibly sift through some distractors.Hard Puzzles (7∼9): These are the most challenging puzzles, designed for advanced players. They require a significant amount of time to unravel, with lengthy and complex scenarios that may include extensive details or embedded subtleties. Players must engage in a high degree of reasoning, with an array of logical deductions and the potential for numerous misleading clues. These puzzles often demand an arbitrary number of steps, deep thinking, and the ability to focus over prolonged periods.Figure 2 (a) suggests a correlation between difficulty level and the average number of tokens, particularly for answers, which tend to increase in length as the difficulty level rises. This could indicate that more complex puzzles require longer, more detailed answers. In Figure 2 (b), the clusters by difficulty level also show that harder puzzles tend to have both longer answers and higher time costs, aligning with intuitive expectations about puzzle solving.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: We offer a comprehensive explanation of our dataset in the paper and supplementary. The code and dataset will be made available upon acceptance. Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
multi-turn player-judge framework: 
 We design such a framework (Figure 3 ) for three main reasons. Firstly, the complexity of scenarios in situation puzzles makes it difficult for a model to accurately predict the final answer in just one attempt (i.e., in a single turn). Secondly, we aim to ensure that the reasoning process remains interpretable and assessable, even in the absence of pre-defined reference reasoning steps. This framework allows for a more thorough evaluation of the model's ability to solve complex puzzles step by step. Thirdly, we hope that the framework or the intermediate reasoning processes derived from this framework can help elicit the lateral thinking ability of LLMs even on other lateral thinking-relevant datasets.Characters. The multi-turn player-judge framework of a situation puzzle game has two primary characters: a player P and a judge J. The player P is a participant (e.g., LLMs we need to evaluate), who attempts to solve the puzzle by asking questions. The judge J is the entity, either a human or an AI model, that knows the solution to the puzzle and provides responses to the player's questions. Due to the page limit, we put the detailed character guidelines in the supplementary.Interaction Dynamics. The interaction between players and the judge is driven by a series of questions Q = {q 1 , q 2 , . . . , q i , . . .} and answers A = {a 1 , a 2 , . . . , a i , . . .}. Specifically, at the start of the game, the player is presented with an incomplete story, denoted as S 0 . The player must read this story and initially decide if the hidden scenario or answer Ŝ can be inferred from the present information (i.e., solely S 0 ). If the scenario remains unclear, the player will then pose a yes-or-no question q 1 to obtain more details about the puzzle.The judge provides a response a 1 ∈ {yes, no, irrelevant} based on the incomplete story S 0 and the hidden scenario Ŝ4 , i.e., a 1 ← J(q 1 , S 0 , Ŝ). Upon receiving the response a 1 , the player will either attempt to deduce the final answer or, if still uncertain, will ask a subsequent question (i.e., q 2 ). This process continues iteratively, with the player refining the understanding based on the judge's answers.Information Sets and Game State. At any turn t in the game, the information set I t represents all the questions asked and answers received up to that time:I t = {(q 1 ,a 1 ), (q 2 , a 2 ), . . . , (q t , a t )}.(1)The game state Γ t includes the incomplete story S 0 , the information set I t and any inferred knowledge K t about the situation puzzle: Γ t = (S 0 , I t , K t ),You are a judge, and your role is to challenge the user with a given puzzle in the form of a short, logically self-consistent story. The story are complete but may intentionally omit some details that the user needs to deduce or inquire about.Here are the guidelines for your role: 1. 2. 3. 4. Now, the system will give your first puzzle. The game starts when the user indicates they are ready. Enjoy the challenge of testing and expanding the user's deductive reasoning! Puzzle Title: Sweet Dreams Story: Matthew keeps reading a bedtime story to his son despite the blackout. Why? Reference Answer: Matthew was blind, and he usually read bedtime stories to his son from a braille book. That night there was a blackout, but this did not stop him from finishing the story. Here, K t is the knowledge inferred by the player based on the story S 0 and information set I t . The player determines the next question based on the current game state byEQUATIONObjective. The objective of the player is to deduce the scenario/answer S that satisfies all given constraints based on the responses from the judge. Mathematically, the goal is to find the hypothesised solution Ŝ that maximises the posterior probability given the game state Γ t :EQUATION5 Experiments 
agreement evaluation: 
 Inspired by [46] , to assess the alignment of judge's responses with human preferences, we consider an agreement evaluation among them w.r.t. judgements for both final predicted answers and the intermediate reasoning processes. We randomly select 20 puzzles from each difficulty level, which is 60 puzzles in total. For each puzzle, we ask three different individuals to assess the final answer agreement and the reasoning process agreement, respectively. For a fair comparison, we use the questions generated by taking Llama3 (70B-Instruct) [1] as the player and WizardLM-2 (8x22B) [44] as the judge. Besides humans, we also consider Llama3 (70B-Instruct) as another judge model to test the agreement among humans and different judge models. More details are in the supplementary.Final Answer Agreement. To quantitatively evaluate the semantic alignment between the final predicted answers and reference answers, we first ask humans to assess the semantic alignment between each predicted answer and the corresponding reference answer/scenario. Subsequently, we measure the agreement performance between the human judgement and the evaluations from the judge model. We totally obtain 156 judgements from humans. In Table 2 , the high level of agreement between the LLM-based judges and humans (e.g., WizardLM-2 achieves 100%, 87.50%, 88.24% agreements with humans in three difficulties, respectively) indicates that LLMs can well align with human standards of reasoning and judgement within this specific context of lateral thinking puzzles. Reasoning Process Agreement. To assess the agreement on reasoning processes, particularly for intermediate questions where no predefined reference answers exist, we use a method where humans are asked to respond to these questions based on the context provided by the given story and the final reference answer. Their responses are categorised as {yes, no, irrelevant}. We then evaluate the alignment of these human responses with those from the judge model. We obtain 2,271 judgements from humans. Table 3 shows that LLMs, e.g., WizardLM-2 and Llama3-70B, demonstrate good alignment with human reasoning in simpler puzzles. As the difficulty level rises, WizardLM-2 keeps a relatively high agreement with human judgements, while the alignment between Llama3-70B and humans decreases dramatically. The results verify that our choice of WizardLM-2 as the judge model aligns well with human judgement across all difficulty levels, maintaining over 80% agreement.Evaluation Metrics. To assess the performance of LLMs using the proposed SPLAT benchmark, we employ two distinct metrics: Accuracy (Acc, %), which measures the correctness of the scenarios deduced by LLMs against the reference scenarios/answers, and Average Round (Rnd), which quantifies the average number of interaction rounds required for LLMs to reach the reference scenarios. Note that if the LLMs fail to deduce the correct scenario before a pre-defined max round, the round count is the same as the max one, and the accuracy for that particular puzzle is recorded as 0. Conversely, if the correct scenario is successfully predicted, the accuracy is set to 1, and the number of rounds taken to reach the correct answer is recorded. Finally, we define an OverAll (O/A) evaluation metric asEQUATIONwhere I(•) is an indicator that returns 1 if the scenario deduced by LLMs matches the reference one semantically, and 0 otherwise. Given that I(sample i ) takes values in {0, 1} and Rnd i ranges from 1 to a maximum, the O/A metric spans from 0 to 100, with higher values indicating better performance.Besides, in the open-ended setting, the player model may deduce other reasonable scenarios that satisfy both the given incomplete story and the responses from the judge model. In this situation, it is hard to judge whether these deduced scenarios are correct or not due to the diversity. Inspired by the evaluation of image captioning on MSCOCO [22] , which uses multiple reference answers for each image and computes metrics based on the closest reference answer, we consider a similar approach. However, generating multiple reference scenarios/answers for situation puzzles is challenging. As an alternative, we repeatedly run each situation puzzle multiple times (denoted as R), and then choose one (maybe r-th) that is closest to the reference scenario from these iterations as the final result. This approach not only maintains the diversity of potential predictions but also reduces the impact of hallucination from LLMs by subjecting each puzzle to several evaluations.Results. We compare the performance of various LLMs on our SPLAT benchmark, including Lama3 (8B-Instruct & 70B-Instruct) [1], Qwen1.5 (32B-Chat & 110B-Chat) [4] , WizardLM-2 (8x22B) [44] , GPT-4 and GPT-4 Turbo [2] . Here, we take these LLMs as players and use WizardLM-2 (8x22B) as a judge in all these performance comparison experiments. We set R = 1 and the max round is 15. In Table 4 , GPT-4 and its Turbo variant, along with the Llama3 (70B-Instruct) and WizardLM-2 (8x22B), show robust capabilities, achieving relatively higher accuracy. Conversely, models like Qwcn1.5 (32B-Chat) and Qwcn1.5 (110B-Chat) show lower accuracy, which might reflect challenges in adapting their chat-based configurations to the lateral thinking required on the SPLAT benchmark.Performance Bias. In the SPLAT benchmark, concerns about bias when a model like WizardLM-2 (8x22B) evaluates its own performance are mitigated through several key design elements. First, the SPLAT benchmark is designed with a specific framework that focuses on how well each LLM, serving as a player, can reason through situation puzzles to arrive at correct scenarios/answers. The judge's role is tasked solely with responding to questions based on the detailed reference scenario or evaluating whether the scenario deduced by the player aligns semantically with the reference one, which is relatively simple and objective. Besides, the evaluation metrics used (Accuracy and Rounds needed) are also objective, reducing the potential for subjective bias or preference.From Table 4 , WizardLM-2 (8x22B) performs well but not exceptionally in every category. It has competitive but not maximal accuracy rates, and its average round count is not the lowest. It suggests that the WizardLM-2 (8x22B) we use has strong enough capabilities to serve as a judge model. Besides, under our benchmark, the used judge model does not have an obvious preference for itself. Models and Benchmarks. As previously discussed, our benchmark serves to evaluate the lateral thinking abilities of LLMs and to elicit these capabilities actively. To verify this, we seek to enhance LLMs with the data and reasoning processes from our benchmark, and then evaluate them on other lateral thinking benchmarks (i.e., RiddleSense [21] and BrainTeaser [19] ) to observe performance enhancements across various LLMs. Our analysis includes both open-source and closedsource LLMs, specifically Llama3 (Instruct version) and GPT-4, with additional considerations for different model sizes, such as Llama3-8B and Llama3-70B.EQUATIONImplementations. In the zero-shot setting, LLMs are challenged with multiple-choice riddles from these benchmarks, where they must choose answers based solely on the riddles presented. The accuracy of their responses is calculated by comparing them to the correct answers. To effectively integrate our benchmark's data and reasoning processes from the proposed player-judge framework, we treat the question-answer pairs generated during the benchmark as auxiliary prompts. These are seamlessly incorporated into the LLMs' reasoning processes to enhance their thinking ability.To implement this, we set up the game where one of the LLMs (e.g., Llama3-70B) acts as the player with WizardLM-2 continuing as the judge. We randomly select m situation puzzles (we set m = 1 in our experiment) from our benchmark dataset and engage both the player and the judge in these selected puzzles using our multi-turn player-judge framework. From each puzzle, we gather the first 5 question-answer pairs to form an auxiliary reasoning prompt set U, comprising a total of 5 × m pairs. This auxiliary prompt set U is then integrated into the original prompts P used by various LLMs.It is important to note that the same auxiliary prompts are used across different LLMs during their evaluations on these two benchmarks. This keeps consistency in the supplementary data provided to each model, ensuring a fair comparison of their performance.Results on RiddleSense and BrainTeaser Benchmarks. On the RiddleSense (dev set), Figure 4 shows that with our reasoning prompts, the accuracy of various LLMs consistently improves (Llama3-8B: 70.51% → 72.18%, Llama3-70B: 83.34% → 85.21%, GPT-4: 88.83% → 89.23%) on the RiddleSense benchmark. The results demonstrate that the data and framework in our benchmark can be used to elicit the lateral thinking ability of LLMs when handling other lateral thinking tasks. BrainTeaser features two main types of puzzles: sentence puzzles and word puzzles. Following its official settings, we assess model performance using two accuracy metrics: 1) Instance-based Accuracy: This metric individually evaluates each question-whether it is the original or a reconstructed version. We present instancebased accuracy for both the original puzzles and their semantic and contextual reconstructions. 2) Group-based Accuracy: Under this metric, each original puzzle and its variants are treated as a group. The model earns a score of 1 only if it correctly solves all three puzzles within a group. If it fails to do so, the score awarded is 0. From Table 5 , in the zero-shot setting, the accuracy of various LLMs consistently improves on BrainTeaser. These demonstrate that the data and framework of our benchmark effectively elicit lateral thinking capabilities of LLMs when applied to various lateral thinking tasks.Ablation Study. We further explore the influence of our data and reasoning processes. In Table 6 , incorporating our data into base LLMs leads to an increase in average accuracy (Llama3-8B: 61.45 → 64.07; Llama3-70B: 80.76 → 83.78). This improvement is further enhanced to 66.71 and 85.42, respectively, when we integrate the reasoning processes. These results further demonstrate that both our data and reasoning processes can effectively enhance the lateral thinking capabilities of LLMs. 
conclusion: 
 In this paper, we address the gap in the lateral thinking capabilities of LLMs despite significant advances in vertical thinking skills. We propose SPLAT, a benchmark using Situation Puzzles to evaluate and elicit LLMs' LAteral Thinking by a multi-turn player-judge framework, which departs from traditional model-based evaluations. This new framework evaluates the LLMs by interacting with a judge model to solve puzzles, reducing the reliance on more robust evaluation models. Our experiments with SPLAT, including a robust evaluation model like WizardLM-2, demonstrate over 80% agreement with human judgements and show improvements when applied to another lateral thinking benchmark-RiddleSense. This highlights SPLAT's effectiveness in evaluating and eliciting lateral thinking in LLMs, suggesting its potential broader application in AI research and development.A Appendix / supplemental material A.1 Limitations and Social Impacts Limitations The nature of lateral thinking tasks often encourages non-linear and novel thought processes, which could lead to unpredictable or unexpected outputs from LLMs. While creativity is desired, these outputs might sometimes include harmful, misleading, or dangerously incorrect information, especially if not properly supervised. To alleviate this, we will engage in robust testing across diverse scenarios to understand the boundaries and limitations of the model's creative outputs.Testing should aim to uncover any potential for generating harmful or inappropriate content.Broader Impacts Our work's advancement in lateral thinking capabilities of Large Language Models (LLMs) could significantly benefit society by enhancing problem-solving abilities across various domains, such as education, engineering, and the arts. By enabling AI to approach problems with a level of creativity akin to humans, these models could provide innovative solutions that defy conventional thinking patterns.However, the enhancement of LLMs in lateral thinking also brings potential risks, such as the misuse of technology for malicious purposes like fraud or misinformation, job displacement in creative and problem-solving professions, and increased societal dependence on technology for innovative thinking. To counter these risks, it is essential to establish robust ethical guidelines and transparent AI usage policies. We should seek to promote a culture of ethical AI use combined with ongoing education about AI's societal impacts. This will help ensure that the benefits of advanced AI are realised while minimising its potential harms. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We thoroughly discuss both the positive and negative potential societal impacts of our work in the supplementary.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes] Justification: We check the scraped data by humans one by one. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Judge: The guidelines for a judge include:• Scenario Understanding: Read the given short story and the corresponding answer. Make sure you can understand the both story and answer, and their logical relationships.• Response: Respond to the user's questions with only "yes", "no", or "irrelevant". Your answers should help guide the user towards understanding the full context of the puzzle.• Evaluation: Evaluate the user's final answer carefully. If the user's answer is correct, confirm their success and say "Congratulations". If incorrect, allow them to continue asking questions or provide subtle hints to steer them in the right direction.• Question Handling: If there are multiple questions, respond to the first question and ignore others.Player: The guidelines for a player include:• Judge will give you a puzzle in the form of a short story that is logically self-consistent. The story will contain all the information needed to solve the puzzle, but some details may be omitted.• Your task is to ask questions to gather more information and help you find the solution to the puzzle. You can ask any yes/no questions, and the judge will answer truthfully with yes/no/irrelevant.• During the questioning process, please carefully analyse the known information and clarify the logical relationships between the clues. Don't overlook any seemingly trivial details.• When you believe you have collected enough information, please provide your answer and explain how you reasoned out the conclusion based on the known clues.• If your answer is correct, the game ends; if it's incorrect, you can continue asking questions until you find the correct answer.• You will never ask questions with answers other than yes/no/irrelevant. Remember, the key to the puzzle is to collect clues through reasonable questions and use logical reasoning to draw conclusions.Inspired by [46] , to assess the alignment of judge's responses with human preferences, we consider an agreement evaluation among them w.r.t. judgements for both final predicted answers and the intermediate reasoning processes. We randomly select 20 puzzles from each difficulty level, which is 60 puzzles in total. For each puzzle, we ask 3 different individuals to assess the final answer agreement and the reasoning process agreement. We finally obtain 156 judgements from humans for the final answer and 2,271 judgements for the reasoning process.Final Answer Agreement To quantitatively evaluate the semantic alignment between the final predicted answers and reference answers, we first ask humans to assess the semantic similarity between each predicted answer and the corresponding reference answer. Subsequently, we measure the agreement performance between the human judgement and the evaluations from the judge model. For example, if three humans vote 'matched', 'matched', and 'unmatched' for a puzzle, respectively, the agreement among them, noted as 'human-human', is only 1 3 , as there are three pairs (matched, matched), (matched, unmatched), and (matched, unmatched). If the judge model vote 'matched', the agreement between humans and the judge model is 2 3 . This process can be mathematically structured as follows:Define an alignment metric M that quantifies the agreement between the judge's judgement and human assessments. The metric can be expressed asEQUATIONwhere δ is an indication function that equals 1 if the judgement j i ∈ {matched, unmatched} from the judge model matches the human judgement h i , and 0 otherwise. Here, m is the total number of comparisons made between the judgements from the judge model and humans, respectively. A higher value of M indicates a stronger alignment of the decisions of the judge model with human preferences, suggesting that the judgements rendered by the automated system closely mirror those of human evaluators in terms of understanding and evaluating the semantic content of the answers.Reasoning Process Agreement To assess the agreement on the reasoning process, particularly for intermediate questions where no predefined reference answers exist, we employ a methodology where humans are asked to respond to these questions based on the context provided by the given story and the final reference answer. Their responses are categorised as {yes, no, irrelevant}. We then evaluate the alignment of these human responses with those from the judge model. This process can be mathematically described as follows:Define a reasoning agreement metric R to quantify the alignment between the human responses and the judgements provided by the judge model for intermediate questions:EQUATIONwhere δ is an indication function that equals 1 if the human response u i to the intermediate question matches the judge model's response a i , and 0 otherwise. Here, r represents the total number of intermediate questions evaluated. A higher value of R indicates a strong alignment between the human responses and the judge model's decisions regarding the intermediate reasoning steps.Instruction for Human Figures 5 and 6 provide the instruction shown for humans and ask them to write the judgement for the final answer and for the reasoning process, respectively.We present several examples illustrating the process of situation puzzle games. In these examples, Llama3 (70B-Instruct) [1] serves as the player andWizardLM-2 (8x22B) [44] functions as the judge.Puzzle 1: The title of the puzzle is Sweet Dreams. Matthew keeps reading a bedtime story to his son despite the blackout. Why? Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Note that the ground-truth reference answer/scenario only can be seen by the judge.Upon publication, we will release our data, as all the puzzles are sourced from publicly accessible websites.https://github.com/phatpiglet/autocorrectNote that the judge knows the detailed scenario, i.e., the reference answer. 


Decision: This paper introduces a novel benchmark for measuring the strength of LLMs. In particular the new benchmark captures the "lateral thinking" which I think is similar to some form of common sense reasoning that is not very vertical specific like current benchmarks on QA, Math abilities etc. The evaluation proposes a new multi-turn player-judge framework instead of the traditional model-based evaluation. The proposed method lessens dependence on more robust evaluation models, enabling the assessment of state-of-the-art LLMs. The paper is well-written and presents a novel way of doing evaluation of LLMs. The authors also show that improving the model in a few shot setting to perform better on their benchmark also improves the performance of the model on another related benchmarks like RiddleSense displaying the generalizability of the model properties while hilclimbing on SPLAT.

Reviews: review: paper_summary: paper_summary: This paper introduces SPLAT, a novel benchmark for evaluating and eliciting lateral thinking abilities in Large Language Models (LLMs) using situation puzzles. The key contributions include: A new dataset of 975 graded situation puzzles across three difficulty levels.
A multi-turn player-judge evaluation framework that reduces reliance on stronger evaluation models. Demonstration that the benchmark can both evaluate and elicit lateral thinking in LLMs. Experimental results showing that a robust evaluation model (WizardLM-2) closely matches human judgments. Evidence that applying SPLAT's data and reasoning processes to other lateral thinking benchmarks leads to performance improvements in LLMs.
strength_weakness: strength_weakness: 1. While the data collection process is described, there's minimal discussion of potential biases in the dataset, such as cultural specificity of the puzzles or biases introduced during the human annotation process.
2. The paper would benefit from ablation studies to isolate the impact of different components of the SPLAT framework, such as the difficulty categorization or the multi-turn questioning process.
3. The paper lacks a discussion of potential ethical implications or misuse of improved lateral thinking capabilities in LLMs.
questions: questions: 1. How does the performance of LLMs on SPLAT correlate with their performance on other, more traditional NLP tasks? Is there evidence that lateral thinking ability is distinct from other language model capabilities?
2. Have you considered ways to automatically generate new situation puzzles to expand the dataset and improve scalability?
3. How sensitive is the performance of LLMs on SPLAT to the specific prompts or instructions given? Could you elaborate on the prompting strategy used?
limitations: limitations: None.
paper_summary: paper_summary: This work focuses on creating a benchmark, as well as a modeling framework for evaluating lateral thinking of LLMs. The benchmark, called SPLAT, consists of 975 situation puzzles. The framework consists of a “judge” and a “player” (the LLM to be evaluated). The judge poses an open ended puzzle that is further clarified by the player till a final answer is reached. The judge evaluates the final answer. The authors posit that this framework eases the challenge faced with typical auto-evaluation models since the judge does not need to be better than the LLM that is being evaluated.

The authors evaluate the quality of the judge (via human agreement), and finally evaluate LLMs using a promising judge.
strength_weakness: strength_weakness: Overall, the idea and motivation behind the work is solid. I also think that the benchmark is good quality. However, my concern is that this work is half-baked. The experiments aren’t conclusive. More work should go into making the experiments thorough and useful to the scientific community.


* Although the motivation and overall idea is strong, I am concerned about the experimental setup. The agreement rates for WizardLM-2 in Table 2 and 3 don’t appear to be high enough (are around ~80%) and 3 individuals seems to be fairly less. How does this fare with human agreement rates in literature? Also, is there any reason we haven’t used the strongest model (GPT-4 from the paper) as the judge?

* I also am concerned about Table 4. The authors should consider using strong base models for evaluation (that are closer to GPT-4 on public benchmarks)

* It would be good to introduce modeling methods to improve upon the metrics that are obtained for models on SPLAT. This will give more insight into how this benchmark is valuable to the community.

* nit: I see mentions of “using data and reasoning processes from our framework” consistently throughout the paper. This should be framed better and be more precise.
questions: questions: See weaknesses mentioned above.
limitations: limitations: NA
paper_summary: paper_summary: This paper focuses on lateral thinking, which is about creativity and viewing problems from multiple angles. To sovle the challenge that the complexity of assessing creative thought processes and the scarcity of relevant data, this paper introduces SPLAT, a benchmark leveraging Situation Puzzles to evaluate and elicit LAteral Thinking of LLMs. By employing a new multi-turn player-judge framework instead of the traditional model-based evaluation, the proposed method lessens dependence on more robust evaluation models, enabling the assessment of state-of-the-art LLMs. Moreover, through applying data and reasoning processes from our benchmark to another lateral thinking-related benchmark, like RiddleSense, leads to performance enhancements.
strength_weakness: strength_weakness: 1. The size of the dataset is samll.
2. The reference answer is limited to 1, while the questions are one-to-many questions.
3. The in-depth analysis is not enough
questions: questions: 1. Why not provide more than one answer for each question (I do not think it is challenging, there are multiple answers to a question on the internet such as Quora)? Since one such question could have multiple answers, more reference answer might also be helpful for evaluation.
2. A overall evaluation metric is needed, some model might have a high acc with a high average round, you should define a overall metric to combine these two metrics for a clear comparison of existing LLMs.
3. It is better to adopt more benchmarks for validating the effectiveness of the proposed dataset (more than just RiddleSense).
limitations: limitations: None
paper_summary: paper_summary: The paper constructs the benchmark SPLAT to evaluate and elicit lateral thinking abilities in LLMs. It includes questions similar to brain teasers, which describe scenarios and ask non-obvious questions for guessing. 

Example: 

__Question/Story__: A hunter aimed his gun carefully and fired. Seconds later, he realized his mistake. Minutes later, he was dead. 
__Reference Answer__: He hunted in snow-capped mountains. The shot provoked an avalanche, which covered the man. He died of strangulation.

They also propose a multi-turn player-judge evaluation framework that reduces reliance on stronger evaluation models.
strength_weakness: strength_weakness: 1. "The dataset and framework are designed not only to evaluate but also to actively elicit lateral thinking in LLMs. Experiments show that using data and reasoning processes from our framework leads to improved performance of LLMs, even when applied to other lateral thinking benchmarks." This part really needs ablation studies. It's unclear which parts are effective. For example, are the reasoning processes, the entire dataset, or similar questions in previous data responsible for the improvement? Did this data improve the model's ability to think about these types of questions, or is the improvement possibly due to some tricks? Moreover, why was testing only done on RiddleSense when many other benchmarks were reported earlier? Is this cherry-picking?

2. A player-judge evaluation framework based on WizardLM-2 is proposed as an alternative to human judges to evaluate the answers, but it shows much worse agreement on hard questions compared to inter-human agreements. However, harder questions are actually proposed as a contribution of this work. As the answers for such puzzles can be "off by a hair but miss by a mile," it might require human evaluation for the hard puzzles. It is also a bit confusing why the human agreement on Medium questions is much lower than on hard and easy ones.
questions: questions: 1. Similar datasets have existed before, as mentioned in the paper, such as RiddleSense and Oogiri. Here's an example from RiddleSense: 

My life can be measured in hours. I serve by being devoured. Thin, I am quick; Fat, I am slow. Wind is my foe. What am I? (A) paper (B) candle (C) lamp (D) clock (E) worm

In comparison, SPLAT has longer questions and answers. Also, the reference answers for SPLAT are open-ended, while previous ones are all multiple-choice, which is claimed as a contribution. However, this also makes the evaluation more challenging. What makes you think open-ended is a much better option that makes it a contribution?

2. Author claim that their methods reduces reliance on stronger evaluation models, but it seems they are still using WizardLM-2 as a component (Judge Model) of their framework?
limitations: limitations: The size for this benckmark is small compared with previous similar datasets that have existed before, which include many more puzzles compared to the proposed SPLAT.

Comments: title: Thanks
Thanks
Thanks
Replying to Rebuttal of Authors
comment: Thank you for the ethical review. We source our situation puzzles from publicly accessible websites, ensuring compliance with ethical standards. Specifically, we prioritise puzzles explicitly marked as public domain or not under copyright protection. For copyrighted puzzles, we employ them under the fair-use doctrine, permitting limited use without needing permission from copyright holders. For instance, the website [1] permits viewing, downloading, and distributing a single copy of the material for personal, non-commercial purposes.

[1] https://www.braingle.com/terms.php
Thank you for your valuable suggestions and for increasing the score. We will explore incorporating multi-answer formats in future updates to better evaluate lateral thinking in LLMs.
Thank you again for the constructive comments. We will add the clarifications in the response to our revised paper.
Thank you very much for your encouragement and valuable comments on our work. We will include the discussions and ablation study in our revised paper and make them clearer.
Thanks, I think W1 and W2 make sense. I have increased the score to reflect this. I do think W3 would be important to improve the potential impact of this work.
Thanks the authors for their response. After reading the response, I think my current score is appropriate.
Thanks for your detailed response. It has addressed some of my concerns, I will raise the score to reflect this. 
However, I am still concerned that the number of answers to each question is limited to 1 since your paper is about "Lateral Thinking". Although "How to build a situation puzzle?" is not proper to obtain multiple answers, there still are many questions that have answers from multiple angles, such as "how to prove 1+1=2".