BRAINTEASER: Lateral Thinking Puzzles for Large Language Models
Authors: Yifan Jiang
Filip Ilievski
Kaixin Ma
Zhivar Sourati
Venue: EMNLP 2023 Main (2023)
Keywords: commonsense reasoning
adversarial robustness
computational creativity
TL;DR: BRAINTEASER is a novel Question Answering task designed to test systems' out-of-box reasoning and reverse thinking abilities by challenging their default rationale using brain teasers from various categories.

Abstract: The success of language models has inspired the NLP community to attend to tasks that require implicit and complex reasoning, relying on human-like commonsense mechanisms. While such vertical thinking tasks have been relatively popular, lateral thinking puzzles have received little attention. To bridge this gap, we devise BrainTeaser: a multiple-choice Question Answering task designed to test the model’s ability to exhibit lateral thinking and defy default commonsense associations. We design a three-step procedure for creating the first lateral thinking benchmark, consisting of data collection, distractor generation, and generation of adversarial examples, leading to 1,100 puzzles with high-quality annotations. To assess the consistency of lateral reasoning by models, we enrich BrainTeaser based on a semantic and contextual reconstruction of its questions. Our experiments with state-of-the-art instruction- and commonsense language models reveal a significant gap between human and model performance, which is further widened when consistency across adversarial formats is considered. We make all of our code and data available to stimulate work on developing and evaluating lateral thinking models.

BRAINTEASER: Lateral Thinking Puzzles for Large Language Models
Yifan Jiang', Filip Ilievski' Kaixin Ma* Zhivar Sourati'
Information Sciences Institute, Viterbi School of Engineering, University of Southern Califomia *Department of Computer Science, Faculty of Science, Vrije Universiteit Amsterdam TTencent AI Lab, Bellevue, WA {yifjia, i edu, f. f.ilievski@vu.n kaixinma@global. tencent. com
Abstract
Vertical Thinking
PIQA How you flóodia room?
The success language models has inspired the NLP community to attend to tasks that require implicit and complex reasoning, rely- ing on human-like commonsense mechanisms. While such vertical thinking tasks have been relatively popular, lateral thinking puzzles have received little attention. To bridge this gap. we devise BRAINTEASER,
cover with water
(A) Fill it with objects.
(B) Fill with water.
RiddleSense
I have five fingers, but am noo alive, What am /?
five separate parts + item like a hand
multiple-choice Question Answering task designed to test the model's ability exhibit lateral thinking and defy default commonsense associations. We design a shre procedure for creating the first lateral thinking benchmark, consisting of data collection, distractor generation, and gen- eration of reconstruction examples, leading to 1,100 puzzles with high-quality annotations. To assess the consistency of lateral reasoning by models, we enrich BRAINTBASH based on a semantic and contextual reconstruction of its questions. Our experiments with state-of- the-art instruction- and commonsense language models reveal a significant gap between hu- man and model performance, which is further widened when consistency across reconstruc- tion formats is considered. We make all of our code and data available stimulate work on de- veloping and evaluating lateral thinking models.
(A) Glove.
(B) Computer.
Lateral Thinking
Sentence Puzzle
A man shaves everyday, yet keeps his beard long.:
his beard gets clean everyday -X
he is a barber and he shaves others
Word Puzzle
What type of 'cheese is made backwards?
Mozzarella
Feta
Edom
Figure 1: Contrasting existing Vertical Thinking tasks (PIQA (Bisk et al., 2020) and RiddleSense (Lin et al., 2021)) our novel lateral thinking task called BRAIN- TEASER. While prior tasks require commonsense to be injected, BRAINTEASER'S lateral thinking puzzles require default commonsense thinking to be deprecated.
outside the box") is divergent and creative pro- cess that involves looking a problem from a new perspective and defying preconceptions, associated with the right-brain hemisphere (De Bono, 1970; Waks, 1997). Lateral thinking is required to solve the puzzle in Figure (bottom), by overwriting the commonsense associations of man shaves to he shaves himself. and regarding the man somebody who shaves others all day (e.g., barber).
1 Introduction
Human reasoning processes comprise two types of thinking: vertical and lateral (Waks, 1997). Vertical thinking, also known as linear, convergent, or logi- cal thinking, is sequential analytical process that is based rationality, logic, and rules, typically associated with the left-brain hemisphere. Vertical thinking, as illustrated in Figure (top), is needed to create a reasoning path from flooding room to filling it with water for physical reasoning, and from inanimate objects with five fingers to gloves in riddles. Meanwhile, lateral thinking (or "thinking
The development of natural language processing (NLP) models and their evaluation has achieved much progress in vertical thinking. In particular, large language models (LLMs) (Devlin et al., 2019; Liu et al., 2019; Brown et al., 2020b) have achieved strong performance across a variety of complex rea- soning tasks (Talmor et al., 2019: 9; Bisk et al., 2020;
Work done when KM was at Carnegie Mellon University

---

Sap et al., 2019b), even with the complete absence (zero-shot) (Sanh et al., 2022) or limited provision (few-shot) of training time exemplars (Chung et al., 2022).! ! ing over physical interactions (Bisk et al., 2020) and social implications (Sap ct al., 2019b), LLMs exhibit better vertical thinking capabilities, includ- ing commonsense association (Wei et al., 2022) and inference ability (Bosselut et al., 2019). While the extent to which these models possess common sense is heavily discussed (Mareus, 2022; Bubeck et al., 2023; Wei et al., 2023), we note that prior work has not considered the lateral thinking ability of LLMs. Creative thinking problems in bench- marks and knowledge bases are often filtered out noise during preprocessing (Vajjala and Meur- ers, 2012; Speer et al., 2017; Sap al., 2019a), and only kept if their resolution can be supported by commonsense associations, as in the case of riddles (Figure 1) (Lin et al., 2021; Gao et al., 2018). As many situations are novel, we expect that lateral thinking puzzles like those in Figure 1-bottom will be hindered by default commonsense associations and cannot be easily solved by further adaptation and scaling of the existing LLM methods.
defying default commonsense associations. This systematic procedure results in a novel BRAIN- TEASER benchmark with 1.1K high-quality data points and nearly 100% human evaluation results. Using BRAINTEASER as the benchmark, we con- duct comprehensive experiments involving differ- ent model structures, model sizes, and prompting strategies. The results reveal a huge gap between human performance and current LLMs, indicating the great need to improve lateral thinking in LLMs.
To perform well on tasks such as reason-
We summarize our contributions as follows: 1) We introduce lateral thinking puzzles, a multiple- choice QA task designed to test the model's 's ability to exhibit lateral thinking and defy default com- monsense associations. 2) We design a three-step procedure for creating the first lateral thinking benchmark, BRAINTEASER, consisting of data collection, distractor generation, and generation of reconstruction examples, leading to 1,100 high- quality puzzles. 3) We conduct comprehensive experiments with state-of-the art LLMs. We make all of our code and data available to stimulate work on developing and evaluating lateral thinking mod- els.?
To bridge this gap, we propose to study the abil- ity of otate-of-the-art L.Mss to reason on lateral thinking puzzles. We formulate lateral thinking puz- zles multiple-choice Question Answering (QA) tasks, making them intuitive to answer by humans and easy to evaluate automatically, Following our task definition, we create a novel BRAINTEASHE benchmark with two tasks of different granularity: Sentence Puzzles and Word Puzzles (cf. Figure 1). To construct the dataset, we design data collec- tion procedure, which crawls relevant puzzles from several publicly available websites, performs semi- automatic filtering of irrelevant question categories (e.g.. pun, dad jokes), and ensures high data quality. To ensure fair and informative questions, we con- struct distractors aumi--utomatically by manual an- notation of the explicit and implicit (commonsense) premises that arise from each puzzle. To address concerns of possible LLM memorization (Carlini et al., 2022) and their lack of consistency (Gold- berg, 2023), enrich BRAINTEASER with two reconstruction strategies: semantic reconstruction and context reconstruction, which create variants of each puzzle without changing its original way of
2 Related work
We review prior work on computational creativity, commonsense reasoning, and model robustness.
Computational Creativity Computational cre- ativity work includes broader set of tasks, some of which have been relatively popular, including pun (Zou and Lu, 2019) and humor (Meaney et al., 2021) detection. A particular class of creative chal- lenges, called brain teasers (Draper, 2009; High- house al., 2019), is designed to evaluate a wide range of human intelligence skills, including strat- egy development, planning, visual-spatial thinking. creativity, and memory (Altun et al., 2016). Most similar to our task, Lin et al. (2021) collects riddles from public websites to challenge current models. While in principle computational creativity puz- zles and brain teasers combine vertical and lateral thinking, prior work has focused on the former cate- gory. Our BRAINTEASER task complements these works with word- and sentence-level lateral think- ing puzzles. BRAINTEASER can serve as a formal platform to evaluate the creative skills of LLMs, which have been partially explored in recent work
'In this paper, we use the terms language model and large language model interchangeably.
'The code is available at https//githhh.com/117 1711- jpg/ Tng/BrainTeaser

---

(Franceschellli and Musolesi, 2023; Bubeck al., 2023; Wang et al., 2023a).
the model improve robustness, we expect that the lateral thinking puzzles in BRAINTEASER re- quire unique associations and creative reasoning paths. In this way, BRAINTEASER is designed to minimize the impact of confounding factors like memorization in LLMs (Bang et al., 2023; Guo et al., 2023; Goldberg, 2023).
Commonsense Reasoning The task of common- sense reasoning has been popular in recent years (Rajani et al., 2019; Ma et al., 2019; Lourie et al.. 2021; Maharana and Bansal, 2022), accompanied by the introduction of numerous challenging bench- marks (Talmor et al., 2019; Sap et al., 2019b; Sakaguchi al., 2019) and availability of large- scale commonsense resources (Speer et al., 2017; Hwang et al., 2021). While each of the existing datasets focuses on different dimensions of com- monsense knowledge (llievski et al., 2021a), most them are constructed in the multiple-choice for- mat, due to the ease of evaluation. Some prior works have focused on generative commonsense reasoning (Lin et al., 2020; Boratko et al., 2020). However, due to the vast plausible answer space, the evaluation has been challenging and large amount of answer annotations have be collected in order to ensure fairness (Boratko et al., 2020). Curiously, while possession of common sense has been a central goal of AI, its role in our BRAIN- TEASER task is as distractor. Namely, successful solutions of the lateral thinking puzzles in BRAIN- TEASER require the models to defy commonsense associations and linear inference chains.
3 Construction of BRAINTEASER
In this section, we first provide a definition of lat- eral thinking puzzles in various granularities. We then present a three-stage pipeline for constructing the multiple-choice puzzles in the BRAINTEASER dataset, consisting of data collection, distractor sampling, and reconstruction sample generation. Finally, we present key data statistics and quality validation results.
3.1 Task Definition
While lateral thinking puzzles are often presented to humans in an open-ended fashion, these are dif- ficult evaluate automatically and are difficult to solve by humans.² An additional complication is that there may be multiple independent, yet correct, puzzle explanations. To alleviate these challenges, we pose lateral thinking puzzles as a multiple- choice QA task, format frequently employed for reasoning tasks. We expect this approach be both facile for human comprehension and amenable automated evaluation. In general, each puzzle con- tains a question Q stating the context, and a lateral explanation e from explanation space E that serves as the correct answer. Q can be decomposed into an atomic premise set P. which includes both ex- plicitly stated clauses and implicit clauses derived through default commonsense inferences associ- ations. For example, in the following puzzle: "How could cowboy ride into town on Friday, stay two days, and ride out on Wednesday?" the set P in- cludes the following premises:
Robustness Studies As novel benchmark, BRAINTEASER relates to other works that evaluate the performance of LLMs. Since these models are surpassing human performance on some existing benchmarks (Xu et al., 2022), the NLP community has shifted the focus towards robustness evalua- tion, i.c., whether the model can retain a similar performance to semantically perturbed or adver- sarially constructed questions (Abdou et al., 2020; Nie et al., 2020). Some recent works have adopted model adversarial approaches to generate datasets that are challenging for models solve (Zellers et al., 2019; Sakaguchi et al., 2019), while oth- ers combine multiple tasks to evaluate the model's behavioral consistency across semantic, logical, and factual categories (Jang et al., 2022). Be- sides dataset construction, analysis studies have also shown that models casily learn shortcuts to solve the datasets (Branco et al., 2021; Elazar et al.. 2021) and their performance heavily depends on the overlap of tokens between training and test data (Ma et al., 2021b). Different from prior works where associative resources are used to finetume
P1: Cowboy rides into town Friday.
P2: Cowboy stays in town for two days.
P3: Cowboy rides out on Wednesday.
P4: Wednesday is the third day of the week.
Ps: Sunday is two days after Friday.
*Our small-scale user study shows that both humans and LLMs are unable to perform this open-ended task well, scor- ing 2.64 and 2.62 on 5-point scale, respectively (see Ap- pendix A.5 for details).

---

The premises pt, p2, and ps are explicitly provided by the context, and the premises P4 and ps are implicitly obtained by default commonsense asso- ciation. The goal of puzzle is to find an expla- nation that does not contradict the premise set P. En = 2, as the premises are the target to explain and support. With vertical thinking, the question appears impossible to answer because P contains statements that conflict with each other. The premises P3 and pl are inconsistent with other premises, leading to an obstacle in explaining the puzzle. The default commonsense inference thus becomes a logic stumper (Bar-Hillel et al., 2018), 8), preventing one from creatively exploring additional explanations in E.
Table 1: Example generated distractors.
Premise
Answer/Distractor
Pw: Wednesday is the
Answer: His horse
third day of the week.
named Wednesday.
P2: Cowboy stays in town for two days.
Distractor: While in town, he stays bed for two days.
P5: Sunday is two days
Distractor: Friday and Saturday are holidays.
past Friday.
3.3 Distractor Sampling
We convert each puzzle and its explanation into a multiple-choice QA format to ensure a straight- forward evaluation process, A key challenge in creating fair and informative multiple-choice ques- tions is sampling distractors that are simultaneously incorrect and challenging (Ma et al., 2021 We propose a systematic approach for distractor sam- pling that directly benefits from our premise-base definition of lateral thinking puzzles.
Lateral thinking leads correct solution to this puzzle: "His horse is named Wednesday.". This creative solution defies the commonsense associa- tion of Wednesday as a third day of the week (p4). Thus, the key point of a lateral thinking puzzle is that some implicit premises generated through de- fault commonsense association incorrectly create an arbitrary "box" that wrongly excludes the possi- ble solution from the explamation space (Bar-Hille) al., 2018).
For sentence puzzles, we list possible premises P {P1, P2. P3.P3....J from the question context manually as the commonsense associations in the data are obvious and straightforward, especially when the answers are provided, like the example in Section 3.1. We know the correct answer P/. is an unconventional overwriting of the wrong premise (logic stumper) Pa generated by default common- sense association. We generate the distractors by overwriting other premises in P Pic- This pro- cedure guarantees that the distractors are incorrect because the misleading premise pw still remains in the premise set and prevents one from reaching the correct explanation. We first use COMET (Hwang et al.. 2021) to generate the possible premise over- writing candidates for the question as a head com- bined with inference relations (e.g., happens after, hindered by, cause). Then we pick the COMET- generated tails that are consistent with the question context as distractors and revise them by manual annotation. Table shows example distractors for our running example puzzle from Section 3.1.
Upon careful exploration, we devise two granu- larity variants of lateral thinking puzzles following our definition (Figure 1): sentence-based, where the puzzle is centered on sentence premises (e.g., Wednesday is the third day of the week), and word- based, where the answer violates the default mean- ing of the word and focuses on the letter composi- tion of the target question (e.g., cheese made back- wards- > edam).
3.2 Data Collection
We collect over ten thousand lateral thinking puz- zles with answers from public websites such as riddles. com and rd. com using web crawlers. We merge the data from different sources and re- move (near- suur-addulls based on sentence similar- ity (Reimers and Gurevych, 2019). We conduct semi-automatic process that corrects typos by us- ing an automatic library, Auto Correct," followed by human verification to ensure that the puzzles preserve their original meaning. We filter the re- maining data manually to preserve QA pairs that fit the definition of the sentence- or word-based lateral thinking puzzles. This process yields 373 unique lateral puzzles, formatted as QA pairs.
com
For word puzzles, ing rather than semantic meaning, distractors can share similar semantic meaning as the correct an- swers and still exhibit similar commonsense as- sociations. We pick distractors from the correct answer's s synonyms in WordNet (e.g., mozzarella for edam in Figure 1) and Wikipedia entries that belong to the same category (e.g.. both edum and cheddar belong to the semi-hard cheese category). Since it is generally possible that none of the cre-
focus on the literal mean-

---

Table 2: A sentence-based lateral thinking puzzle and its reconstruction variations. We present an analogous word-level puzzle in the Appendix A.3.
Adv Strategy
Question
Answers
His horse is named Wednesday. While in town, he stays in bed for two days. Friday and Saturday are holidays.
How could cowbuy ride into town on Friday, stay two days, and ride out on Wednesday?
None of the above.
His horse named Wednesday. While town, he stays in bed for days. Friday and Saturday holidays. None of the above.
Semantic Re- construction
How could cowboy come into town Friday, stay days, and then ride away on Wednesday?
The pilot's airplane is named Tuesday. flies straight for 24h flies quickly hours left. There was one week long holiday.
Context Re- construction
How can a pilot take off in Los Angeles on Tuesday. fly for 48 hours, and land Tokyo on Tuesday?
None of the above.
ative solutions will be sensible for some of the ques- tions, we also include the option None of the above in all questions' candidates set. This answer candi- date simulates the situation where humans cannot overwrite their commonsense inference and give up on explaining the lateral thinking puzzle. To cre- ate puzzles where lateral thinking fails (i.e., with answer None of the above). we replace the correct answer with distractor in 6% of the questions. Af- ter this procedure, each question in BRAINTEASER has four answer candidates.
Table Key statistics of the BRAINTEASER dataset. Choices combine the correct answer with all the distrac-
tors. Standard deviation is computed without the None
of the above choice, as its token length fixed and not related to the question context.
Sentence
Word
# Puzzles
627
492
Average Question Tokens
34.88
10.65
% Long Question (>30 tokens)
48.32%
2.23%
Average Answer Tokens Std of Choice Tokens
9.11
3.0
2.36
0.52
3.4 Generating Reconstruction Examples
model that learns to reason should be able to solve these two reconstruction variants of the questions easily, whereas the model that memorizes the an- swer would stumble.
Since the latest LLMs are pretrained on massive web snapshots, it is possible that the data sources for BRAINTEASER are also included in their train- ing data. Consequently, it is possible for LLMs to memorize the correct answer without performing any reasoning. To ensure that our task evaluates lateral thinking ability rather than memorization, we construct reconstruction versions of the original data in two parallel ways (Table 2): (1) Seman- tic Reconstruction rephrases the original question without changing its answer, distractor, and any premises in P. To do so, we use an open-source rephrasing tool,5 after which human annotators re- fine and validate that all premises remain the same. (2) Context Reconstruction keeps the misleading commonsense premise intact and changes both the question and the answer to a new situational con- text. For this purpose, we prompt GPT-4 for initial reconstructions, which are then manually refined by human annotators. The new distractors are gen- erated following the same process as in Section 3.3. The premise set and the corresponding distractors also get translated to the new context. Intuitively,
3.5 Data Analysis and Validation
Key Statisties BRAINTEASER includes 1,119 data samples including reconstruction variants. Table 3 reports key statistics of each subtask of BRAINTEASER. The questions in the Sentence Puzzle category are much longer because they are in narrative story format rather than simple short questions, like those in the Word Puzzle category. The difference between the standard deviation in the number of choice tokens between Sentence Puz- zle and Word Puzzle can be ascribed to the different strategies for generating distractors, i.e., overwrit- ing various premises with new statements versus generating similar words from the synonym set. We use ChatGPT prompting to extract the con- text topic from each question and to analyze the major topics in each subtask. The topic distribution shows that both subtasks involve large range of (more than 80) areas. Sentence Puzzle is denom- inated by math, physics, and nature while Word Puzzle is denominated particularly by the language
/ htttps://quilbbot.com

---

topic. For both tasks, there is a long tail of less common topics. The details of topic extraction and its obtained statisties are given in the Appendix A.I. The data statistics and the topic analysis suggest that, despite its limited size, BRAINTEASER can function as a comprchensive benchmark for assess- ing model performance across diverse topics and varying lengths of context.
common sense: 1) RoBERTa-L (CSKG) (Ma al., 2021a), a model finetuned on the synthetic QA pairs generated from a diverse set of commonsense knowledge graphs (CSKG) (Tlievski et al., 202 2) CAR (Wang et al., 2023b), a model finetuned similar pipeline as Ma et al. (2021a) but with en- hanced negative sampling strategy and reportedly superior performance. For reference, we also in- clude the vanilla RoBERTa model (Liu et al., 2019) to understand the impact of commonsense knowl- edge. We evaluate all of the models in a zero-shot fashion, following the scoring method defined (Ma et al., 2021a). We select RoBERTa because its widespread usage of the commonsense task and impressive zero-shot performance. RoBERTa-L. (CSKG) achieve SOTA zero-shot result on multiple Commonsense tasks, while CAR even outperforms ChatGPT on commonsense tasks.
Human Validation To ensure the quality of our dataset, we invited three expert annotators to ver- ify the validity of the QA pairs and their recon- struction variants. We sampled 102 examples from BRAINTEASER randomly and asked the annotators the following two questions: 1) Does the original puzzle and correct answer make sense? 2) Are the reconstruction variants still consistent with the orig- inal questions in terms of the required reasoning process? On average, the human annotators rated 99% of the original question guestion-answering answering pairs as valid. 100% of the semantic reconstructions and 97% context reconstructions were marked as con- sistent with the original question-answer pair. The overall Fleiss (1971) kappa annotaoro agree- ment 0.948, which is an almost perfect score.
Human Evaluation To assess the upper bound per- formance on BRAINTEASER, we randomly sample 102 questions from and invite three experts an- notator to solve the test. On average, takes one hour for an annotator to complete the task.
4.2 Evaluation Metries
4 Experimental Setup
As accuracy is fair evaluation metric for the MCQA format and it has been adopted by many popular commonsense reasoning tasks (Mihaylov et al., 2018; Talmor al., 2019; Bisk et al., 2020), we evaluate model performance using two accuracy metrics: Instance- Accuracy considers each (original or reconstruction) question separately. We report instance- bastance-bbsed accuracy on the original puz- zles, and their semantic and context reconstructions. Group-based Accuracy considers each original puzzle and its variants score only when it successfully solves all three puzzles the group, otherwise, its score
We describe the models selected for our experi- ments and the metrics used to evaluate the reason- ing accuracy and consistency of these models.
4.1 Model selection
Instruction-Based Models We evaluate the ceatictiion-innntnee LLMs in zero/few-shot set- ting: 1) ChatGPT, a publicly available of the-art LLM from the GPT (Brown et al., 2020a) series. 2) TO (Sanh et al., 2022), a LLM trained with multitasking instruction tuning that has strong zero-shot generalization ability. 3) FlanT5 (Chung al., 2022), enhanced version of T5 (Raffel al.. 2020) which instruction finetuned et al., 2021) in both zero-shot and few-shot setting. For a fair comparison with humans, while running zero-shot prompting on ChatGPT, we add a descrip- tion indicating that the question is a brain teaser puzzle that needs creative thinking to solve. For the rest of the models, we use the same instruction templates found in their training sets (for full details, please refer to Appendix A.2).
group. The model will
Results
faneettion-finetueeed (Wei
Our experiments target five questions: 1) Can LLMs reason on lateral thinking puzzles similar humans? 2) How do LLMS perform on reconstruc- tion variants? 3) Are model predictions consistent across partitions? 4) Does tuning on commonsense knowledge help to answer BRAINTEASER puzzles better? 5) Can LLMS do better in the few-shot setting with more demonstrations?
Commonsense Models To understand the effect commonsense knowledge on our task, we eval- uate the following models that are enhanced with
Overall Performance The main results are shown in Table 4. For both word and sentence BRAINTEASER puzzles, the performance of the

---

Table 4: Main zero-shot results over two BRAINTEASED subtasks across all models all metrics: Ori = Original. Sem Semantic, Con Context. The best performance among all models is in bold, and the best performance in commonsense augmented models is underlined. The human evaluation (*) is computed over 102 randomly sampled data. The random base average over three different seeds.
Instance-haid hasance-hased
Group-based
Overall
Category
Model
Original
Semantic Context
Orl & Sem
Ori & Sem & Con
Sentence Puzzle
Random
25.52
24.88
22.81
5.58
1.44
24.40
F578T)(780M)
18.66
16.27
22.01
10.53
4.31
18.98
FS3BT
26.79
25.36
35.41
20.10
12.92
29.19
FlanTS(11B)
33,49
31.58
36.84
22.01
33,97
Instruction
TO(1IB)
22.01
22.01
29.67
16.27
11.00
24.56
TOP(
23.92
17.70
27.11
TOPP(1IB)
26.32
27.27
37.80
19.14
11.96
30.46
ChatGPT
60.77
59.33
50.72
39.71
62.68
RoBERTa-L
43.54
40.19
46.41
33.01
20.10
43.38
Commonsense
RoBERTa-L(CSKG)
35.41
36.84
44.98
28.71
18.18
39.07
CAR
10.53
10.53
11.48
5.74
2.39
10.85
Human"
90.74
90.74
94.44
90.74
88.89
91.98
Word Puzzle
Random
26.02
27.85
22.51
7.32
1.83
25.34
FlanT5(78(M)
22.56
17.68
28.66
9.15
3.66
22.97
FlanTS(3B)
37.80
29.88
42.68
23.17
12.80
36.79
FlanT5(11BB
42.68
32.93
43.90
28.66
20.12
39.84
Instruction
TO(1IB)
17.07
23.17
9.76
18.09
TBP(11B)
28.66
26.22
34.15
19.51
12.80
29.67
TOPP(11B)
33.54
31.10
20.12
34.76
ChatGPT
56.10
52.44
43,90
53.46
RoBERTa-L.
19.51
T9.51
23.17
14.63
20.73
Commonsense
RoBERTa-L(CSKG)
18.90
16.46
30.49
12,80
6.10
21.95
CAR
38.41
31.10
20.12
26.22
6.10
29.88
Human
91.67
91.67
91.67
91.67
89.58
91.67
strongest model, ChatGPT (53 and 63%) is halfway between random (25%) and human performance (92%). In general, neither type of model is able to perform consistently well across the two subtasks: instruction-basee models perform better word puzzles, whereas commonsense models perform slightly better on sentence puzzles. The perfor- mance of the models is often close to random, with around a third the models performing equal or worse than random guessing. As it can be expected, we see that scaling up cestruction-iietuunne models leads to improved performance on both subtasks. Yet, the large gap between human and model per- formance clearly shows that even the most power- ful LLMs are unable to exhibit lateral thinking in multiple-choice puzzles and confirms the challeng- ing nature of our BRAINTEASER dataset.
structures. Meanwhile, we note that our contextual reconstruction mechanism yields puzzles that are more familiar or casier to solve than the original puzzle, possibly because some of the commonsense associations are relatively weaker. An exception to this trend is ChatGPT's performance word puzzles, where ChatGPT performs the best the original examples. We believe that this due a combination two factors. First, the word puzzle reconstructions only have a limited impact on the vocabulary domain and sentence structure, because of the much shorter questions. Second, ChatGPT may have memorized some of the word puzzles, e.g., given the question "How do you spell COW in thirteen letters?". its answer begins with "The question seems be a brain teaser ... We pro- vide representative examples of the prevalent lateral thinking errors of memorization and commonsense associations in Table 5.
Original vs Reconstruction Partitions In most cases, all models and humans perform the best the context reconstruction partition. We hypoth- esize that this is because original lateral thinking puzzles are designed to mislead humans to wrong choice based on commonsense associations, often involving rare words and unconventional sentence
Consistency of Model Predictions We further compare the performance on instance- and group- based metrics understand whether the models can solve lateral thinking puzzles by following a consistent reasoning path. model understand-

---

Table 5; Error analysis on memorization and commonsense association.
Question
Answer
LLM choice
Memorization
The calls his dog on the other side of the river, and the dog crosses the river without getting wet and using tools.
The was frozen.
The river frozen.
The man to cross the rivers. can't It swim or use any tools like bridge. How does the man succeed in the end?
river was frozen.
He jumped half-mile far to across the river.
Commonsense Association
What animal has no wings. but yet will fly?
A caterpillar.
An cagle.
There is no light on the road and the car's 's headlight is broken.
It was daytime.
The driver is good at listening.
How can the driver see the black dog?
How can Jenny read in totally light house night?
The bonk is in Braille.
It was daytime.
ing rather than memorizing the reasoning path of the original brain teaser should be able to answer its adversarial reconstructions with ease. Notably, human performance only has minimal drop on group-based metrics whereas all models suffer sig- nificant drops. Further analysis (see Appendix A.6) reveals that ChatGPT and RoBERTa-L fail to an- swer many (45 and 61% %, respectively) of the origi- nal or semantically changed puzzles when contextu- ally translated puzzles are solved correctly. These observations suggest that the ability of the models to perform consistent lateral thinking is far from human ability.
ChetGPT Sentence Puzzle
((118) Sentence Puzzle
Accuracy (82)
50
30
40
35-
15
ChatGPT Word Puzzle
FlanT5(11B) Word Puzzle
60
55-
Accurary
45-
30
40
25
15
Impact of Commonsense Knowledge We ob- serve that commonsense knowledge has a salient negative impact on the model's performance on sentence puzzles. The west-performing model in the commonsense category is the vanilla RoBERTa model, whose adaptation with commonsense knowledge leads to a significant drop in results, es- pecially with the CAR method. This trend confirms our initial hypothesis that learning commonsense associations generally detrimental to complex lat- eral thinking tasks. Commonsense knowledge has a limited positive impact on the word-puzzle task, possibly because much of the commonsense associ- ations learned by these models hold between words, including synonyms. Finally, given the apparent similarity of riddles and lateral thinking puzzles, we finetuned a RoBERTa model on the Riddle- Sense dataset and evaluated it on our task. Again, we observe that the model struggles on solving the puzzles despite gaining better results compared to the vanilla RoBERTa model (see Appendix A.7).
Original
Samartic
Contant
DrikZimm
Figure 2: Few-shot prompting performance of ChatGPT and FlanB)
and FlanT5 (1IB). We randomly pick 8 puzzles (4 from each subtask) and create new context re- constructions as demonstrations. We experiment with few-shot prompting with 2,4, 6, and 8 of these demonstrations, balanced between the two subtasks. The few-shot results are shown in Figure 2, and we present the full results in Appendix A.4. The num- ber of few-shot demonstrations has clear impact on sentence puzzles, which confirms that lateral thinking puzzles are unique and the models can hardly learn generalizable patterns from in-context examples. Providing more few-shot demonstra- tions has a marginal positive impact for word puz- zles. Given this task's focus on the letter compo- sition of each word, the in-context examples may be used to teach the model to pay attention the surface form of the answer candidates. It's worth noting that, although few-shot examples might ex- hibit superficial resemblances, their contribution to model generalization for sentence puzzles remains
Impact of FSw-Shot Demonstrations As LLMs are good few-shot learners (Brown al., 2020b), we are interested to see in-context learning can help them better solve our task. We experiment with our two most powerful models: ChatGPT

---

minimal, given the abstract nature of reasoning pattern in this subtask.
that combines original puzzles and their reconstruc- tion variants. Our experiments showed that Chat- GPT's performance on this task is halfway between random and humans, whereas other models often perform close to random. While scaling up model size improved performance, enriching with com- mon sense or providing few-shot demonstrations yielded limited benefits. Mcanwhile, all models tend to solve the variants of the same puzzle in- consistently. Our error analysis showed that the models' lateral thinking is often hindered by memo- rization and mislcading commonsense associations. In the future, we intend to develop lateral thinking models, create additional lateral thinking evalua- tion tasks (e.g., relating to alteration (De Bono, 1970)), and investigate flexible ways to combine lateral and vertical thinking.
Qualitative Error Analysis We analyze two prevalent lateral thinking errors in the ChatGPT and FlanT5 (11b) LLMs: memorization and common- sense associations, both of which become more apparent with scaling up (Carlini al., 2022). We show examples in Table 5.
Memorization We find that memorization hap- pens in both subtasks. Given the sentence puzzle "The man calls his dog on the other side of the river, crosses the river without getting wet and using ant tools." the LLMs picked the correct answer "The river was frozen." for both the original and its se- mantic reconstruction. However, when the question in new context becomes "The man had to cross the rivers. He can't 't swim or use any tools. like the bridge. How does the man succeed in the end?". all LLMs failed to answer. Memorization more frequent in word puzzles. A semantic reconstruc- tion will cause confusion in the model, as is also apparent from the gap between original accuracy and the ori&sem accuracy in Table 4.
Limitations
While our work focuses on both Sentence puzzles and Word puzzles, we intend to develop a compre- hensive lateral thinking benchmark according to de Bono's four skills: awareness, random stimula- tion, alternatives, and alteration (De Bono, 1970). Moreover, while our paper tries to provide a clear distinction between lateral and vertical thinking, it remains open question to which extent other brain teaser categories, e.g. puns and visual puz- zles, require lateral or vertical thinking. As these tasks are not the focus of our present paper, we leave to future work to comprehensively evalu- ate models' ability to think out of the box on such tasks and to characterize the complementary and opposing aspects of vertical and lateral thinking.
Commonsense association Similarly, we also find that commonsense association often confuses LLMs. For example, for "What animal has no wings, but yet will fly?", , the models associate the words "wings" and "fly" with birds and pick the wrong answer "An eagle." despite the contradiction between "eagle" and "no wings". Meanwhile, the correct lateral thinking answer "A caterpillar." is not picked by the models. Interestingly, common- sense associations that mislead models in some examples can be the needed hint in others. For example, one puzzle, "There no light on the road and the car's headlight is broken. How can the driver see the black dog?", , the answer "It was daytime." is hindered by the commonsense associa- tion between mentioning no light and night. How- ever, in another example, "How can Jenny read in totally light house night?", the same commonsense association leads the model to the correct answer: "The book is in Braille.". In the second example, the answer is misled by another commonsense association related to reading.
Also, opt for constructing the dataset in multiple-choice QA format to reduce the burden of the evaluation process. However, this inevitably reduces the difficulty of the task and permits the situation where the models solve the questions cor- rectly for the wrong reasons. Future work should also look into better evaluation metrics that are suitable for creative and open-ended generations such that our task can also be adapted to open- ended setting. Finally, while our current puzzles are provided in a static manner, future work should also investigate an interactive (multi-step) setup, where the model (or human) may ask clarification questions or receive contextual hints.
6 Conclusions and Outlook
We defined the task of lateral thinking for LLMs, formulated as a multiple-choice QA with a sentence- and word-level puzzles. We developed BRAINTEASER, a 1. lateral thinking benchmark

---

Ethical Considerations
Language Processing (EMNLP), pages 1122-1136, Online. Association for Computational Linguistics.
As our lateral thinking puzzles are "folk knowl- edge" and are published on a range set of websites, it is hard to check their original licenses compre- hensively. Yet, the website owners declare per- mission to print and download material for non- commercial use without modification on the mate- rial's copyright. Therefore, we provide the corre- sponding copyright statements and website URLs for each original lateral thinking puzzle and its re- construction version. In addition, we will create a form to ask future dataset users to sign a document claiming that the only aim of the data usage is re- search before providing them with the data. We note that, despite our best efforts, the task data may still contain bias in terms of gender or politics. We will indicate that future research should use the task data with caution.
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chai- Lanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. Comet: Commonsense transformers for knowl- edge graph construction. In Association for Compu- lational Linguistics (ACL).
Ruben Branco, António Branco, João António Ro- drigues, and João Ricardo Silva. 2021. Shortcutted commonsense: Data spuriousness in deep learning of commonsense reasoning. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1504-1521, Online and Punta Cana, Dominican Republic. Association for Computational Linguisties.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffney Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020a. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems. volume 33, pages 1877-1901. Curran Associates, Inc.
References
Mostafa Abdou, Vinit Ravishankar, Maria Barrett, Yonatan Belinkov, Desmond Elliott, and Anders Sø- gaard. 2020. The sensitivity of language models and humans Winograd schema perturbations. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics. pages 7590- 7604, Online. Association for Computational Lin- guistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020b. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-9911
Meryem Altun, Muhsin Hazar, and Zekihan Hazar. 2016. Investigation of the effects of brain teasers on atten- tion spans of pre-school children. International jour- nal of environmental and science education, 11:8112- 19.
Sébastien Bubeck, Varun Chandrasekaran, Ronen El- dan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lund- berg, et al. 2023. Sparks of artificial general intelli- gence: Early experiments with gpt-4. arXiv preprint 2231122303.12712
Yejin Bang, Samuel Cahyawijaya, Naycon Lee, Wen- liang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. A multitask, multilin- gual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity.
Nicholas Carlini, Daphne Ippolito. Matthew Jagielski. Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022. Quantifying memorization across neural lan- guage models. arXiv preprint 22Xx:22002.076466
Maya Bar-Hillel, Tom Noah, and Shane Frederick. 2018. Learning psychology from riddles: The case of stumpers. Judgment and Decision Making, 12-22112-1222
Hyung Won Chung. Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, al. 2022. Sealing instruction instuuctionfinetuned finetuned language models. arXiv preprint ar 14466.
Yonatan Bisk, Rowan 7ellers, Jianfeng Gao, Yejin Choi, al. 2020. Piqa: Reasoning about physical com- monsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432-7439.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot. Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafford. 2018. Think you have solved question an- swering? try arc, the ai2 reasoning challenge. ArXiv, 1bs/1803.05457.
Michael Boratko, Xiang Li, Tim O' O'Gorman. Rajarshi Das, Dan Le, and Andrew McCallum. 2020. Pro- toQA: A question answering dataset for prototypi- cal common-sense reasoning. Proceedings of the 2020 Conference on Empirical Methods in Natural
Edward De Bono. 1970. Lateral thinking. New York.

---

Jacob Devlin, Ming-Wor Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Mycongjun Jang. Deuk Sin Kwon, and Thomas Lukasiewicz. 2022. Becel: Benchmark for consis- tency evaluation of language models. In Proceedings of the 29th International Conference on Computa- tional Linguistics, pages 3680-3696.
Bill Yuchen Lin, Ziyi Wu, Yichi Yang. Dong-Ho Lee, and Xiang Ren. 2021. Riddlesense: Reasoning about riddle questions featuring linguistic creativ- ity and commonsense knowledge. arXiv preprint arXiv:2101.00376.
Stephen W Draper: 2009. Catalytic assessment: under- standing how meqs and evs can foster deep learn- ing. British Journal of Educational Technology, 40(2):285-22
Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen. Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. 2020. CommonGen: A constrained text gen- eration challenge for generative commonsense rea- soning. In Findings of the Association for Computa- tional Linguisties: EMNLP 2020, pages 1923-1849 Online. Association for Computational Linguistics.
Yanai Elazar, Hongming Zhang, Yoav Goldberg, and Dan Roth. 2021. Back to square one: Artifact detec- tion, training and commonsense disentanglement in the Winograd schema. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 10486-10500, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Yinhan Liu, Myle Ou, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy. Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach.
Joseph I. Fleiss. 1971. Measuring nominal scale agree- ment among many raters. Psychological bulletin, 76(5):378.
Nicholas Lourie, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Unicom on rainbow: A uni- versal commonsense reasoning model on new mul- titask benchmark. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 35, pages 13480-1333 13488.
Giorgio Franceschelli and Mirco Musolesi. 2023. On the creativity of large language models.
Ge Gao, Eunsol Choi, Yejin Choi, and Luke Zettle- moyer. 2018. Neural metaphor detection in context. Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 607-613, Brussels. Belgium. Association for Com- putational Linguistics.
Kaixin Ma, Jonathan Francis, Quanyang Lu, Eric Ny- berg, and Alessandro Oltramari. 2019, Towards gen- eralizable neuro-symboll. systems for commonsense question answering. In Proceedings of the First Workshop Commonsense Inference in Natural Lan- guage Processing, pages 22-32, Hong Kong, China. Association for Computational Linguisties.
Yoav Goldberg. 2023. Two kinds of recall. arXiv preprint 2333v 70527.
Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nic, Yuxuan Ding, Jianwei Yuc, and Yupeng Wu. 2023. How close is chatgpt to human experts? comparison corpus, evaluation, and detection.
Kaixin Ma, Filip llievski, Jonathan Francis, Yonatan Bisk, Eric Nyberg. and Alessandro Oltramari. 2021a. Knowledge-driven data construction for zero-shot evaluation in commonsense question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 13507-13515.
Scott Highhouse, Christopher D Nye, and Don C Zhang. 2019. Dark motives and elective use of brainteaser interview questions. Applied Psychology, 68(2):311 340.
Kaixin Ma, Filip Ilievski, Jonathan Francis, Satoru Ozaki, Erie Nyherg, and Alessandro Oltramari. 202 Exploring strategies for generalizable commonsense reasoning with pre-trained models. EMNLP 2021.
Jena D. Hwang. Chandra Bhagavatula, Ronan Le Bras, Jeff Da. Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi. 2021. Comet-atomic 2020: On sym- bolic and neural commonsense knowledge graphs. In AAAI
Filip Ilieskki, Alessandro Oltramari, Kaixin Ma, Bin Zhang, Deborah I. McGuinness, and Pedro Szekely. 2021a. Dimensions of commonsense knowledge. Knowledge-Bass Systems.
Adyasha Maharana and Mohit Bansal, 2022. On cur- riculum learning for commonsense reasoning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 983-992, Seatlle, United States. Association for Computational Linguistics.
Filip llievski, Pedro Szekely, and Bin Zhang. 2021b. Cskg: The commonsense knowledge graph. In The Semantic Web: 18th International Conference, ESWC 2021, Viruual Event. Jane 6-10, 2021, Proceedings 18, pages 680-696. Springer.
Gary Marcus. 2022. Deep learning is hitting a wall. Nautilus, Accessed, pages 03-11.

---

J. A. Meaney, Steven Wilson, Luis Chiruzzo, Adam Lopez, and Walid Magdy. 2021. SemEval 2021 task 7: HaHackathon, detecting and rating humor and offense. In Proceedings of the 15th Interaaionall Workshop on Semantic Evaluation (SemEval-202)) pages 105-119, Online. Association for Computa- tional Linguisties.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019b. Social iqa: Com- monsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNILP)) pages 4463-4473.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can suit of armor conduct elec- tricity? new dataset for open book question an- swering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381 13891, Brussels, Belgium. Association for Computational I Linguistics.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of gen- eral knowledge. In Proceedings of the AAAI confer- ence on artificial intelligence, volume 31.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsenseqq: A question answering challenge targeting commonsense knowl- edge. In Proceedings of NAACL-HLT, pages 4149- 4158.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language under- standing. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 4885 4901, Online, Association for Computa- tional Linguistics.
Sowmya Vajjala and Detmar Meurers. 2012. On im- proving the accuracy of readability classification us- ing insights from second language acquisition. In Proceedings of the seventh workshop on building ed- ucational applications using NLP, pages 163-173.
Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Learning Research, (1(5555555)5551
Shlomo Waks. 1997. Lateral thinking and technology education. Journal of Science Education and Tech- nology, 6:245-255.
Guanzhi Wang. Yugi Xie, Yunfan Jiang, Ajay Man- dlekar, Chaowei Xiao. Yuke Zhu, Linxi Fan, and An- ima Anandkumar. 2023a. Voyager: An open-ended embodied agent with large language models.
Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Explain your- self! leveraging language models for commonsense reasoning. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 4932-4942. 4942, Florence, Italy, Association for Computational Linguistics.
Weiqi Wang, Tianqing Fang, Wenxuan Ding, Baixuan Xu, Xin Liu, Yangqiu Song, and Antoine Bosse- lut. 2023b. Car: pareepptalizatiooauuneee soner for zero-shot commonsense question answer- ing. arXiv preprint UxXiv:235..4466
rea-
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNIPP, pages 3982-3992, Hong Kong, China. Association for Com- putational Linguistics.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M Dai, and Quoc V Le. 2021. Finetuned lan- guage models are zero-shot learners. arXiv preprint 1XX.:2109.0015:
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph. Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, al. 2023. Emergent abilities of large language models. Transactions on Machine Learning Research.
Keisuke Sakaguchi. Ronan Le Bras, Chandra Bhagavat- ula, and Yejin Choi. 2019. Winogrande: An adver- sarial winograd schema challenge at scale.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey. al. 2022. Multitask prompted training enables zero-shot task generalization. In International Con- ference on Learning Representations.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Fdd Chi, Quac Le, and Denny Zhou. 2022. Chain-of-thought prompting elic- its reasoning in large language models. In Advances in Neural Information Processing Systems.
Yichong Xu, Chenguang Zhu, Shuohang Wang. Siqi Sun, Hao Cheng, Xiaodong Liu, Jianfeng Gao. Pengcheng He, Michael Zeng, and Xuedong Huang. 2022. Human parity on commonsenseqa: Augment- ing self-attention with externa attention. Proceed- ings of the Thirty-Firsi International Joint Confer- ence on Artificial Intelligence.
Maarten Sap. Ronan Le Bras, Emily Allaway, Chan dra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi. 2019a. Atomie: An atlas of machine commonsense for th- reasoning. In AAAI Conference on Artifi- cial Intelligence.

---

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a ma- chine really finish your sentence? In Proceedings of the 57th Anmual Meeting of the Association for Com- putational Linguistics, pages 4791-4800, Florence, Italy, Association for Computational Linguistics.
A
Appendix
A.1
Puzzle topics
We use few-shot prompting in ChatGPT to extract the context topic for each question. Table 6 shows the top 10 topics for each subtask, for which the prompting template is as follows:
Yanyan 7ou and Wei Lu. 2019. Joint detection and
location of English puns. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 21 2117-2123, Minneapolis, Minnesota. Association for Computational Linguistics.
Can you provide context environment in the fol- lowing brain teasers? Here are several examples: [examples)"
Table Top-10 topics extracted for both subtasks.
Sentence Puzzie
Word Puzzic
Topic
Prequency
Topic
Frequency
Mathematics
45
Language
79
Physics
Food
27
Nature
37
Mathematics
26
Transportation
32
Animals
24
Animals
25
Science
22
Sports
24
Time
18
Family
23
Geography
19
Nature
Education
16
Entertainment
Law
16
Finance
11
Others
339
Others
244
A.2 Prompting templates
ChatGPT We use the following instruction to prompt ChatGPT:
"Please pick the best choice for the brain teaser. Each brain teaser has only one possible solution, including the choice none of the above, answer should only provide the choice:"
FlanT5 We use the instruction template for the AI2 Reasoning Challenge (ARC) (Clark et al., 2018): "Question: (Question]
What is the correct answer to the question from the following choices?
(A) (choice)
(B) [choice)
(C) [choice)
(D) [choice]"
TOPP We use the instruction template for the Com-
mensenseQA task (Talmor et al., 2019):
"[Question]
Choose the most suitable option to answer the above question.
Options:
A. (choice)
B. (choice)
C. [choice]

---

D. [choice)"
score 1:
Try answer the question, but the answer doesn't make sense.
A.3 Word puzzle example
The answer is wrong but related the golden label.
score 2:
Table 7 presents a word puzzle with its reconstruc- tion examples.
The answer is wrong, but the reason- ing strategy is similar to the golden answer and may lack some keywords.
score 3:
A.4 Few-shot prompting result
Table 8 shows the fhw-shot result of ChatGPT and FlanT5 (1IB) on the two REASER subtasks.
score 4: The answer is wrong but lacks minor information. Or the answer makes sense but is not the same as the golden answer.
A.5
Annotation Details
Human evaluation We give the following instruc- tion to human evaluation participants:
score The answer is correct.
"Hi, welcome to the brain teaser test. Each brain teaser has only one possible solution (none of the above is possible!). Please select the choice in the answer column. Try Think out of Box :)" Human validation We give the following instruc- tion:
Both humans and LLMs cannot perform this task well, scoring 2.64 and 2.62 on 5-point scale. Humans give up more often (18%) rather than gen- erating meaningless text like ChatGPT, making the comparison harder the task is an open-end format.
"Congratulations on passing the brain teaser test. You should notice that some brain teasers are similar each other :)! Actually, the brain teasers be divided in groups like the following: In each brain teaser group, we have original question, semantic reconstruction questions, and context reconstruction questions. Semantic reconstruction question rephrases the original question without changing the correct answer and the distractors. Context reconstruction question keeps the original reasoning path but changes both the question and the answer to describe new situational context.
A.6 Evidence of stronger distractors in the original puzzle
The barber example in Figure 1, "shaves everyday" and "keep his beard long" triggers a commonsense association that the man shaves himself every day. The contextually reconstructed puzzle of the barber example is "How can a man go to football team ev- ery day but doesn't 't play football at all?". This new question still aims to guide the model to think in the default commonsense way that "He is football player." " but the correct answer "He is coach." " is also highly probable, resulting in an inherent decrease in difficulty.
Please help with the following three tasks: Whether the original question and its answer make sense. 2)Whether the semantic reconstruc- tion question rephrases the original question. 3) Whether the context reconstruction question keeps the original reasoning path."
A.7 Fine-tuned on Riddle Sense
We finetuned RoBERTa-L on RiddleSense (Lin et al., 2021) to analyze whether being aware of linguistic creativity can enhance the model's per- formance on BRAINTEASER. We train RoBERTa- (RS) on the training data of RiddleSense in 3 epochs with a learning rate at -6- batch size at 4. RoBERTa-L (RS) reaches 59.95 on the valida- tion set, which is on par with the original paper (60.72). We then adapt Roberta-L (RS) to do the zero-shot evaluation on BRAINTEASER. The re- sults shown in Table 9.
Open-ended Human Performance We give the following instruction:
"Please write down the answer of each brain teaser. Anything that makes sense is welcome!! Also, no answer is acceptable!"
We let both humans and ChatGPT write down the most possible answer 30 context reconstruc- tion questions based on their understanding. Three experts score the answers on a scale 5, based on the following rubrics:
Even though Roberta-L (RS) already gains in- sight into creative thinking, it still struggling on B BRAINTEASER. The better results show that en- hancing creative thinking during the training may
score 0: Fail to answer.

---

Table 7: Overview of a word puzzle example and its reconstruction versions.
Adv Strategy
Question
Answers
The letter N.
What part London France?
The letter
The
None the above.
The letter
Semantic Re- construction
Which area London is inside France?
The letter O.
The letter I..
None the above.
The letter A.
Context Re- construction
What part of Korea is in China?
The letter E.
The letter R.
None of the above.
Table Main few-shot results ChatGPT F5an on two BRAINTEASER subtasks. Ori Original, Sem Semantic, Con Context. The best performance among all models is in bold.
Instance-basee
Group-hased hased
Overall
Model
Original
Semantic
Context Sentence puzzle
Ori & Sem
Ori & Sem Con
CharGPT(zeoo-satt)
60.77
59.33
67.9 .4
50.72
39.71
62.68
ChalGP((wwo-shot))
61.72
60.77
68.90
51.67
40.67
63.80
ChatGPT(four-sho))
59.33
55.98
62.20
47.85
32,06
59.17
ChaiGPT(six-shot)
60.29
59.81
66.51
51.20
40.19
62.20
ChaiGPTeiiht-shoot)
63.16
62.68
67.46
54,55
44.02
64.43
FlanT3(2ero-shot)
33.49
31.58
36.84
22.01
11.00
33.97
F5anT5(two--5)
37.80
33,49
38.76
26.79
13.40
36.68
(ffors((o
38.28
34.45
41.15
26.79
13.40
37.96
Flan (sixst(s)
38.28
34.45
41.63
27.27
13.88
38.12
-leghstcigttsshot)
38.76
33.01
41.63
26.79
14.35
37.80
Word puzzle
ChaiGPTizero-shot)
$6.10
52.44
51.83
43.90
29.27
53.46
-ha
55.49
53.66
$1.22
44.51
30.49
53.46
ChaiGPT(four-soot)
54.27
53.66
51.83
43,90
28.05
53.25
CharGPT(six-shott
56.71
51.83
54.27
45.12
28.66
54.27
ChalGPTreight-soot)
58.54
56.71
34.27
48.17
34.76
56.50
Flan (lzeoS5(2ro)
42.68
32.93
43.90
28.66
20.12
39.84
FlanTS S(twoo-hho))))
44.51
34.76
45.73
30.49
18.90
41.67
(ffurss((o))hhoot
43.29
35.98
47.56
30.49
20.73
42,28
PlanTS(ssx-sholl
44.51
36.59
47.56
29.88
17.68
42.89
(eightteig)))
45.73
33.54
46.95
27.44
16.46
42.07
be a possible solution to defying commonsense. Yet, we note that the performance this model also declines on the group-based metrics. More- over, we point out the possible data distribution overlap between BRAINTEASER and RiddleSense, RiddleSense was collected publicly from similar online websites and contains much more samples (5.7k) than BRAINTEASER.
A.8 Human Annotator Information
Our human annotators major in computer science come from East Asia, Europe and the Middle East. All annotators all fluent English.

---

Table RoBERTa-L (RS) zero-shot results over two BRAINTEASER subtasks.
Instance-bait based
Group- based
Overall
Model
Original
Semantic
Context
Ori & Sem
Ori & Sem & Con
Sentence Puzzle
ROBERTa-I(RS)
42.11
45.93
54.55
37.32
27.75
47.53
Word Puzzle
RoBERTa-L(RS)
23.78
26.22
31.10
20.73
9.76
27.03


Reviews:
[{"review_id": "dWe1tl1toP", "reviewer": "Reviewer_aqde", "summary": "", "questions": "see weaknesses.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive feedback and for investing the time to explore the content and context of our paper! Here is a list of answer together with a quick pointer to the place in the paper:\n\n### Reasonableness of the Task:  \nWe have tried an open-ended format in our initial experiment (Line 218-221) and both human and LLMs perform poorly. Moreover, as the reviews point out (and as we note in Lines 221-223), the puzzles often have multiple creative solutions. Therefore, we consider the MCQA format for a fair comparison and construct a multi-step pipeline to ensure that the task is reliable and the distractors are incorrect (Line 223-227). Similar to the performance improvement observed in your experiment, our main result shows a huge gap between humans and LLMs: the distractors won’t cause confusion to humans, while LLMs perform much worse.  Human participants in our study are chosen from diverse backgrounds to ensure a broad representation. We will provide details on the demographics and selection criteria of the human participants in the appendix.\n\n### MCQA Format and Spurious Correlations:\nBased on our comparison with an open-ended task format (Line 218-221), the MCQA format has proven to be the most reliable option for presenting the BrainTeaser task. MCQA formats are commonly used for reasoning tasks [1,2,3], including the most similar task to ours, RiddleSense [4]. Our well-motivated definition of brain teasers (Sec. 3.1) and distractor sampling strategy (Sec 3.3) ensure there exists only one reasonable answer, preventing spurious correlations or wrong reasons to our best effort. While we acknowledge that spurious correlations may remain in the data, none of our baseline methods was able to solve the task, which indicates that simple surface patterns are not sufficient.\n\n\n[1] [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://aclanthology.org/N19-1421) (Talmor et al., NAACL 2019)\n\n[2] [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://arxiv.org/abs/1803.05457) (Clark et al., arxiv 2018)\n\n[3] [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://aclanthology.org/D18-1260) (Mihaylov et al., EMNLP 2018)\n\n[4] [RiddleSense: Reasoning about Riddle Questions Featuring Linguistic Creativity and Commonsense Knowledge.](https://aclanthology.org/2021.findings-acl.131.pdf) (Lin et al.,  ACL-IJCNLP 2021)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well written, easy to follow, and fun to read. The experiments are well motivated and laid out clearly.\n\nThe paper identifies lateral thinking as a useful/relevant task to push the creative-reasoning abilities of LLMs when the key focus in this general direction has been relatively limited to Vertical Thinking puzzles. The dataset could definitely be of interest to the folks working on commonsense reasoning. The data collection procedure seems sound and the evaluation is adequate.\n\nThe results reveal a huge gap b/w human vs. LLM performance, indicating the great opportunity/need to improve lateral thinking in LLMs.", "reasons_to_reject": "In the process of reviewing this paper, I went through a number of *lateral thinking* puzzles on the internet. While this is indeed an intriguing task, solving these puzzles is a hard task. The answers to the questions are often not very obvious. Consider the following example that I found [online, [1] ](https://web.archive.org/web/20230807191758/https://parade.com/1288259/marynliles/lateral-thinking-puzzles/):\n\n*Three kids enter a room, but only two walkout. The room is empty. Where is the third kid?* --> The answer is: The third kid uses a wheelchair, so they roll out instead of walking out.\n\nAs evident, the solution goes well beyond the commonsense aspect of a statement or a human's/machine's *general* reasoning ability. This makes me rethink if the task is reasonable.\n\nTo further quench my curiosity, I asked 5 computer science PhD students to answer a few of the examples in the paper and none of them came up with the same answers as described in the paper. (Now, this changed positively after providing choices when they could see the \"path\" to the right answer more clearly. I also ACK that this not the most scientifically grounded technique to verify an intuition.) This makes me wonder how smart the human/LLM needs to be in order to answer these questions? When getting humans to answer these questions, how were they chosen? If the *average* human is not able to answer these questions, is it reasonable to assume that an LM *should* be able to answer these?\n\nI find these questions compelling when looking at Table 4 with the huge performance gap in the human vs. LLM reasoning.\n\nFinally, while using the MCQA format reduces the difficulty of the task, it increases the chances of spurious correlations or wrong reasons. Table 5 has some good examples.\n\n(While this is not my primary area of research, I have done my best to read the literature and spend time reading the paper. I'm open to be convinced that these are not major problems and that the positives of the work outweigh the negatives.)\n\n[1] https://web.archive.org/web/20230807191758/https://parade.com/1288259/marynliles/lateral-thinking-puzzles/", "questions_to_authors": "see weaknesses.", "paper_topic_and_main_contributions": "Vertical Thinking tasks involve a sequential analytical process that is based on rationality and logic, whereas a Lateral Thinking task requires a creative thinking process that involves applying a new perspective to the problem statement and defying preconceptions. For instance, the answer to *What type of cheese is made backwards?* is `Edam` which is `made` written backwards. \n\nThis paper proposes to study the ability of the state-of-the-art LLMs to reason on lateral thinking puzzles by formulating such puzzles as MCQA tasks, making them intuitive to answer by humans and easy to evaluate automatically. It further describes a novel **BrainTeaser** benchmark involving two tasks with varying granularities: Sentence Puzzles, and Word Puzzles. Authors design a data collection procedure with semi-automatic filtering. To address memorization and inconsistency, the benchmark is enriched with semantic reconstruction and context reconstruction. Results reveal a huge gap b/w human performance and current LLMs.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TsNKTsgAGQ", "reviewer": "Reviewer_c7pC", "summary": "", "questions": "A. Have you considered classifying the dataset into more nuanced question types? I'm curious to know whether the model could be benefited from few-shot examples if the given examples are more similar to test examples.\n\nB. I'm curious about the exploration regarding common sense knowledge. Rather than \"misleading commonsense knowledge\", it is possible that the mistakes are due to difficulty in recognizing negations, etc. For other instruction-tuned larger models, have you tried to ask the model to explain the reason for its answers? Do they have a good understanding of what is a brainteaser?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We genuinely appreciate your praise of our work and even more the comments and insightful suggestions! Here we list the answers in relation to your comments:\n\n### Use of RoBERTa Model for Commonsense-Related Analysis: \nWe agree that large language models are better choices in zero-shot settings, which is why we include GPT3.5, T0 and FlanT5. As a complementary baseline with clearer training data than LLMs, we study RoBERTa to focus on the impact of common sense in lateral thinking. We select RoBERTa because of its widespread usage of commonsense tasks and its impressive performance on zero-shot evaluations. RoBERTa-L (CSKG) achieve SOTA zero-shot result on multiple commonsense tasks, while CAR shares similar structures with RoBERTa-L (CSKG) and even outperforms ChatGPT on commonsense tasks.\n### Few-Shot Examples and Generalizable Patterns: \nThe goal of the few-shot setting is to introduce the model to the task at hand (lateral thinking in our case) by example. While few-shot examples can provide examples similar on the surface (Line 577-582), these are not helpful for model generalization on this subtask (Line 572-577) because brain teasers in the sentence puzzles are similar on a more abstract level. Retrieving examples based on high-level similarity patterns for overall few-shot prompting performance is non-trivial and it serves as an exciting research direction highlighted by our paper. \n### Possibility for more nuanced question types (Q.A): \nAs shown in Appendix A.1, the reasoning paths and topics within each brain teaser are too diverse to support a reasonable sample of each category. For further insight, please refer to one of the public resource websites (https://www.riddles.com/posts).\n### Exploration Regarding Commonsense Knowledge and Model Explanations (Q.B):\nOur analysis of the model explanations revealed that the model recognized an estimated 28% of the questions as brain teasers (36% for Word Puzzle, 20% for Sentence Puzzle), based on the model mentions of brain teasers (e.g., “This is a popular brain teaser...”. To study the remaining causes in a more systematic manner, we annotate 5% of the error cases with their reasons and find that 73% of the cases are led by commonsense associations. In 33% of cases, the commonsense distractors lead models to overwrite or ignore the original context in question, while in 40% of cases, models make the wrong choices and make wild imagination due to the stereotype (e.g. Denying sources of illumination implies darkness). While this analysis is not exhaustive, it provides insight into the most common reasons for the model behaviour, revealing that commonsense associations are a frequent cause, which is expected given the training objectives of LLMs. It is certainly possible that the model's associations neglect other aspects of language like negation. We will extend this evaluation with a thorough user study in follow-up work to understand the model behaviour on brain teasers through its explanations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed dataset is challenging and of good quality according to human validation, which could inspire further studies. The authors construct two adversarial versions considering the models may have seen the data, and their experiments and analyses are detailed.", "reasons_to_reject": "A. For a challenging task such as this, larger model is preferred over smaller models under zero-shot settings. While the results show that common sense knowledge has limited benefits (or even negative impact), using the RoBERTa model for commonsense-related analysis is not convincing enough to me.\n\nB. While the number of few-shot examples has no clear impact on sentence puzzles, it is possible that the given examples are very different from test examples, i.e., it is possible due to the \"not good enough\" examples that the model can hardly learn generalizable patterns.", "questions_to_authors": "A. Have you considered classifying the dataset into more nuanced question types? I'm curious to know whether the model could be benefited from few-shot examples if the given examples are more similar to test examples.\n\nB. I'm curious about the exploration regarding common sense knowledge. Rather than \"misleading commonsense knowledge\", it is possible that the mistakes are due to difficulty in recognizing negations, etc. For other instruction-tuned larger models, have you tried to ask the model to explain the reason for its answers? Do they have a good understanding of what is a brainteaser?", "paper_topic_and_main_contributions": "This paper proposes a multiple-choice question answering dataset focusing on lateral thinking (thinking outside the box) puzzles rather than vertical thinking. They design a series of procedures to collect data and pick distractors, and human efforts are involved in filtering data and picking/generating distractors for good quality.\n\nAs the puzzles are possibly in training data of models, they construct two adversarial versions, i.e., rephrasing the question, or reconstructing the puzzle to some new situational context. A subset of the dataset, including the adversarial versions, is annotated by humans which shows the dataset is of good quality.\n\nThey also test both instruction-based and commonsense models on the constructed dataset and evaluate by instance-based and group-based accuracy. They find that this dataset is very challenging for current models, models are lacking consistency across variants of the same puzzle, and commonsense knowledge and few-shot in-context learning have limited benefits.\n\n\n\n\n", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1Ccwv9B5n0", "reviewer": "Reviewer_KrFG", "summary": "", "questions": "Line 370: The intuition seems plausible, however it is not completely clear that the model is not relying on a memorized instance to solve a paraphrased task. It would be interesting to see whether there is any work that tests this approach on models where all the (pre)training data is known so that we can see whether this method definitely counteracts the problem of memorization. I am also not convinced that the term ‘adversarial’ applies to such paraphrases, as they do not have different labels from the original items. \n\nLine 530: It is not clear how the authors test the whether the models follow a reasoning path, could you elaborate on this point? ", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your recognition of the value of our research topic! We really appreciate your suggestions, and here we list the main response in relation to your comments:\n\n### Evaluation Metrics:\nAccuracy is a fair evaluation metric for the MCQA format and it has been adopted by many popular commonsense reasoning tasks [1,2,3]. As the original question, semantic reconstruction and context reconstruction have different levels of difficulty, we propose Instance-based and Group-based accuracy to test whether the model performs consistently in cascade. F1 score is not adaptable to MCQA tasks, yet, we would be happy to include F1 scores in the final paper if the reviewer can provide more detail on their idea.\n### Model Memorization (Q.A): \nWhile it is impossible to guarantee that the model is not solving a brain teaser by memorization, our reconstruction methods address memorization by testing whether the model’s performance is consistent over the group of reconstructed questions. A model understanding rather than memorizing the reasoning path of the original brain teaser should be able to answer its adversarial reconstructions with ease. The human performance shows the method is reasonable (Line 530-532), while language model performance drops significantly on the group accuracy metrics.\n### Use of the Term ‘Adversarial’ (Q.A): \nWhile we understand your consideration of the term 'adversarial' in the context of our paraphrases, we state clearly that we use the term ‘adversarial’ to refer to reconstructions of the original question (Line 354-365), we understand the reviewer's point. We will gladly replace this term with ‘reconstruction’, or another term that may be suggested by the reviewer.\n### Testing the Model's Reasoning Path (Q.B): \nWe test whether models follow a consistent reasoning path by comparing the model’s performance on instance- and group-based metrics (Line 526-530). The instance-based metric indicates the model’s understanding of single questions and the group-based metric test whether the model understand the whole group of original question and its adversarial examples. As our two adversarial strategies won’t change the original questions’ reasoning path (Line 098-105), a model should also solve two adversarial variants if it understands the original reasoning path (Line 368-372). Like humans, a consistent performance between instance-based metrics and group-based metrics can indicate the model follows a reasoning path (Line 530-532).\n\n[1] [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://aclanthology.org/N19-1421) (Talmor et al., NAACL 2019)\n\n[2] [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://arxiv.org/abs/1803.05457) (Clark et al., arxiv 2018)\n\n[3] [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://aclanthology.org/D18-1260) (Mihaylov et al., EMNLP 2018)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Interesting idea of using common-sense reasoning for distractor generation, since it is usually one of the goals of LLMs, as the authors note.\n\n- A novel way of generating distractors by using COMET for sentence puzzles\n", "reasons_to_reject": "- The evaluation metrics selected are not very reliable. Accuracy is well known to not be a good indication of good performance, especially if there is imbalance in the dataset. It would be more informative to know how the models score on other metrics such as F1. \n", "questions_to_authors": "Line 370: The intuition seems plausible, however it is not completely clear that the model is not relying on a memorized instance to solve a paraphrased task. It would be interesting to see whether there is any work that tests this approach on models where all the (pre)training data is known so that we can see whether this method definitely counteracts the problem of memorization. I am also not convinced that the term ‘adversarial’ applies to such paraphrases, as they do not have different labels from the original items. \n\nLine 530: It is not clear how the authors test the whether the models follow a reasoning path, could you elaborate on this point? ", "paper_topic_and_main_contributions": "The paper proposes a new multiple-choice question answering task, which is intended to test lateral rather than vertical reasoning. They use instruction and common-sense language models to learn to perform the task of lateral thinking. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work describes and evaluates a novel and interesting benchmark for lateral thinking composed of 1,100 puzzles accompanied by high-quality annotations.

All three reviewers agree that the soundness of the proposal is either good or strong and there is a clear consensus on the excitement being strong for all three too. To the extent possible, I would like to see the reviewers questions & concerns incorporated into the final version of the paper in order to improve its clarity. These include: further explanations on the difficulty of the task itself and the evaluation metrics chosen.
Decision: Accept-Main