Large Language Models are Fixated by Red Herrings: Exploring Creative Problem Solving and Einstellung Effect using the Only Connect Wall Dataset
Authors: Saeid Alavi Naeini
Raeid Saqur
mozhgan saeidi
John Michael Giorgi
Babak Taati
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Natural language processing
Large language models
creative problem solving
creativity
Remote Associates Test
cognitive neuroscience
artificial general intelligence
human-imitative AI
GPT
in-context learning
TL;DR: We present a novel dataset and tasks for evaluating creative problem solving in LLMs that mimic analogical cognitive neuroscience (RAT) tests. We report baseline evaluation results on a suite of PLMs, LLMs and human performance.

Abstract: The quest for human imitative AI has been an enduring topic in AI research since inception. The technical evolution and emerging capabilities of the latest cohort of large language models (LLMs) have reinvigorated the subject beyond academia to cultural zeitgeist. 
While recent NLP evaluation benchmark tasks test some aspects of human-imitative behaviour (e.g., BIG-bench's `human-like behavior' tasks), few, if not none, examine *creative problem solving* abilities. Creative problem solving in humans is a well-studied topic in cognitive neuroscience with standardized tests that predominantly use ability to associate (heterogeneous) connections among clue words as a metric for creativity. Exposure to misleading stimuli --- distractors dubbed *red herrings* --- impede human performance in such tasks via the *fixation effect* and Einstellung paradigm. In cognitive neuroscience studies, such fixations are experimentally induced by pre-exposing participants to orthographically similar incorrect words to subsequent word-fragments or clues. The popular British quiz show Only Connect's *Connecting Wall* segment essentially mimics Mednick's Remote Associates Test (RAT) formulation with built-in, deliberate red herrings, that makes it an ideal proxy dataset to explore and study fixation effect and Einstellung paradigm from cognitive neuroscience in LLMs. In addition to presenting the novel Only Connect Wall (OCW) dataset, we also report results from our evaluation of selected pre-trained language models and LLMs (including OpenAI's GPT series) on creative problem solving tasks like grouping clue words by heterogeneous connections, and identifying correct open knowledge domain connections in respective groups. We synthetically generate two additional datasets: OCW-Randomized, OCW-WordNet to further analyze our red-herrings hypothesis in language models.
The code and link to the dataset is available at [url](https://github.com/TaatiTeam/OCW).

Large Language Models are Fixated by Red Herrings: Exploring Creative Problem Solving and Einstellung Effect using the Only Connect Wall Dataset
Saeid Alavi Naeini 1,34 Raeid Saqur2.4 Mozhgan Saeidi John Giorgi Babak Taati 2,33,4
Kite Research Institute, Toronto Rehabilitation Institute, University Health Network Department of Computer Science, University of Toronto
'Institute of Biomedical Engineering, University of Toronto Veector Institute for Al Donnelly Centre for Cellular Biomolecular Research, University of Toronto DDeparmentt of Biomedical Data Science, Stanford University
alavi, john. bohn.giorgil Maargl Amail. utoronto.ca
raeidsaqurdcs. toronto.o. edu, mozhgans@stanford. edu, babak. taati@uhn.c..o...
Abstract
The quest for human imitative AI has been an enduring topic in Al research since its inception. The technical evolution and emerging capabilities of the latest cohort of large language models (LLMs) have reinvigorated the subject beyond academia to the cultural zeitgeist. While recent NLP evaluation benchmark tasks test some as- pects of numan-imitative behavior (e.g.. BIG-bench's ' 'human-like behavior' tasks), few, if not none, examine creative problem solving abilities. Creative problem solv- ing in humans is a well-studied topic cognitive neuroscience with standardized tests that predominantly use the ability to associate (heterogeneous) connections among clue words as metric for creativity. Exposure misleading stimuli distractors dubbed red herrings impede human performance in such tasks via the fixation effect and Einstellung paradigm. In cognitive neuroscience studies, such fixations are experimentally induced by pre-exposing participants to orthographi- cally similar incorrect words to subsequent word- ford-ffaagmens or clues. The popular British quiz show Only Connect's Connecting Wall segment essentially mimics Mednick(s is Remote Associates Test (RAT) formulation with built-in, deliberate red herrings, which makes it an ideal proxy task to explore and study the fixation effect and Einstellung paradigm from cognitive neuroscience in LLMs. In this paper, present the novel Only Connect Wall (OCW) dataset and report results from our evaluation of selected pre-trained language models and LLMS on creative problem solving tasks like grouping clue words by heterogeneous connections and identifying correct open knowledge domain connections in respective groups. We synthetically generate two additional datasets: OCW-Randomized, OCW-WordNeet to further analyze our red-herrings hypothesis in language models. The code and link to the dataset are available at attttss: :/ggtthh//gtttbb com coom/aatiieee Team/WN
Introduction
The remarkable capabilities of state-of-thh-ee ohethh-tthrrrrr large language models (LLMs) 191, across variety of domains and downstream tasks [0], have spurred their comparisons with artificial general intelligence (AGI) [4] and human-imitative Al [3I] systems. The extraordinary leap in capabilities of these L.Mss over short span from the advent of transformer- [69 pre-trained, context- aware language models (PLMs) 52
36 circa 2018 to 2020, to the current and latest
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Wall A: Season 11, Episode 23
Wall B: Season 12, Episode 27
Gala
Twaltth
Bontire
Hen
night
Honeygold
Jonothon
Apples
Oriov
Churchill
Digby
Tony
Advart Animals
Healing
Join
Greg
Show of
Can Precacde
Honds
Burns
Marx
Clarke
Bender
Cigar Smokers
Pippiin
Merry
Galler
Types
Canal Street
Costro
Chelsea
Donlinghurst
eay Villages
Tapa
Wall Season Episode
Wall D: Season Episode
Cameo
Fuji
Bromlay
Applis
Shrowsbury
Wellington
Ludiow
Modeley
Shropahire
Townna
Amy
Lody Bird
Dakota
Dwayne
Johnsons
Boarding
Doge
Cothode
Begin with Animals
US
Thunder
Magic
Heat
Caltics
Bascwtboll Teams
Choloon
6um
Snow
Cowboy
Boots
Galla
Costuma
Goggles
Pool
Swimming
Bolt
Bond
Churchill
Coward
Englith Playwrights
Figure 1: Examples of Only Connect walls with ground-truth groupings (rows) and connections (last column). Red herrings include orthorrapically identical words, e.g., Gala, Churchill and Chelsea in different connected groups Gala: Gala night, Apples, Swimming gala, Churchill: Advert Animals, English Playwrights and Chelsea: Gay Villages. Boots across walls. In Wall A (top left), the clues Churchill, Marx, Castro provide misleading stimuli inducing plausible fixation on historical figures within the wall.
cohort of increasingly larger (billions of parameters) LMs [59 77
67 18 spearheaded 57
89 16
justifiably warrants such comparisons. Several natural language processing (NLP) benchmarks have been proposed to standardize the evaluation of these LLMs, including MMLU (27). BIG-bench [66), HELM (38), and Global-Bench (65) The tasks inventory under these benchmarks are open (type of tasks) and dynamic (rolling additions). While a subset of these tasks aims to test for human imitative intelligence (e.g., nineteen tasks listed under the human-like behaviour category BIG-bench), none tests for creative problem solving abilities 44 a hallmark of human-like intelligence [31].
by the OpenAl's GPT series [33, notably ChatGPT 49 and GPT-4 48
Creative problem-solving by humans is well-studied topic in cognitive neuroscience and human behavioural sciences literature. These studies and methods use (word) associative fluency to model and test creativity objectively 44 9 Empirical research in this context commonly employs single or continuous word association tests that are variants of Mednick's ' seminal Remote Associates Test (RAT) 45). Such tests entail finding connections links among presented group of words using associations that can be heterogeneous (e.g., synonymy, semantic, compounding) 886 43) To exemplify, consider the cue words: /Tennis, Same, Head). correct connection this triplet is: Match, which connects by semantic link (tennis match), synonymy (same match), and compound- ing (match head). Further, the word connections can also vary in degrees of figurativeness (c.g.. Star-Actress vs. Star-Planet) and abstractness (c.g., Humor-Sense vs. Apple-Tree). In humans, such creative cobblem-solvngg abilities are impeded by exposure to wrong answers [6]
finding referred to as the fixation effect [34] 82). A closely related similar concept is the Einstellung effect (42, which postulates the negative effect of previous experience when solving new problems.
Studies examining the fixation effect induce fixations by presenting clue words intended as wrong an- swers (misleading stimuli) 61 dubbed "red herrings" or, by pre-exposing participants red herrings before attempting creative problem-solving tasks like the RAT [45). A slew of works in negative transfer learning human cognition attempt to explain the RAT fixation phenomenon that involves pre-exposure to red herrings by the negative effects of prior learning on indirect or implicit measures of memory [63] This negative transfer effect was demonstrated and studied using orthographically similar words to subsequent test word fragments red herrings (63) Intuitively, the red herrings lead participants away from the memory retrieval (or down incorrect neurological pathways by Hebbian terminology) required for correct responses and fixate on wrong connections [60]. Fixation creative problem-solving can be increased by making red herrings more retrievable. Thus, creative -ollveg-oolvig can be thought of type of indirect memory measure whose retrieval is degraded by red herrings due to the negative transfer effect. The red herring retrieval hypothesis states that factors that make red herrings more retrievable should reduce creative problem-sol solving performance,

---

as measured with RAT problems. Two such factors are repetition and context. A following corollary states that the memory strengths of red herrings determine the magnitude of fixation effect g
In this work, we study the juxtaposition these theories from human cognitive neuroscience (fixations, negative transfer learning, red herring memory retrieval hypothesis) from the context of L.LMs and natural language processing. While negative transfer learning has been observed and studied in Al research 776 (22]. the context of these studies is limited to strict machine learning sub-domains like statistical distribution measures and computer vision. There has not been any work that systematically examines these specific concepts' relation in Al research. Our major contributions are as follow:
1. Only Connect Wall (OCW) Dataset and creative problem solving tasks. We introduce a novel dataset for evaluating creative problem solving tasks by curating the problems and human performance results from the popular British quiz show Only Connect [8] 3]. Specifically, the Connecting Walls segment of the show, where the tasks entail grouping sixteen (16) jumbled up clue words into four (4) connected groups, and naming the correct connections (Figure 1). The presented words have heterogeneous connections with open-domain knowledge retrieval, e.g. history, places, famous people, tools, and cultural references. These 'walls" contain red herrings misleading stimuli by design, which makes this dataset an analogical proxy for RAT tests in evaluating LLMs for creative problem-solving. Section $2 provides detailed description of the dataset.
Experiments, results, and key findings of baseline LLMs evaluation. We evaluate suite of NLP models from static embeddings to PLMs to LLMs and demonstrate that none can solve the tasks of the OCW dataset. Our findings show that SOTA LLMs (e.g. GPT-4 [48) perform significantly worse than the expert human baseline, and somewhat surprisingly. that increasing the number of in-context examples in few-shot in-context-leamee learnnng is ineffective. Sections | and $4 provide details.
Only Connect Walls Dataset
Here we focus on the Connecting Walls segment (usually the third round) of the quiz-show. Each wall contains sixteen jumbled-up word clues that must be sorted into four groups, each with four connected words. Once the groups are formed, contestants must also identify the right connection or relationship among the items in each group. While there is only one correct solution to each wall, the puzzles are designed include several red herring clues that can fit into another category and red herring categories fitting multiple clues. Figure shows solved sample walls from the show highlighting a couple of typical red herrings.
2.1 Dataset Collection and Structure
The OCW dataset contains 618 connecting wall puzzles and solutions in total from 15 seasons of the show. Each show episode has two walls. The total number of walls per season varies based on the (varying) number of aired season episodes. The walls were scraped from fan websites]] and human performance results (for grouping and connection tasks) were manually curated by watching all the episodes. Figure 2 depicts the high-level structure of the dataset in JSON format with self-explanatory object keys and comments.
2.2 Tasks and Evaluation Metries
The two dataset tasks: Task 1 (Grouping), and Task 2 (Connections) are identical to the quiz-show's human participant tasks. We evaluate Task 1 (Groupings) via six metries: number of solved walls, number of correct groups (max. four per wall), Adjusted Mutual Information (AMI) [71]. Adjusted Rand Index (ARD) (28). Fowlkes Mallows Score (FMS) (2I), and Wasserstein Distance (WD) (54). normalized to (0, 1) range, between predicted and ground-truth labels [88 70).
We similarly evaluate Task (Connections) with three metries: exact string matching, ROUGE-1 Fl [39]. and BERTScore Fl [90]. Exact match is the most strict, assigning a score of when the predicted connection is identical to the ground- and O otherwise. ROUGE-1 F1 relaxes this criterion; it is large when there is a high proportion of ground-truth tokens in the model's predicted
'The primary source was the Only Connect fan website: https://oedb.co
3

---

Predicted Connection
Ground-truth Connection
Exact Match
ROUGE-1 FI
BERTScore Fi
Types of numbers
Types of numbers
1.00
1.00
1.00
Slang terms for money
Slang for money Trees
0.00
0.86
0.79
Types of trees
0.00
0.50
0.63
Bridges in London
Thames bridges
0.00
0.40
0.31
Medieval occupations
Chaucer characters
0.00
0.00
0.15
Table I:
Examples of predicted and ground-truth connections and their performance according the chosen metrics. Exact match is 0 for anything but identical strings. Empirically, we observe that a ROUGE-I/BERTScore Fl of 2 0.5 indicates that predicted connection is likely correct.
connection and low proportion of non-ground truth tokens. BERTScore Fl is similar but further relaxes this criterion, assigning a non-zero score for semantically similar (but non-identical) predicted tokens. Together these three metries provide more holistic view of model performance on Task than any one metric alone. Empirically, we find that a ROUGE-I or BERTScore FI of > 0.5 indicates that a predicted connection would likely be considered correct Table Tj. Note that BERTScore has many parameters affecting the final score; a hashcode is produced and reported for reproducibility.
Each of the evaluation metrics for Task 1 of Task 2 could be calculated per wall, per episode, per season, or for the entire test set. We present results on the entire test set in this paper ($4). We split the dataset into train set (62 walls), validation set (62 walls), and test set (494 walls). The primary goal of our dataset is to evaluate the zero- and few-shot creative problem-solving abilities of LLMs; as such, we elect to set the size of the test be much greater than train or validation sets.
Experiments: Language Model Evaluations
This section describes methods and models used to provide baseline results for the dataset. For Task I (Grouping). we use clustering techniques word-embeddings from classical and pre-trained language models (PLMs) ($3.1. and few-shot in-context learning (ICL) with LLMs (33.2). For Task 2 (Connections), only provide baseline results using few-shot ICL with LLMs (3..)).
t// Contains mapping of number walls per season "season_to. casoonttt_walll_seeeeeeeaaaaaaee "oo_walls_mapp' {*1"; 30, "2": 16, ...), Contains the actual dataset, one object per wall "datazet"; [[
// Each wall l has unique sdentsfier, season and episode nunder "vall_id": "3ca8", "season": 1, "episode": 8, // The 14st of 16 words or "clues" associated with the well "words": ["Holmes", "Indiana Jones", "Bannister", ...], // trouh connections for each of the four groups of this wall "gt. coscconnectios":: ["Parts of a staircase", -_-- cake", ...],
// The four ground-trus truth groups for this watt "groupe::
// Each group is object wsth...
"group_1":
// ...G unsque I0
"group_id"; "3ca8_01",
...ggronnd-ruthh words
"gt_worda": ["Newel", "Bannister", "Tread", "Riser"),
...and ground- truth connection
"t__cemecttonn: "Parts staircase",
// Hunan performance és recorded solved (1) unsolved (0)
"hunnn "human_per performance" ('grouping": 1, "connection"; 1}
)
// Overail human performance "overall, "overall_human human Saiman_perfformaecee "grouping": [1, 1, 1, 1], "connections": [1, 1, 1,
each group ws thin the wall
{
}
}...])
Figure 2: JSON Structure of the OCW dataset. One truncated example is shown.

---

Mann
Hesse
Grimm
Beaters bat
Graph
Taxis
Journalism
Synthesis
Broomstick
Journalism
Synthesis
Taxis
Shop
Bludger
Quaffle
Beaters bat
Grass
Graph
Sneak
Grimm
Snitch
Broomstick
Schiller
Hesse
Sneak
Mann
Shop
Quafile
Schiller
Group Gemmar-born
Group equipment
Smitch
Grass
Group Betray
Bludger
Photo
Figure 3: Solved wall "aall_id==Bcde"" for Task 1 (Grouping) using best performing model (ESBAsE) with both static and contextual embeddings. Left: solved wall using static em- beddings. Right: unsolved wall using contextual embeddings. 2D projection of embeddings using t-SNE shown. Colors and shapes correspond to true clusters, and grey convex regions correspond to predicted clusters. The legend shows the ground truth connection for each group.
3.1 Task 1: Grouping using Word Embeddings
clustering algorithms word-embeddings of the sixteen clue words each wall, to group them into four predicted groups that are subsequently evaluated against the four ground-truth groups for each wall. A vanilla k-means (with À 4) clustering algorithm (25) does not guarantee each predicted group to have four words, thus we use variants like constrained clustering.
For the grouping task evaluation (.2.).
Clustering Semi-supervised constrained clustering 72 is used when the user has pre-existing knowledge about the desired partition (in our case, 4 groups). Here, adopt minimum cost flow network clustering approach [12] with a cluster size of four for grouping. Our preliminary analysis showed that clustering results exhibited slight variations across runs. This slight discrepancy could be attributed to the initializations of cluster centroids. To address this issue and ensure reliable and variance of results Table 3)across sixteen (16) runs, each with
results, report the a unique seed and randomized order sixteen-word clues. We tested two additional clustering approaches motivated by 47 19 (1) We constructed self-similarity matrix containing pair-wise
similar information about the words prior to applying constrained clustering; (2) We performed dimensionality reduction using Principal Component Analysis (PCA) (58) and t-distributed stochastic neighbor embedding (I-SNE) (68) before applying constrained clustering. Neither approach improved performance over raw embeddings' clusters, and, for brevity, results are not included.
Static word embedding We used two well-known classic word embedding models, GloVe [5 and FastText (23), both of which are accessed through the Flair library (Table 2). We used two Fext models, one pre-trained on the Common Crawl corpus and another on Wikipedia. Approximately 10% of the total clues encountered in the dataset were out-of-vocabulary (OOV). significant portion (-80%) of the OOV cases were addressed by mean pooling for clues comprised of multiple words to obtain one unified embedding, For the remaining OOV instances, we combined the static embeddings with BytePair encoded (6 sub-words.
PLMs We explored general-purpose PLMs (BERT 17
RoBERTa
40
DistilBERT [56].
ELMo ) as well as Sentence Transformers (MPNet
64
E5
75
see Table 2p.
We evalu-
ated performance with and without contextual embeddings Depending on the context, some clues in the dataset may appear across different walls with different meanings. As an example, the word
Static embeddings obtained from the PLMs by passing clues through the model independently.
5

---

Model
Parameters
Version
Accessed via
Word Embeddings
BPEmb 26
En
Flair 4
GloVe 51
6B
glove
Flair
FastText
2
Crawl
craul
Flair
News
news
Flair
Pre-trained Language Models (PIMs)
ELMo; ARGE
52
94M
large
Flair f
DistilBERT
56
uncased
66M
Seetillbert-bae--ueeee
HuggingFace 88
BERTRASE
17
uncased
110M
bert-baae-uncaaed
HuggingFFace
BERT LARGE
uncased
340M
bert-large-uncased
HuggingFace
RoBERTALARGE
355M
roberta-large
llugging Face
Sentence Transformers
all-mpnetrasE
64
V2
110M
sentence weteence-traasseee www.ttttwwnnneeaae...eea.ae
HuggingFace
ES 76
110M
Intfloat/e5-based
HuggingFace
ASL
340M
intfloat/e5-large-22
HuggingFace
Large Language Models (LLM)
GPT-3.5-turbo
gpt-3.5-turbo- 5.ttburbo-0001 gpt-4-0314
OpenAl API OpenAl API
GPT-4
Table Details about the baselines and models used our experiments.
"Gala" was found in three distinct walls, each associated with a different meaning: apples, swimming and lond_nnigh night (Figure T). The contextual embeddings were aimed capture contextual semantic similarity among the clues (if any). They were generated by joining the 16 clues the wall as a pseudo-sentence. We randomly shuffle the word order across sixteen different runs for each wall to account for the positional ordering. We note that such faux sentences (for inducing context) are not valid English syntactic sentences. We used mean pooling to generate embeddings for clues comprised of multiple words to capture the collective meaning of the entire clue.
3.2 Task 2: Connections using Few-shot In-context Learning (ICL) with LLMs
Few-shot ICL with LLMs has emerged performant and broadly applicable paradigm in NLP B To evaluate the performance of this approach on our proposed dataset, we designed a few-shot prompt for
and GPT- [88] which amongst the strongest performing LLMSs currently available For Task 1 (Grouping. 22.2). the prompt consists of some natural language instructions, several examples of solved walls from the training set, and the current example's 16 clues, randomly sorted. For Task 2 (Connections), in place of the 16 clues, the prompt contains a solved wall without the connections (Figure 4).
We developed our prompts on the validation set and reported the final performance on the test set. In-context examples are randomly selected from the train set; the same examples used across all test inputs. We experiment with 0, 1, 3, 5, and 10 in-context examples. When necessary, we the L.LMs output. For example, in both Task 1 and Task 2,
apply simple post-processing take a maximum of 4 predictions for the groups and connections, respectively, and pad up to 4 with the empty string in cases where the model outputs fewer than 4 To make results reproducible as possible, we set the temperatures and used the 03/01/2023 GPT-3.5-turo snapshot and the 03/14/2023 GPT-4 snapshot. The max output length set 144 tokens. All other hyperparameters of the OpenAl API are left at their defaults 2 Prompts were designed as per the Guidance library MI]
Results and Discussions
4.1 Task I: Grouping Results
Embedding Clustering Techniques In Table 3 ding baselines Task (Grouping). ESBASE was the most performant model and, average, solved
we report the performance of several static embed-
'In preliminary experiments. we found open-source LLMs like LLaMA 67 typically do not follow the task instructions.
perform poorly and
Please codebase for all post post-processing steps: htttp:://gthabb.co///tatttttt

---

Task (Grouping) Prompt
Task (Connections) Prompt
System Role (Natural Language Instructions)
System Role (Natural Language Instructions)
currently competing Round Connecting Wall the Only Connect. Your took: glven "clues" (wards phroses), salve the wall by grouping the cluss into four groups of four. You will as a list. You are olso given exomples nolved walls, which incluree the connections. Provide your onewer as lst of four groups of tour cluse: soparate gr groups by newinee and cluos by commos. Do not try to guess the connection, only use the clues given and don't make up your own.
currently compating
Connecting groupa
paroses), determine the connection for group (words the on four lists of four walls which include the connections. Provide your answer by repeating the four groups and odding ofter "conncction"
the clues
examples solved
Note connections might be thematii,, Viguaatlo factuot mathematisal
de carafual conneeding Wall is dutibratell difficutt puzzles designed inctude nernnge ond suggest more connectinn metually
and ruly on both arcone subject creas and popular culture.
oppeor
one category.
only
User Role (in-context examples input)
Example1
User Role onn-tontxxt examples Input)
Newel Bonnister, Tread Riser, Connection: Forts of
Jotto, Excles, TBnnbuyy Chorley, Connection
Example
Forsyth, Edmondas Holmes, Parsona, Connection: Quiz show hosts Plum, Moilany, inciona Jones, Higgins Connection: Fictional professors
Agnow, Biofold, Boycott, Johnston Connection: Test Match speciarregulan Knees, fiew Lest legs Martss ts Connection: On your Borsheee, Tory, Brack, Golore, connection Words originating nating from trihh Angled, Uppers, Bloting Eighth. Connection: Last letter to front new word
Groups:
Newel, Bonnister, Tread Riser Connection Jatho, Eccles, Banbury, Chorley, Cornection:
Clues: stanc. Brooks, smith, Screwdriver, Hommer Ginlle, Wrench, Sidecar, Monhotion, olbson, Morgarita, Puzzle Business, Nuts, Bult
Forsyth, Edmonds, Holmes, Parsona, Connection Plam, Mostany, indiona Jones, Higgina Connections
Figure 4: Example prompts for Task I (Grouping) and Task 2 (Connections) used with GPT-3.5- turbo and GPT-4. The system's role includes natural language instructions. The user role includes in-context examples and the current examples 16 clues (Task 1) or the solved wall without connections (Task 2). For Task 1, the model is instructed to output the solved wall as four lines of four clues separated by commas. For Task 2, the model is instructed to copy the solved wall and fill in the connections. Emphasis and bold text are for visualization purposes only.
1/4 Groups Solved
2/4 Groups Solved
4/4 Groups Solved
Up, Down, Charm, Strange Santa, Satan, Grass, Snout Log. Puck. Bottom, Dr Riviera Elastic band, squash ball, Condom, Screw
V Lambeth, Queen Elizobeth II Millenium, London
Strudel, Knish, Bridie, Calzone Scar, Ursula, Stramboli, Hades Scratch, Nick, Harry Doll, Bird, Dame, Sheila
Chariot. Hermit, Bottle, Bell, Stocking, Spider Print, Shore, Fiddler, Velvet
Ground-truth Ground-tuuthcommecttnn connection(s))
Quarks
Ground-truth connection(s): Thomes bridges, Blue
Ground-truth connection(s). Turnovers, Disney animoted villains, 'Old" for the Devil, Nicknames for women
Figure 5: Examples of partially and fully solved walls predicted by GPT-4.
1 wall and correctly clustered 89 groups. Contextual embeddings had the lowest overall performance amongst all methods (Table 6/in Appendix $A). One explanation is that by concatenating all the clues in each wall, the resulting input may not adhere to the sentence structure that PLMs are accustomed to during training. This may disrupt the natural flow of information and, in turn, lead to less meaningful contextual embeddings. Moreover, the context may change abruptly when combining clues from different parts of the wall. This can introduce ambiguity and contextual shifts that the model may struggle to interpret accurately. Another possible explanation is the effect of positional encoding in the underlying models. Unlike other main components of PLMs, positional encoding is variant to sequence order 33 Even though the addition of positional to word embeddings helps with learning the contextual representation of words at different positions, intrinsic similarities may be more important than contextual usage. The embedding dependence on neighboring clues may have hindered the clustering process by introducing noise and capturing irrelevant information that is specific to a particular context. For instance, in In/Fiuure 3 the contextual embedding model erroneously associated the clue "Shop" with the connection "Photo "Photoshop", however, this association is incorrect as it is an example of a red herring in the wall. In contrast, the static embedding model correctly mapped "Shop" to its British slang meaning connection "Betray". Please refer to Appendix (B for more examples.
resulting in the formation of the word
Fhw-shot ICL with LLMs Performance of GPT-4 far surpassed the static Table 3) and contextual embedding baselines (Table 6). particularly in terms of the number of solved walls and correct groups (>2X the next most performant model, E5), but was still far below human performance ($4
see $ for example predictions). Examining the predictions of the best best-performing model (GPT-4,
7

---

WD +
FMS
ARI
AMI
# Solved Walls
# Correct Groups
Classic Word Embeddings
GloVe
84.9
31.5 3
14.4
17.6 +:
0
68 +:
FastText (Crawl)
84.2
32.1
13
18.4
0
80 +
Text (News)
85.5 + .5
30.4
13.0
15.8 +
0+0
62 +
Pre-trained Language Models (PLMs)
ELMO LARGE
86.3 + 6
29.5 + 3
11.8 - 4
14.5 + .4
0 0
55 +
DistilBERT
86.7 + 66
29.1 .2
11.3 1.
14.0 +
0 0
49 + 4
BERTLARGE LARGE
BERT
88.3 .5
26.5
8.2 124.0
10.3 +
0 0
33 +
BERT BASE
89.5 .4
25.1
6.4
8.1
0
22 + 2
ROBE.RTLLARGE
88.4
26.7
8.4+.3
9.4
0+0
29
Sentence Transformers
all-mpnelgAsE
29.4 +: 3
11.7
14.3 1: .5
0:0
50
EARG
84.4 + 7
32.3 + 4
1544
18.5
0
76 +
ESSASE
83.8 + 6 33.1 + +.3
16.3 4.4
19.5 4.4
1 0
89 = 6
Human Performance
285 494
1405 1976
Table 3: Results of selected models Task (Grouping) using static embeddings. WD: Wasserstein Distance. FMS: Fowlkes Mallows Score. ARI: Adjusted Rand Index. NMI: Normalized Mutual Information. Mean + standard deviation over 16 random seeds shown. Bold: best scores.
# In-context Examples
WD
FMS f
ARI
AMI
Solved Walls
# Correct Groups
GPr-b.5-turb
O-shot
82.5
34.0
18.4
21.6
0
114
1-shot
82.3
34.4
18.2
21.2
0
123
3-shot
80.9
36.8
21.3
24.7
0
140
5-shot
80.6
37.3
22.0
25.4
10-shot
81.2
36.1
20.4
24.0
137
GPT-4
0-shot
75.8
41.5
27.2
30.7
239
1-shot
73.4
43.7
29.7
33.5
262
3-shot
73.7
43.9
29.9
33.6
5-shot
72.9
43.4
29.1
32.8
269
10-shot
73.6
42.8
28.5
32.3
249
Human Performance
283
/94
1405
1976
Table 4: Results on Task 1 (Grouping) using Large Language Models. WD: Wasserstein Distance. FMS: Fowlkes Mallows Score, ARI: Adjusted Rand Index. NMI: Normalized Mutual Information. Bold: best scores.
3-shot), found common sources of error to include misformatted outputs (4.4% of all predicted groups) and hallucinated clues (6.6%).
Surprisingly, more in-context examples (from to 10 shot) did not improve performance. One possible explanation for this observation is that, due to the huge variety of possible connection types, the in-context examples' primary benefit is demonstrating the expected output format opposed to demonstrating how to perform the task which likely requires only single example. This is related to the concepts of task learning versus task recognition, which are thought to be the two distinct mechanisms through which ICL leverages demonstrations (50, 32). Many clues require open-domain, arcane, cultural and intimate knowledge of niche subject areas (e.g., "Professional snooker players", "Female Radio DJs"" that, without prior memorization, are unlikely to help. The presence of orthographically similar clue words in the in-context examples could themselves act as red herrings and plausibly induce negative transfer learning, An interesting future direction would be the evaluation of retrieval augmented models 24 groups about highly specific subject areas.
29
which may be capable of solving
4.2 Task 2: Connections Results
In Figure we present the results for Task (Connections). In general, GPT-4 outperforms GPT- 3.5-turbo, especially in the 0-shot regime. Performance for GPT-4 improves monotonically with an increasing number of in-context examples, although improvements are sometimes small (c.g..

---

GPT-3.5-turbo
GPT-4
0.4
0.3
Exact match ROUGE-1 F1
Score
0.2
BERTScore F1
0.0
0-shot
1-shot
3-shot
5-shot
10-shot
0-shot
1-shot
3-shot
5-shot
10-shot
Number of in-context examples
Figure 6: Results for Task 2 (Connections) with GPT-3.5-turde and GPT-4. For reference, hu- man performance is approximately 80% (fraction of correctly answered connections). We report max(BERTScore, 0) in the case of GPT-3.5-turbe for readability.
< 0.01). As expected, the exact match score for both models is low (< 15%). This is explained by the fact that even insignificant differences between the model's predictions and the ground truth will result in a score of 0 (e.g.. "Made of rubber" vs. "Made from rubber"), For this reason, we also report ROUGE-1 and BERTScore Fl scores (.2). Although not a perfect comparison, we can contextualize these results with human performance, which we recorded as the fraction of correctly guessed connections: 880 on the test set. The quiz show Only Connect allows for some small deviations in guessed connections that will be accepted as correct, making the comparison to ROUGE and BERTScore more suitable than to exact match. Our results suggest that at 41 445% F1, the best performance achieved with few-shot ICL (GPT-4, 10-shot) is far below human performance. Lastly, we note that a common source of model error was the inclusion of clues in the predicted connection (occurring in 8.2% of all predicted connections for the best performing model), e.g., "Fireplace tools (Spade, Brush, Poker, Tongs)", even though (1) the model was not instructed to do so, and (2) the in-context examples were not formatted like this.
"Tree of Thoughts" (87)) could mitigate these issues and improve performance. However, applying these more complicated prompting strategies to the OCW dataset is non-trivial, as they require breaking down the problem into intermediate steps, and the number nature these intermediate reasoning steps should take is unclear. We leave their application to the OCW dataset for future work.
More complicated post-processing or prompting strategies (e.g., "Chain Thought"
4.3 Effects of Red-Herrings: Additional Datasets, Experiments and Analyses
To analyze red-herring hypothesis language models, designed and performed additional ablative experiments. The original OCW dataset contains red- herrings as distractors by design. We generate two additional datasets from DCW to decrease the presence of red-herrings: Ran-Rindd and OCW-WordNet. The goals, construction and other details are presented Appendix C..
In Row-Randomized, we diluted the presence of red herrings by randomly swapping groups among the walls in the test set thus negating the inherent deliberate distractor groups in each wall. We further simplify the grouping task in OCW-WordNet by removing red herrings altogether. This is (or byponym-bypernyym) word hierarchy and synonyms in
achieved by using subordinate-superlla the English lexical database WordNet 46 20). Thus the results in Table present results datasets with decreasing proportion of red herrings from left to right, and by our hypothesis, increasing task simplicity for LLMs. The results aligned with expectations, with 3PT-3.5-turbo and GPT-4 performance increasing significantly with the reduction of red herrings from the test set.
Related Work
Various datasets and tasks have been proposed for evaluating language models against human-like linguistic capabilities. Earlier examples of such tasks include word sense disambiguation (WSD) (55).

---

OCW
OCW-NordNet
à Solved Walls
A Correct Groups
Salved Walls
f Correct Groups
# Salived Walls
# Correct Groups
GPT-3.5
337
1522
1-shot
0
123
12
315
321)
14000
3-shot
0
140
306
415
1748
5-shot
2
149
337
115
1759
10-sbot
2
137
333
428
1800
GPT-4
0-shot
6
239
595
171
1926
1-shot
4
262
57
614
501
1581
279
153T
5-shot
7
269
68
655
295
1584
10-sbot
3
219
614
378
1741
Human Performance
285/ 494
1405 1976
Table 5: Coalesced results of LLMs performance on Task 1 (grouping) using two additional test datasets NCW-Randomize and DCW-WordINg with decreasing presence of red-herrings from left to right in the walls, juxtaposed against the original OCW test set (left-most column). Only the main metrics are shown (details and full results in Appendix $C). Bold: best scores.
Winograd schema challenge 35 and word sense induction (WSI) 80 WSD aims to determine word's 's correct meaning or sense within a specific context. WSI focuses on automatically clus- tering words into different senses or semantic categories based on their contextual usage patterns. Benchmarks like GLUE 74 and SuperGLUE 773 are aimed at aggregating and standardizing these classical NLP tasks to evaluate language models. The PLMs (e.g., BERT variants) and the first generation of LLMs, mostly solved or attained human-level performance on these tasks by 2020s 41
In order to evaluate the have been proposed in recent benchmarks like BIG-bench [66] and HumanEval [13]. BIG-bench aims to address the limitations of existing benchmarks by providing a more comprehensive. open, and dynamic (tasks added tasks, including suite of tasks targeted specifically for human-like behavior. HumanEval is an evaluation set measure the functional correctness of code synthesis from docstrings (15). This benchmark includes 164 original programming problems that assess language comprehension, algorithms, and simple mathematics comparable simple software interview questions. While these recent benchmarks include wide net of complex tasks, evaluating broad range of LLM capabilities. our work here orthogonal these since none of them aims specifically measure creative problem-solving or creativity and their impediments in LLMs.
capabilities of modern LLMs, more challenging tasks
rolling basis) evaluation benchmark. It covers wide range of
6 Limitations & Future Work
As with any machine learning dataset, especially one designed to evaluate the performance of LLMs, the OCW dataset has several limitations. First, we noticed that the performance of contextual approaches can vary significantly depending on the order that clues are provided to the model. To alleviate this (and where feasible), we evaluate models across 16 random sortings of the clues. Due to cost, we did not evaluate GPT-3.5-turbo and GPT-4's sensitivity to this ordering: future work should report performance across multiple random sorts. Second, due to the nature of the quiz show Only Connect, the clues, groups, and connections in the dataset tend to be Western- (and specifically UK-) centric (e.g. "Doctor Who companions", "English cricket captains", "Irish counties"). Therefore. performance on the OCW dataset may not extrapolate to languages cultures outside of Western English. In fact, the US-centric bias of LLMs like GPT-3.5/4 [84] might partially explain their poor performance the UK-centris OCW dataset. We hope to add additional Only Connect inspired walls in multiple languages and with clues derived from various cultures & subcultures future work. Finally, given that the walls are publicly available as text on fan sites like ocdb.ec, there is always the possibility that they are included in the training sets LLMs like GPT. However, we think this is unlikely, given the low performance of publicly available datasets like our OCW from "leaking" into the training sets of LLMs remains an interesting and open problem. We have taken basic steps against this leakage by distributing our dataset in a compressed format (3)
the grouping and connection tasks. Preventing the test sets
10

---

References
[1] Microsoft guidance: A guidance language for controlling large language models. 2023.
https
[2] OpenAl API completions reference. 2023.
[3] Only Connect. Television show, 2008-2024 Created by Presentable, RDF Television and Parasol, Presented by Victoria Coren Mitchell.
[4] A. Akbik, T. Bergmann, D. Blythe, K. Rasul, S. Schweter, and R. Vollgraf. FLAIR: An easy-to-use framework for state-of-the-art NLP. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)) pages 54-59, Minncapolis, Minnesota, 2019. Association for Computational Linguistics. doi:
10.18653/11/1194400
[5] Altman. Planning for AGI and beyond. OpenAl Blog, February, 2023.
[6] C. Anotado, T. Ruddle, and Halbur. The Only Connect Database.
[7] S. Basu, I. Davidson, and K. Wagstaff. Constrained clustering: Advances in algorithms, theory, and applications. CRC Press, 2008.
[8] Z. Beda and S. M. Smith. Chasing red herrings: Memory of distractors causes fixation in creative problem solving. Memory & Cognition, 46:671-684, 2018.
(9] M. Benedek and A. Neubauer. Revisiting Mednick's model on creativity-related differences in associative hierarchies. Evidence for common path to uncommon thought. The Journal of creative behavior, 47(4):273-289, 2013.
[10] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al. On the opportunities and risks of foundation models. ArXiv preprint, Abs/2100800255 2021.
[11] S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. van den Driessche,
J. Lespiau, B. Damoc, A. Clark, D. de Las Casas, A. Guy. J. Menick, R. Ring, T. Hennigan, S. Huang, L. Maggiore, C. Jones, A. Cassirer, A. Brock, M. Paganini, G. Irving, O. Vinyals, S. Osindero, K. Simonyan, W. Rac, E. Elsen, and L. Sifre. Improving language models by retrieving from trillions of tokens. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning. ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 2206-2240. PMLR, 2022.
[12] P. S. Bradley, K. P. Bennett, and A. Demiriz. Constrained k-means clustering. Microsoft Research, Redmond, 20(0);0, 2000.
[13] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, Agarwal, A. Herbert-Voss, G. Knueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Lan- guage models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrlPS 2020, December 6- 12, 2020, virtual, 2020.
[14] S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundherg, et al. Sparks of artificial general intelligence: Early experiments with GPT-4. ArXiv preprint, abs/2303. 12712, 2023.
[15] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, al. Evaluating large language models trained on code. ArXiv preprint, abw/2107..aa. 033744 2021.

---

[16] A. Chowdbery, S. Narang. J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, al. PaLM: Scaling language modeling with pathways. 2022. ArXiv preprint, abs/2204.02311 2022.
[17] Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 4171-4186 Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 1011183/1//199 1423.
[18) N. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, B. Zoph. L. Fedus, M. P. Bosma, Z. Zhou, T. Wang, Y. E. Wang, K. Webster, M. Pellat, K. Robinson, K. S. Meier-Hellstent T. Duke, L. Dixon, K. Zhang, Q. V. Le, Y. Wu, Z. Chen, and Cui. Glam: Efficient scaling of language models with mixture-of-experts. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning. ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 5547-5569 PMLR, 2022.
[19] K. Ethayarajh. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 55-65, Hong Kong, China, 2019. Association for Computational Linguistics. doi: 10.18653/v 1/D19- 1006.
[20] C. Fellbaum. Wordnet. In Theory and applications of ontology: computer applications, pages 231-243 Springer, 2010.
[21] E. B. Fowlkes and C. L. Mallows. A method for comparing two hierarchical clusterings. Journal of the American statistical association, 78(383):553-563 1983.
[22] J. Gao, L. Ge, K. Li, H. Q. Ngo, and A. Zhang. On handling negative transfer and imbalanced distributions in multiple source transfer learning. In Proceedings of the 13th SIAM International Conference on Data Mining, May 2-4, 2013. Austin, Texas, USA, pages 261-269. SIAM, 2013. doi: 00.1137/1.97811 1972832.29.
[23] E. Grave, P. Bojanowski, P. Gupta, A. Joulin, and T. Mikolov. Learning word vectors for 157 languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, 2018. European Language Resources Association (ELRA).
[24] K. Guu, K. Lee, Z. Tung, P. Pasupat, and M. Chang. Retrieval augmented language model pre- training. In Proceedings the 37th International Conference on Machine Learning, ICMI. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 9929-3938.. PMLR, 2020.
[25] A. Hartigan and M. A. Wong. Algorithm as 136: A k-means clustering algorithm. Journal of the royal statistical society. series (applied statistics), 28(1):100-108; 1979.
[26] B. Heinzerling and M. Strube. BPEmb: 'ooeenizttion-rres pre-trained subword embeddings in 275 languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, 2018. European Language Resources Association (ELRA).
[27] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[28] L. Hubert and P. Arabie. Comparing partitions. Journal of classification, 2:193-218, 1985.
[29] G. Izacard, P. Lewis, M. Lomeli, L. Hosseini, F. Petroni, T. Schick, J. A. Yu, A. Joulin, S. Riedel, and E. Grave. Few-shot learning with retrieval augmented language models, ArXiv preprint, has/2208.03299 2022.
12

---

[30] A. Jacovi, A. Caciularu, O. Goldman, and Y. Goldberg. Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks. ArXiv preprint. abs/2305.10166 2023.
[31] M. Jordan. Artificial intelligence the revolution hasn't happened yet. Harvard Data Science Review, 1(1):1-9, 2019.
[32] J. Kaddour, J. Harris, M. Mozes, H. Bradley, R. Raileanu, and R. McHardy, Challenges and applications of large language models. ArXiv preprint, abs/2307.10169, 2023.
[33] G. Ke, D. He, and T. Liu. Rethinking positional encoding in language pre-training. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[34] N. Kohn and S. M. Smith. Partly versus completely out of your mind: Effects of incubation and distraction resolving fixation. The Journal of Creative Behavior, 43(2):102-115 118, 2009.
[35] H. Levesque, E. Davis, and Morgenstern. The winograd schema challenge. Thirteenth international conference on the principles of knowledge representation and reasoning, 2012.
[36] M. Lewis. Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871 7871-7880, Online, 2020. Association for Computational Linguistics. doi: 10.18653/v 1an5://///220...lmmmi.m7700
[37]
S. H. Lewis, E. Perez, Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. Yih, T. Rocktäschel, S. Riedel, and D. Kiela. Retrieval-augmented generation for knowledge- intensive NLP tasks. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[38] Liang, R. Bommasani, Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar, al. Holistic evaluation of language models. ArXiv preprint, abs/2211 1,09110, 10, 2022.
[39] C.-Y. Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain, 2004. Association for Computational Linguistics.
[40] Y. Liu, M. Out, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, Zettlemoyer, and V. Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv preprint, abs/1901 1692,, 2019.
[41] N. Lu, S. Liu, R. He, and K. Tang. Large language models can be guided to evade Al-generated text detection. ArXiv preprint, 0bss/3005.18444 2023.
[42] A. Luchins and E. Luchins. Rigidity of behavior: variational approach to the effect of Einstellung. 1959.
[43] M. Marko, D. Michalko, and I. Ricčanský. Remote associates test: An empirical proof of concept. Behavior research methods, 51:2700-2711, 2019.
[44] S. Mednick. The associative basis of the creative process. Psychological review, 69(3):220, 1962.
[45] A. Mednick. The remote associates test. The Journal of Creative Behavior, 1968.
[46] G. A. Miller. WordNet: A lexical database for English. In Speech and Natural Language: Proceedings of Workshop Held at Harriman, New York, February 23-26, 1992, 1992.
[47] R. Navigli. Word sense disambiguation: A survey. ACM computing surveys (CSUR), 41(2): 1-69, 2009,
[48] OpenAl. GPT-4 technical report. ArXiv preprint, abs/2303.0877.. 2023.
13

---

[49] L. Ouyang. J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin. C. Zhang, S. Agarwal, K. Slama, A. Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 55:27730-27744, 2022.
[50] J. Pan, T. Gao, H. Chen, and D. Chen. What in-context learning "learns" in-context: Disentan- gling task recognition and task learning. Annual Meeting of the Association for Computational Linguisties, 2023.
[51] J. Pennington, R. Socher, and C. Manning. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-15433 Doha, Qatar, 2014. Association for Computational Linguistics. doi: 10.3115/v 1/D11/DD11D11 1162.
[52] M. E. Peters, M. Neumann, M. lyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long Papers), pages 2227-2237, New Orleans, Louisiana, 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18- 1202.
[53] C. Raffel, N. Shazeer, A. Roberts, K. Lee. S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 11:140:1-140:66 2020.
[54] A. Ramdas. N. Gareía Trillos, and M. Cuturi. On wasserstein two-sample testing and related families of nonparametric tests. Entropy, 19(2):47, 2017.
[55] O. Sainz, O. L. de Lacalle, E. Agirre, and G. Rigau. What do language models know about word senses? zero-shot wsd with language models and domain inventories. ArXiv preprint, abs/2302.03353, 2023.
[56] V. Sanh, L. Debut, J. Chaumond, and T. Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. ArXiv preprint, abs/1910.01108 108, 2019.
[57] T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilié, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. ArXiv preprint, abs/2211, 11.2210.0110. 2022.
[58] Shlens. A tutorial on principal component analysis. arXiv:1404.1100, 2014.
[59] M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, Casper, and Catanzaro. Megatron-Im: Training multi-billion parameter language models using model parallelism. ArXiv preprint, abs/1909.0800 00055, 2019.
[60] S. M. Smith. The constraining effects of initial ideas. Group creativity: Innovation through collaboration, pages 15-31, 2003.
[61] S. M. Smith and S. E. Blankenship. Incubation effects. Bulletin of the Psychonomic Society, 27 (4):311-314, 1989.
[62] S. M. Smith and S. E. Blankenship. Incubation and the persistence of fixation in problem solving. The American journal of psychology, pages 61-87, 1991.
[63] S. M. Smith and D. R. Tindell. Memory blocks in word fragment completion caused by involuntary retrieval of orthographically related primes. Journal of Experimental Psychology: Learning, Memory, and Cognition, 23(2):355, 1997.
[64] K. Song, X. Tan, T. Qin, Lu, and T. Liu. Mpnet: Masked and permuted pre-training for language understanding. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information P Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[65] Y. Song, Cui, S. Khanuja, P. Liu, F. Faisal, A. Ostapenko, G. I. Winata, A. F. Aji, S. Cahyaw- ijaya, Y. Tsvetkov, et al. GlobalBench: benchmark for global progress natural language processing. ArXiv preprint, abs/2305. 16, 2023.
14

---

[66] A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso, al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. ArXiv preprint, abs/2206.04615. 2022.
[67] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, Azhar, al. LLaMA: Open and efficient foundation language models. ArXiv preprint, abs/2302. 13971, 2023.
[68] L. Van der Maaten and G. Hinton. Visualizing data using t-SNE. Journal of machine learning research, 9111), 2008.
[69] A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus. S. V. N. Vishwanathan. and R. Garnett, editors, Advances Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA. USA. pages 1998-6008, 2017.
[70] C. Villani. Topics in optimal transportation, volume 58, American Mathematical Soc., 2021.
[71] N. X. Vinh, J. Epps, and B. J. Information theoretic measures for clusterings comparison, volume 09. Proceedings of the 26th Annual International Conference on Machine Learning ICML, 2009.
[72] K. Wagstaff. C. Cardie, S. Rogers, and S. Schrödl. Constrained k-means clustering with background knowledge. In C. E. Brodley and A. P. Danyluk, editors, Proceedings of the Eighteenth International Conference
Machine Learning (ICMI. 2001), Williams College, Williamstown, MA, USA, June 28 July 1, 2001, pages 577-584. Morgan Kaufmann, 2001.
[73] A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, Michael, F. Hill, O. Levy, and S. R. Bowman. Superglue: A stickier benchmark for general-purposs, language understanding systems. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. Alché Buc, E. B. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 3261-3275. 2019.
[74] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, IA, USA, May 6-9, 2019. OpenReview.net, 2019.
[75] L. Wang, N. Yang, X. Huang, B. Jiao, L. Yang, D. Jiang, R. Majumder, and F. Wei. Text embeddings by weakly- supervsd contrastive pre-training. ArXiv preprint, abs/2212.03533,, 2022.
[76] Z. Wang. Z. Dai, B. Póczos, and J. G. Carbonell. Characterizing and avoiding negative transfer. In IEEE Conference Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 11293-11302 Computer Vision Foundation IEEE, 2019. doi: 10.1109/CVPR.2219001155
[77] J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le. Finetuned language models are zero-shot learners. The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[78] J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, et al. Emergent abilities of large language models. ArXiv preprint, abs/2206.07682, 2022.
[79] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, F. Chi, V. D. Zhou, et al. Chain-of- thought prompting elicits reasoning in large language models. Advances Neural Information Processing Systems, 35:24824-2464 248377 2022.
15

---

[80] L. White, R. Togneri, W. Liu, and M. Bennamoun. Finding word sense embeddings of known meaning. In Computational Linguisties and Intelligent Text Processing: 19th International Conference, CICLing 2018, Hanoi, Vietnam, March 18-24, 2018, Revised Selected Pupers, Part II, pages 3-16. Springer, 2023.
[81] Wikipedia contributors. Only connect Wikipedia, the free encyclopedia. https: //en wikipedia. werg/w/indeen line; accessed
2023. [On-
[82] J. Wiley. Expertise as mental set: The effects of domain knowledge in creative problem solving. Memory & cogmition, 26:716-7300 1998.
[83] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, Gugger, M. Drame, Q. Lhoest, and Rush. Transformers: State-of-the-as natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Prucessing: System Demonstrutions, pages 38-45, Online, 2020. Association for Computational Linguistics. doi: 10.
[84] R. Wolfe and A. Caliskan. =merican== white in multimodal language-and-image ai. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pages 800-812, 2022.
[85] G. Wood and J. Pennington. Encoding and retrieval from terg-term storage. Journal of Experimental Psychology, 99(2):243, 1973.
[86] C.-L. Wu, S.-Y. Huang, P.-Z. Chen, and H.-C. Chen. A systematic review of creativity-related studies applying the remote associates test from 2000 2019. Frontiers in psychology, 11: 573432, 2020.
[87] S. Yao, D. Yu, Zhao. I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. ArXiv preprint, abs/2305. 10601, 2023.
[88] B. Yin, M. Zhao, L. Guo, and L. Qiao. Sentence-BERT and k-means based clustering technology for scientific and technical literature. In 2023 15th International Conference Computer Research and Development (ICCRD), pages 15-20. IEEE, 2023.
[89] S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, al. OPT: Open pre-trained transformer language models. ArXiv preprint, abs/2205.0100 2022.
[90] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi Bertscore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Abuba, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[91] W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, et al. A survey large language models. ArXiv preprint, abs/2303. 18223, 2023.
16

---

Appendices
A Additional Experiments
Task 1 Grouping addition to grouping clue words using token embeddings (discussed in the main paper $ we also ran grouping the words by clustering on 'contextual" embeddings. We experimentally induce 'context" by joining the sixteen (16) word tokens (in a random order) into single pseudo-sentence, The embeddings for each token were different based on the ordering of the tokens. We repeat the random ordering sixteen times and report the mean and variance of the results obtained in T
WD 4
FMS f
ART
AMI t
# Solved Walls
# Correct Groups
ELMOLARGE
90.0 = 3
23.6 = 4
4.
5.6 7
0 0
19 +
DistiBBERT
88.4 -7
26.7 + 3
8.
10.4+
0+0
30 +
BERT BERTLARGE LARGE
87.2 -6
28.3
1.
1 1
0
46 + 5
BERT
8..
28.0 2
10.0
12
010
1
RoBERTaI,AGGE
8..4
25.9 .2
7.4
9.3 0.3:44
0+0
30 +
all-mpneIBASE
87.6
28.0
10.0
12.4
04:
38 +
ESLARGE
87.7 5
28.1
14
12.7 55
0
37 :
ESBASE
87.2 = 3
28.2
10
12.5 .4
0:0
46 + 5
Human Performace
285 //9
1405 1976
Table 6: Results of selected models on Task (Grouping) using contextual embeddings. WD: Wasserstein Distance. FMS: Fowlkes Mallows Score. ARI: Adjusted Rand Index. NMI: Normalized Mutual Information. Mean = standard deviation over 16 random seeds is shown. Bold: best scores.
Task Connections In addition to prompting based results on GPT-4 (discussed in $4). we ran experiments additional LLMSs like LLaMa (67] (7B, 13B) using pre-trained configuration weights obtained by permission from Meta AI. However, without additional fine-tuning on the specific task, these LLMs were unable solve the task in meaningful manner. To elucidate, LLaMa generated a bunch of hallucinated words with unequal group sizes. We omit these unintelligible results for brevity.
17

---

B Additional Figures
In this section, we provide additional t-SNE projections of embeddings from various methods used.
Grief
Wheat
Spud
Dusty
White
Gracious
Root
Chalky
God
leavens
Waldheim
Ginger
Nobby
Lord
Lauda
Schwarzenegger
Shea
Arch
Group
Laver
Popper
Group 2: "Good
Klimt
Group Fasteners
Group 4: People who to sports
Beam
Interval
Suspension
Swing
Group
Zip
Group Types
Buckle
Cadence
Triad
Group harmany
Toggle
Pontoon
Group Bridges
Button
Figure 7: Solved wall for Task (Grouping) using GloVe. Left: (wall_id="7ed3",, the embedding model erroneously associated the clue "Suspension" with the connection "Bridges"; however, this association is example of red herring. "Suspension" is "a lerm used in musicul harmony" in this context. Right: "hall_id=55ee<"" shows that clue "Lord" is close to "God, Heavens, and Grief" in the embedding space, which matches the "Good _ connection. However, this is another example of a red herring as, this context, "Lord" refers to "Lord's cricket Ground", a cricket stadium named after "Thomas Lord""
Watts
Group Birds
Group 1: Patt oC's loum
Heddle
Treadle
Group Hymm writers
Newlon
Group Apple dEches
Group National coronation
Alexander
Group Hoooophnnes Greek letters
Group 4 Variant forms of Elizabeth
Wesley
Group &; Star Trek techmolugics
Strudel
Marsh
Willow
Beater
Pic
Liz Bethan
Capper
Cobbler
Blue
Turnover
Crumble
Bearded
Row
Buffy
Elspeth
Warp beam
Shuttle
Scone
Reims
Tricorder
Krakow
Hypospray
Phaser
Transporter
Figure 8: Solved wall for Task (Grouping) using FastText (Crawl), Left: (vall_id="d5edf the embedding model erroneously associated the clue "Tara" other girls' names; but here, "Tara" is short for "Hill of Tara" and belongs to the "national coronation sites" group. Right: (vall_id="4c23) shows that clue "Pie" associated with the connection "Apple". Even though is acceptable in general context, here it represents a homophone for the Greek letter "x".
18

---

Group I: The Monkees
Stoen
Group 1. The Markees
Group 2: Dutch puinters
Rembrandt
pointers
Group Lamps
Group 3: Lampa
Mickey
Group 4 Toothgasie beands
Mondrian
V Vermeer
Group Tootlpuee
Peter
Mike
Davy
Bosch
Stoen
Lava
Crest
Fog
Plasma
Bosch
Crest
Davy
Peter
Signal
Signal
Punch and Judy
Punch and Judy
Mike
Mickey
Rembrandt
Hurricanc
Hurricane
Vermeer
Plasma
Mondrian
Fog
Figure Solved wall (wall_id="2d8t for Task 1 (Grouping) using LERTLARGE with both static and contextual embeddings. Left: contextual embedding solved 3/4 groups. Here the clue "Rambrandi" placed near other Dutch painters. The correct grouping for this clue in this wall "Toothpaste Brands". Right: static embedding solved 0/4 groups
19

---

C Effects of Red-Herrings: Additional Experiments, Analysis and Results
C.1 Additional Datasets
Both of the additional datasets described in this section for ablation experiments have been made available via our code repositories on Github and HuggingFace
C.1.1 OCW-Randomized Dataset
This test dataset generates a version of the test set where red herrings are removed or largely reduced in frequency. This is achieved by rebuilding every wall using randomly selected group from different walls. We only applied the process to the (original OCW) test set, the train and validation sets are left untouched.
Method For each wall in the existing test set, leave the first group untouched, and sample three new groups, each from a different wall, such that none of the groups share a word in common. The connections for each group unmodified. The result wall is composed of random groups from different walls.
version of the test set where every
C.1.2 OCW-WordNet Dataset
is a large lexical database of English. Nouns, verbs, adjectives, and adverbs are grouped into sets of cognitive synonyms (synsets), each expressing a distinct concept. We use the hypernym/hyponym (or superlaiive/sbbordinatvee hierarchical lexical structure aggregated in WordNet to generate an easy test set to further analyze the effects of red-herring in OCW.
WordNet 46 20
Method We the existing words in wall select synonyms from the word's synsets. We only consider synsets that have at least five synonymous lexical names, then randomly sample four words. The original test set word and its definition (ss. iaffinitinn ()) subsequently becomes the connection phrase for the group. Four groups were generated for each wall, and the easy wall generation process was repeated for the total number of walls (494) in the original test data
For the group connections, we concatenate the superlative parent word with a synset definition giving description of the word. This allows for an ideal semantic similarity score be calculated using BERTScore. For a few cases (approx. 70/494 walls in the test set), the number of generated groups per wall is less than four, due the unavailability of direct synonyms from word synsets. In those edge cases, we generate and append groups using common hypernym words like animal, mammal, furniture,
ensure wall is valid with four groups
sample generated easy group is shown below, where we prefix the group id from the original OCW dataset with 'casy" to aid with mapping identification.
"group_3": {
"group_idt" id": "easy_691a_3 "gt_words": ["gibe", "shaft", "jibe", "barb"",
"gt "gt__connetioo".: "Shaft: an aggressive remark directed at a person like missile and intended to have a telling effect"
Further, we generate easy to train and validation sets mimicking the original dataset, package and release these three additional easy sets, OCW-WordNet as added contributions.
20

---

C.2 Results of Ablation Experiments
C.2.1 PLMs: Performance on Task 1 (Grouping)
We perform and present the results using 'static' embeddings due the noted superior results and the word order related deficiency already shown by using contextual embeddings pertinent to our task setup.
WD 4
FMS
ARI
AMI
# Solved Walls
Correct Groups
Claxsic Word Embeddings
GloVe FastText (Crawl)
+
39.2
24.0
27.7 +
7
213 +
40.5
25.0
28.6 =
13 5+1
236
FastText (News)
79.3 +
36.8
23
21.5 +
176 +
Pre-trained Longuage Models (PL
EL ARGER
80.9 +
35.2 3
18.9 3
22.2 + A
3
154 +
DistilBERTBAS BASE
82.3
34.2 = 4
21.1 +
141
124 +
BERT
86.2 + .4
29.2 = 3
11.5 3
14.2 1
0
66 4
BERT
87.5 A
27.7 = 3
9.6 66
11.8 +
0+0
48
RoBERTaLARC
86.7
28.6 + .2
10.8 +
13.4
1+0
56
Sentence Transformers
All-mpne
81.
35.1
18.9 .5
22.0 + 66
8
154 +
ESLARG
76.0
40.7
25.9
29.7
8
230 +
ES
8
41.8
27.2 +
31.1 1.
8
249
Human Performance
Table 7: Results of OCW-Randomized using static embeddings. WD: Wasserstein Distance. FMS: Fowlkes Mallows Score. ARI: Adjusted Rand Index. NMI: Normalized Mutual Information. Mean + standard deviation over 16 random seeds is shown. Bold: best scores.
WD L
FMS
ARI
AMI
# Solved Walls
Correct Groups
Classic Word Embeddings
GloVe
43.0 = 1.0
66.1 + .4
37.4 55
60.9 +
118 3
886 +
FastText (Crawl)
30.6 = 1.0
75.8
69.6
72.4
195 6
1173 = 18
FastText (News)
44.9 + 1.2
64.9 55.9 .6
59.5
105 3
844
Pre-troined Language Models (PLMs)
EL ARGE
52.5
58.9
48.2 .4
52.5
682 +
DistilBERTRRa BASE
45.5 1.0
64.1
55.0
58.7
105 33
BERT LARGE
76.9
3..
23
27.5
7
197 +
BERT BASE
73.0 - 1.3
4
27.9 6
32.5 6
8
268 = 12
ROBERTaLARGE
57.4 13
54.833
43.3 = 3
47.5 3
48
573 1
Sentence Transformers
all-mpnetbASE
22.6 +: 7
81.9 + 4
77.1 + 5
79.4 + .4
256 + 4
1365 + 12
E5LARGE
23.6 + 8
80.9 + 4
75.9 +
78.3 + 4
250 + 4
1347 = 12
E5BBSE
26.9 + 9
78.0 + 72.3
75.0 15.05.
221 +
1259 + 10
Human Performance
Table 8: Results of OCW-WordNee using static embeddings. WD: Wasserstein Distance. FMS: Fowlkes Mallows Score. ARI: Adjusted Rand Index. NMI: Normalized Mutual Information. Mean standard deviation over random seeds is shown. Bold: best scores.
C.2.2 LLMs: Performance Task (Grouping) using GPT3.5/4
Here present the results of repeating Task (grouping) the ablation datasets OCW- Randomized (C.I.I) and DCW-Wordnet (C.1.2) to analyze the effects of red-herrings in walls on LLM performance.
The results adhere to the expected results of superior performance with the dilution/removal of red-hec red-herrings from the walls.
21

---

In-context Examples
WD
FMS
ARI
AMI
Solved Walls
# Correct Groups
GPT-3.5-turb
O-shot
74.3
40.4
26.4
29.8
1-shot
72.0
43.1
29.0
32.3
315
3-shot
72.7
43.4
29.4
32.9
10
306
5-shot
70.7
44.6
30.9
34.4
16
337
10-shot
70.5
43.8
30.0
33.5
17
333
GPT-4
0-shot
58.2
56.2
45.4
48.8
59
595
1-shot
55.1
58.0
47.5
51.0
57
644
3-shot
55.0
37.5
46.9
30.3
62
649
5-shot
54.1
58.0
47.5
50.9
68
655
10-shot
56.6
56.1
45.1
48.5
Human Performance
Table 9: Results of OCW-Randomized using Large Language Models. WD: Wasserstein Distance. FMS: Fowlkes Mallows Score. ARI: Adjusted Rand Index. NMI: Normalized Mutual Information. Bold: best scores.
In-context Examples
WD
FMS f
ART
AMT
Salved Walls
# Correct Groups
GPT-3.5-turo
0-shot
15.9
86.3
83.4
84.9
337
1522
1-shot
24.8
76.4
74.4
75.4
320
1400
3-shot
8.65
92.7
91.2
415
1748
5-shot
8.09
94.0
92.4
93.1
415
1759
10-shot
6.55
95.3
94.0
94.7
428
1800
GPT-4
0-shot
1.51
98.5
98.0
98.2
471
1926
1-shot
19.2
87.9
84.3
83.7
304
1581
3-shot
21.5
86.6
82.5
81.8
279
1537
5-shot
19.1
88.1
84.5
83.8
298
1584
10-shot
11.2
92.9
90.7
90.4
378
1742
Human Performance
Table 10: Results of OCW-WordNer using Large Language Models. WD: Wasserstein Distance. FMS: Fowlkes Mallows Score. ARI: Adjusted Rand Index. NMI: Normalized Mutual Information. Bold: best scores.
22


Reviews:
[{"review_id": "teXyLhVpMi", "reviewer": "Reviewer_fDpy", "summary": "", "questions": "", "limitations": "authors note limitations of variance (for contextual embeddings) and language (current data based on UK show), see more above.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "- the cognitive motivation to study creative problem solving (, fixation effect, Einstellung paradigm, etc.) is very interesting and reasonable, and novel in the NLP community. \n\n- experiments (baseline design, task setup, metrics, results) are generally reasonable, clear,  comprehensive, and helpful for future research in this direction. some concrete findings/insights:\n1. contextual embeddings do not perform well for RAT-style tasks, due to e.g. format mismatch \n2. more in-context examples do not improve performance for such \"creative\" tasks\n3. still space for improvement to humans", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "cQbwXNoyer", "reviewer": "Reviewer_zJGy", "summary": "", "questions": "", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The task is interesting, and evaluations on the state of models show that they are challenging. Therefore it should give headroom to improve the capabilities needed for these sorts of tasks.\n\nThe evaluations are comprehensive and easy to follow. I especially appreciate the inclusion of more classic methods using clustering and embeddings, instead of pure prompting.\n\nThe framing of the red herring is very interesting and thought provoking.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "Xc8AvyOYdt", "reviewer": "Reviewer_H46b", "summary": "", "questions": "", "limitations": "See opportunities for improvement.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. Propose a challenging task, word grouping with heterogeneous connections, that requires creative problem solving and association and build the evaluation dataset.\n2. Provide various baseline results including clustering algorithms and in-context learning of LLMs, along with the human performance baseline.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "EOon3nAG3W", "reviewer": "Reviewer_9y7c", "summary": "", "questions": "", "limitations": "There are two limitations as previously alluded to:\n\n1. LLMs might exhibit USA-centric bias, and Western-bias in general. To the authors' credit, they have acknowledged this, as:\n> Therefore, performance on the OCW dataset may not extrapolate to language or subcultures outside of Western English. We hope to add additional Only Connect inspired walls in multiple languages and with clues derived from various subcultures in future work.\n\nAlso, this may pose an additional [technical] limitation: per Wolfe & Caliskan (2022) (https://arxiv.org/pdf/2207.00691.pdf) an LLM which could hypothetically contain implicit bias towards, say, US-based content, might not translate well to performance on a UK-based quiz show. The authors might need to expand on this a bit more.\n\n* also a side note: the authors might want to add \"**cultures**\" to \"...extrapolate to language or subcultures\" -- it is not only subcultures (i.e. as defined by the Cambridge Dictionary as \"...group of people within a society that are different from the rest of that society\"), but also different cultures more broadly.\n\n2. The JSON file is, in the main, self-documenting. However, there might be the need for additional information, as not all are readily clear especially to a non-OC viewer (disclaimer: this reviewer watches OC quite regularly but had to do a double-take when studying the JSON).\n* For example: describing the flow of OC's CW round -> grouping attempts -> completion OR timeout OR 3-attempts-lost -> connection determination.\n* In addition, the case of: {timeout OR 3-attempts-lost scenario AND successful connections post-grouping by the OC's presenter} might need to be treated differently to a case of {a team not knowing the grouping at all AND failed connections even after reveal due to a lack of, say, domain expertise on an esoteric topic}\n* It is beneficial to add the year (month?) to the season data, as it allows one to gauge the performance of, say, different releases of LLMs with respect to the chronological context of a set of clues (in terms of season).", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The authors have done an impressive amount of work augmenting an existing dataset (ocdb.cc - i.e. Anotado, Ruddle & Halbur (2023)) with other data from watching the actual show itself to fill in incomplete gaps. The authors have also managed to operationalise an LLM-based and an embedding-based clustering benchmark on a structured RAT problem.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper creates a new task/dataset for creative problem solving. It builds an evaluation dataset for the task from the quiz show Only Connect. Various baselines evaluation results are reported, including clustering algorithms based on static and contextualized word embeddings and in-context learning based on LLMs, along with human performance results from the quiz show. The results show that SOTA LLMs perform significantly worse than human performance.

Pros:
1) The task to study creative problem solving is very interesting and reasonable, and novel.
2) The evaluations are generally reasonable, clear, comprehensive, and helpful for future research in this direction, providing various baseline results including clustering algorithms and in-context learning of LLMs, along with the human performance baseline.

Cons:
1) The framing of the fixation effect is not sufficiently supported by empirical evidence. Better evaluation around fixation effect & Einstellung paradigm are necessary.
2) The paper should provide analysis of influence of red herrings on the model performance.

In summary, all the four reviewers agreed that this paper is above acceptance threshold and authors also handled concerns from reviewers during discussion period. I recommend acceptance.
Decision: Accept (Poster)