SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Saeid Alavi Naeini $^{1,3,4}$  Raeid Saqur $^{2,4}$  Mozhgan Saeidi $^{2,4,6}$  John Giorgi $^{2,4,5}$  Babak Taati $^{1,2,3,4}$
$^{1}$  Kite Research Institute, Toronto Rehabilitation Institute, University Health Network
$^{2}$ Department of Computer Science, University of Toronto
$^{3}$ Institute of Biomedical Engineering, University of Toronto  $^{4}$ Vector Institute for AI
<sup>5</sup>Donnelly Centre for Cellular & Biomolecular Research, University of Toronto
$^{6}$ Department of Biomedical Data Science, Stanford University
{saeid.alavi, john.giorgi}@mail.utoronto.ca
raeidsaqur@cs.toronto.edu, mozhgans@stanford.edu, babak.taati@uhn.ca
The quest for human imitative AI has been an enduring topic in AI research since its inception. The technical evolution and emerging capabilities of the latest cohort of large language models (LLMs) have reinvigorated the subject beyond academia to the cultural zeitgeist. While recent NLP evaluation benchmark tasks test some aspects of human-imitative behavior (e.g., BIG-bench's 'human-like behavior' tasks), few, if not none, examine creative problem solving abilities. Creative problem solving in humans is a well-studied topic in cognitive neuroscience with standardized tests that predominantly use the ability to associate (heterogeneous) connections among clue words as a metric for creativity. Exposure to misleading stimuli — distractors dubbed red herrings — impede human performance in such tasks via the fixation effect and Einstellung paradigm. In cognitive neuroscience studies, such fixations are experimentally induced by pre-exposing participants to orthographically similar incorrect words to subsequent word-fragments or clues. The popular British quiz show Only Connect's Connecting Wall segment essentially mimics Mednick's Remote Associates Test (RAT) formulation with built-in, deliberate red herrings, which makes it an ideal proxy task to explore and study the fixation effect and Einstellung paradigm from cognitive neuroscience in LLMs. In this paper, we present the novel Only Connect Wall (OCW) dataset and report results from our evaluation of selected pre-trained language models and LLMs on creative problem solving tasks like grouping clue words by heterogeneous connections and identifying correct open knowledge domain connections in respective groups. We synthetically generate two additional datasets: OCW-Randomized, OCW-WordNet to further analyze our red-herrings hypothesis in language models. The code and link to the dataset are available at https://github.com/TaatiTeam/OCW.
The remarkable capabilities of state-of-the-art large language models (LLMs) [91], across a variety of domains and downstream tasks [78, 10], have spurred their comparisons with artificial general intelligence (AGI) [5, 14] and human-imitative AI [31] systems. The extraordinary leap in capabilities of these LLMs over a short span — from the advent of transformer-based [69] pre-trained, context-aware language models (PLMs) [52, 17, 40, 36, 53] circa 2018 to 2020, to the current and latest
Wall A: Season 11, Episode 23
Wall B: Season 12, Episode 27
Gala	Twelfth	Bonfire	Hen	__night	Jazz	Gala	Honeygold	Jonathan	Apples
Orlov	Churchill	Digby	Tony	Advert
Animals	Healing	Join	Greg	Show of	Can Precede
Hands
Burns	Marx	Clarke	Bender	Cigar
Smokers	Pippin	Merry	Gaffer	Sam	Hobbits
Canal Street	Castro	Chelsea	Darlinghurst	Gay Villages	Twill	Duct	Ticker	Cassette	Types of
Tape
Wall C: Season 15, Episode 10
Wall D: Season 10, Episode 2
Cameo	Fuji	Bramley	Jazz	Apples	Shrewsbury	Wellington	Ludlow	Madeley	Shropshire
Towns
Amy	Lady Bird	Dakota	Dwayne	Johnsons	Bath	Boarding	Doge	Cathode	Begin with
Animals
Thunder	Magic	Heat	Celtics	US
Basketball
Teams	Chelsea	Gum	Snow	Cowboy	Boots
Gala	Costume	Goggles	Pool	Swimming	Bolt	Bond	Churchill	Coward	English
Playwrights
Figure 1: Examples of Only Connect walls with ground-truth groupings (rows) and connections (last column). Red herrings include orthographically identical words, e.g., Gala, Churchill and Chelsea in different connected groups — Gala: Gala night, Apples, Swimming gala, Churchill: Advert Animals, English Playwrights and Chelsea: Gay Villages, Boots — across walls. In Wall A (top left), the clues Churchill, Marx, Castro provide misleading stimuli inducing plausible fixation on historical figures within the wall.
cohort of increasingly larger (billions of parameters) LMs [59, 77, 57, 89, 16, 67, 18] spearheaded by the OpenAI's GPT series [13], notably ChatGPT [49] and GPT-4 [48] — justifiably warrants such comparisons. Several natural language processing (NLP) benchmarks have been proposed to standardize the evaluation of these LLMs, including MMLU [27], BIG-bench [66], HELM [38], and Global-Bench [65]. The tasks inventory under these benchmarks are open (type of tasks) and dynamic (rolling additions). While a subset of these tasks aims to test for human imitative intelligence (e.g., nineteen tasks listed under the human-like behaviour category in BIG-bench), none tests for creative problem solving abilities [44] — a hallmark of human-like intelligence [31].
Creative problem-solving by humans is a well-studied topic in cognitive neuroscience and human behavioural sciences literature. These studies and methods use (word) associative fluency to model and test creativity objectively [44, 9]. Empirical research in this context commonly employs single or continuous word association tests that are variants of Mednick's seminal Remote Associates Test (RAT) [45]. Such tests entail finding connections or links among a presented group of words using associations that can be heterogeneous (e.g., synonymy, semantic, compounding) [86, 43]. To exemplify, consider the cue words: {Tennis, Same, Head}. A correct connection in this triplet is: Match, which connects by semantic link (tennis match), synonymy (same match), and compounding (match head). Further, the word connections can also vary in degrees of figurativeness (e.g., Star-Actress vs. Star-Planet) and abstractness (e.g., Humor-Sense vs. Apple-Tree). In humans, such creative problem-solving abilities are impeded by exposure to wrong answers [61, 62, 85] — a finding referred to as the fixation effect [34, 82]. A closely related similar concept is the Einstellung effect [42], which postulates the negative effect of previous experience when solving new problems.
Studies examining the fixation effect induce fixations by presenting clue words intended as wrong answers (misleading stimuli) [61] dubbed "red herringrs" or, by pre-exposing participants to red herrings before attempting creative problem-solving tasks like the RAT [45]. A slew of works in negative transfer learning in human cognition attempt to explain the RAT fixation phenomenon that involves pre-exposure to red herringrs by the negative effects of prior learning on indirect or implicit measures of memory [63]. This negative transfer effect was demonstrated and studied using orthographically similar words to subsequent test word fragments as red herringrs [63]. Intuitively, the red herringrs lead participants away from the memory retrieval (or down incorrect neurological pathways by Hebbian terminology) required for correct responses and fixate on wrong connections [60]. Fixation in creative problem-solving can be increased by making red herringrs more retrievable. Thus, creative problem-solving can be thought of as a type of indirect memory measure whose retrieval is degraded by red herringrs due to the negative transfer effect. The red herringr retrieval hypothesis states that factors that make red herringrs more retrievable should reduce creative problem-solving performance,
as measured with RAT problems. Two such factors are repetition and context. A following corollary states that the memory strengths of red herrings determine the magnitude of a fixation effect [8].
In this work, we study the juxtaposition of these theories from human cognitive neuroscience (fixations, negative transfer learning, red herring memory retrieval hypothesis) from the context of LLMs and natural language processing. While negative transfer learning has been observed and studied in AI research [76, 22], the context of these studies is limited to strict machine learning sub-domains like statistical distribution measures and computer vision. There has not been any work that systematically examines these specific concepts' relation in AI research. Our major contributions are as follow:
1. Only Connect Wall (OCW) Dataset and creative problem solving tasks. We introduce a novel dataset for evaluating creative problem solving tasks by curating the problems and human performance results from the popular British quiz show Only Connect [81, 3]. Specifically, the Connecting Walls segment of the show, where the tasks entail grouping sixteen (16) jumbled up clue words into four (4) connected groups, and naming the correct connections (Figure 1). The presented words have heterogeneous connections with open-domain knowledge retrieval, e.g., history, places, famous people, tools, and cultural references. These 'walls' contain red herrings or misleading stimuli by design, which makes this dataset an analogical proxy for RAT tests in evaluating LLMs for creative problem-solving. Section §2 provides a detailed description of the dataset.
2. Experiments, results, and key findings of baseline LLMs evaluation. We evaluate a suite of NLP models from static embeddings to PLMs to LLMs and demonstrate that none can solve the tasks of the OCW dataset. Our findings show that SOTA LLMs (e.g. GPT-4 [48]) perform significantly worse than the expert human baseline, and somewhat surprisingly, that increasing the number of in-context examples in few-shot in-context-learning is ineffective. Sections §3 and §4 provide details.
2 Only Connect Walls Dataset
Here we focus on the Connecting Walls segment (usually the third round) of the quiz-show. Each wall contains sixteen jumbled-up word clues that must be sorted into four groups, each with four connected words. Once the groups are formed, contestants must also identify the right connection or relationship among the items in each group. While there is only one correct solution to each wall, the puzzles are designed to include several red herring clues that can fit into another category and red herring categories fitting multiple clues. Figure 1 shows solved sample walls from the show highlighting a couple of typical red herrings.
2.1 Dataset Collection and Structure
The OCW dataset contains 618 connecting wall puzzles and solutions in total from 15 seasons of the show. Each show episode has two walls. The total number of walls per season varies based on the (varying) number of aired season episodes. The walls were scraped from fan websites<sup>1</sup>, and human performance results (for grouping and connection tasks) were manually curated by watching all the episodes. Figure 2 depicts the high-level structure of the dataset in JSON format with self-explanatory object keys and comments.
2.2 Tasks and Evaluation Metrics
The two dataset tasks: Task 1 (Grouping), and Task 2 (Connections) are identical to the quiz-show's human participant tasks. We evaluate Task 1 (Groupings) via six metrics: number of solved walls, number of correct groups (max. four per wall), Adjusted Mutual Information (AMI) [71], Adjusted Rand Index (ARI) [28], Fowlkes Mallows Score (FMS) [21], and Wasserstein Distance (WD) [54], normalized to  $(0,1)$  range, between predicted and ground-truth labels [88, 70].
We similarly evaluate Task 2 (Connections) with three metrics: exact string matching, ROUGE-1 F1 [39], and BERTScore F1 [90]. Exact match is the most strict, assigning a score of 1 when the predicted connection is identical to the ground-truth and 0 otherwise. ROUGE-1 F1 relaxes this criterion; it is large when there is a high proportion of ground-truth tokens in the model's predicted
<sup>1</sup>The primary source was the Only Connect fan website: https://ocdb.cc [6].
Predicted Connection	Ground-truth Connection	Exact Match	ROUGE-1 F1	BERTScore F1
Types of numbers	Types of numbers	1.00	1.00	1.00
Slang terms for money	Slang for money	0.00	0.86	0.79
Types of trees	Trees	0.00	0.50	0.63
Bridges in London	Thames bridges	0.00	0.40	0.31
Medieval occupations	Chaucer characters	0.00	0.00	0.15
Table 1: Examples of predicted and ground-truth connections and their performance according to the chosen metrics. Exact match is 0 for anything but identical strings. Empirically, we observe that a ROUGE-1/BERTScore F1 of  $\geq 0.5$  indicates that a predicted connection is likely correct.
connection and a low proportion of non-ground truth tokens. BERTScore F1 is similar but further relaxes this criterion, assigning a non-zero score for semantically similar (but non-identical) predicted tokens. Together these three metrics provide a more holistic view of model performance on Task 2 than any one metric alone. Empirically, we find that a ROUGE-1 or BERTScore F1 of  $\geq 0.5$  indicates that a predicted connection would likely be considered correct (Table 1). Note that BERTScore has many parameters affecting the final score; a hashing code is produced and reported for reproducibility.
Each of the evaluation metrics for Task 1 of Task 2 could be calculated per wall, per episode, per season, or for the entire test set. We present results on the entire test set in this paper (§4). We split the dataset into a train set (62 walls), validation set (62 walls), and test set (494 walls). The primary goal of our dataset is to evaluate the zero- and few-shot creative problem-solving abilities of LLMs; as such, we elect to set the size of the test set to be much greater than train or validation sets.
3 Experiments: Language Model Evaluations
This section describes methods and models used to provide baseline results for the dataset. For Task 1 (Grouping), we use clustering techniques on word-embeddings from classical and pre-trained language models (PLMs) ( $\S 3.1$ ), and few-shot in-context learning (ICL) with LLMs ( $\S 3.2$ ). For Task 2 (Connections), we only provide baseline results using few-shot ICL with LLMs ( $\S 3.2$ ).
{ // Contains a mapping of number of walls per season
"season_to_walls_map": {"1": 30, "2": 16, ...},
// Contains the actual dataset, one object per wall
"dataset": [
// Each wall has a unique identifier, season and episode number
"wall_id": "3ca8", "season": 1, "episode": 8,
// The list of 16 words or "clues" associated with the wall
"words": ["Holmes", "Indiana Jones", "Bannister", ...,],
// Ground-truth connections for each of the four groups of this wall
"gt(connections": ["Parts of a staircase", "_ cake", ...,],
// The four ground-truth groups for this wall
"groups": {
// Each group is an object with...
"group_1": {
// ...a unique ID
"group_id": "3ca8_01",
// ...ground-truth words
"gt_words": ["Newel", "Bannister", "Tread", "Riser"]
// ...and ground-truth connection
"gt_connection": "Parts of a staircase",
// Human performance is recorded as solved (1) or unsolved (0)
"human_performance": {"grouping": 1, "connection": 1}
}, ...
}
// Overall human performance for each group within the wall
"overall_human_performance": {
"grouping": [1, 1, 1, 1],
"connections": [1, 1, 1, 0],
}
}
Figure 2: JSON Structure of the OCW dataset. One truncated example is shown.
Figure 3: Solved wall (wall_id="8cde") for Task 1 (Grouping) using best performing model (E5<sub>BASE</sub>) with both static and contextual embeddings. Left: solved wall using static embeddings. Right: unsolved wall using contextual embeddings. 2D projection of embeddings using t-SNE is shown. Colors and shapes correspond to true clusters, and grey convex regions correspond to predicted clusters. The legend shows the ground truth connection for each group.
3.1 Task 1: Grouping using Word Embeddings
For the grouping task evaluation (§2.2), we use clustering algorithms on word-embeddings of the sixteen clue words in each wall, to group them into four predicted groups that are subsequently evaluated against the four ground-truth groups for each wall. A vanilla  $k$ -means (with  $k = 4$ ) clustering algorithm [25] does not guarantee each predicted group to have four words, thus we use variants like constrained clustering.
**Clustering** Semi-supervised constrained clustering [72, 7] is used when the user has pre-existing knowledge about the desired partition (in our case, 4 groups). Here, we adopt a minimum cost flow network clustering approach [12] with a cluster size of four for grouping. Our preliminary analysis showed that clustering results exhibited slight variations across runs. This slight discrepancy could be attributed to the initializations of cluster centroids. To address this issue and ensure reliable results, we report the mean and variance of results (Table 3) across sixteen (16) runs, each with a unique seed and randomized order of sixteen-word clues. We tested two additional clustering approaches motivated by [47, 19]: (1) We constructed a self-similarity matrix containing pair-wise similar information about the words prior to applying constrained clustering; (2) We performed dimensionality reduction using Principal Component Analysis (PCA) [58] and t-distributed stochastic neighbor embedding (t-SNE) [68] before applying constrained clustering. Neither approach improved performance over raw embeddings' clusters, and, for brevity, results are not included.
Static word embedding We used two well-known classic word embedding models, GloVe [51] and FastText [23], both of which are accessed through the Flair library (Table 2). We used two FastText models, one pre-trained on the Common Crawl corpus and another on Wikipedia. Approximately  $10\%$  of the total clues encountered in the dataset were out-of-vocabulary (OOV). A significant portion  $(\sim 80\%)$  of the OOV cases were addressed by mean pooling for clues comprised of multiple words to obtain one unified embedding. For the remaining OOV instances, we combined the static embeddings with BytePair encoded[26] sub-words.
PLMs We explored general-purpose PLMs (BERT [17], RoBERTa [40], DistilBERT [56], ELMo [52]) as well as Sentence Transformers (MPNet [64], E5 [75]; see Table 2). We evaluated performance with and without contextual embeddings. Depending on the context, some clues in the dataset may appear across different walls with different meanings. As an example, the word
${}^{2}$  Static embeddings are obtained from the PLMs by passing clues through the model independently.
Model	# Parameters	Version	Accessed via
Word Embeddings
BPEmb [26]	En	-	en	Flair [4]
GloVe [51]	6B	-	glove	Flair
FastText [23]	Crawl	-	crawl	Flair
News	-	news	Flair
Pre-trained Language Models (PLMs)
ELMoLARGE [52]	94M	large	Flair [4]
DistilBERTBASE [56]	uncased	66M	distilbert-base-uncased	HuggingFace [83]
BERTBASE [17]	uncased	110M	bert-base-uncased	HuggingFace
BERTLARGE	uncased	340M	bert-large-uncased	HuggingFace
RoBERTaLARGE [40]	355M	roberta-large	HuggingFace
Sentence Transformers
all-mpnetBASE [64]	V2	110M	sentence-transformers/all-mpnet-base-v2	HuggingFace
E5BASE [75]	V2	110M	intfloat/e5-base-v2	HuggingFace
E5LARGE	V2	340M	intfloat/e5-large-v2	HuggingFace
Large Language Models (LLM)
GPT-3.5-turbo	-	gpt-3.5-turbo-0301	OpenAI API
GPT-4	-	gpt-4-0314	OpenAI API
Table 2: Details about the baselines and models used in our experiments.
"Gala" was found in three distinct walls, each associated with a different meaning: apples, swimming , and _ night (Figure 1). The contextual embeddings were aimed to capture contextual semantic similarity among the clues (if any). They were generated by joining the 16 clues in the wall as a pseudo-sentence. We randomly shuffle the word order across sixteen different runs for each wall to account for the positional ordering. We note that such faux sentences (for inducing context) are not valid English syntactic sentences. We used mean pooling to generate embeddings for clues comprised of multiple words to capture the collective meaning of the entire clue.
3.2 Task 2: Connections using Few-shot In-context Learning (ICL) with LLMs
Few-shot ICL with LLMs has emerged as a performant and broadly applicable paradigm in NLP [13]. To evaluate the performance of this approach on our proposed dataset, we designed a few-shot prompt for GPT-3.5-turbo and GPT-4 [48], which are amongst the strongest performing LLMs currently available. For Task 1 (Grouping, §2.2), the prompt consists of some natural language instructions, several examples of solved walls from the training set, and the current example's 16 clues, randomly sorted. For Task 2 (Connections), in place of the 16 clues, the prompt contains a solved wall without the connections (Figure 4).
We developed our prompts on the validation set and reported the final performance on the test set. In-context examples are randomly selected from the train set; the same examples are used across all test inputs. We experiment with 0, 1, 3, 5, and 10 in-context examples. When necessary, we apply simple post-processing to the LLMs output. For example, in both Task 1 and Task 2, we take a maximum of 4 predictions for the groups and connections, respectively, and pad up to 4 with the empty string in cases where the model outputs fewer than 4. To make results as reproducible as possible, we set the temperature  $= 0$  and used the 03/01/2023 GPT-3.5-turbo snapshot and the 03/14/2023 GPT-4 snapshot. The max output length is set to 144 tokens. All other hyperparameters of the OpenAI API are left at their defaults [2]. Prompts were designed as per the Guidance library [1].
4 Results and Discussions
4.1 Task 1: Grouping Results
Embedding Clustering Techniques In Table 3 we report the performance of several static embedding baselines on Task 1 (Grouping). E5<sub>BASE</sub> was the most performant model and, on average, solved
3In preliminary experiments, we found that open-source LLMs like LLaMA [67] perform poorly and typically do not follow the task instructions.
4Please see our codebase for all post-processing steps: https://github.com/TaatiTeam/OCW
Task1(Grouping) Prompt
System Role (Natural Language Instructions)
You are currently competing in Round 3: Connecting Wall on the quiz show Only Connect. Your task is given 18 "clues" (words or phrases), solve the wall by grouping the clues into four groups of four. You will be given the clues as a list. You are also given examples of solved walls, which include the connections. Provide your answer as a list of four groups of four clues; separate groups by newlines and clues by commas. Do not try to guess the connection; only use the clues given and don't make up your own.
Be careful! Connecting Wall is deliberately difficult. The puzzles are designed to include red herring and to suggest more connections than actually exist. Some clues appear to fit into more than one category. Still, there is only one perfect solution for each wall.
User Role (In-context examples + input)
Example I
Agnew, Blofeld, Boycott, Johnston. Connection: Test Match Special regulars  
Knees, Bike, Last legs, Marks. Connection: On your __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ 
Banshees, Tory, Brock, Galore. Connection: Words originating from Irish  
Angled, Uppers, Elating, Eighth. Connection: Last letter to front = new word
Clues: Blanc, Brooks, B, Smith, Screwdriver, Hammer, Gimlet, Wrench, Sidecar, Manhattan, Gibson, Margarita, Puzzle, Business, Nuts, Suit
Solved wall:
Task 2 (Connections) Prompt
You are currently competing in Round 3: Connecting Wall on the quiz show Only Connect. Your task: given 4 groups of 4 "clues" (words or phrases), determine the connection for each group. You will be given the groups as four lists of four. You are also given examples of solved walls, which include the connections. Provide your answer by repeating the four groups and adding it after 'Connection.'
Note: Connections might be thematic, linguistic, factual, mathematical and rely on both arcane subject areas and popular culture.