SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Newel, Bannister, Tread, Riser. Connection: Parts of a staircase  
Jaffa, Eccles, Banbury, Chorley. Connection: cake  
Forsyth, Edmonds, Holmes, Parsons. Connection: Quiz show hosts  
Plum, Morialty, Indiana Jones, Higgins. Connection: Fictional professors
Groups:
Newel, Bannister, Tread, Riser. Connection: Jaffa, Eccles, Banbury, Chorley. Connection: Forsyth, Edmonds, Holmes, Parsons. Connection: Plum, Moriarty, Indiana Jones, Higgins. Connection:
Figure 4: Example prompts for Task 1 (Grouping) and Task 2 (Connections) used with GPT-3.5-turbo and GPT-4. The system's role includes natural language instructions. The user role includes  $n$  in-context examples and the current examples 16 clues (Task 1) or the solved wall without connections (Task 2). For Task 1, the model is instructed to output the solved wall as four lines of four clues separated by commas. For Task 2, the model is instructed to copy the solved wall and fill in the connections. Emphasis and bold text are for visualization purposes only.
1/4 Groups Solved
Up, Down, Charm, Strange Santa, Satan, Grass, Snout Lag, Puck, Bottom, Dr Riviera Elastic band, Squash ball, Condom, Screw
2/4 Groups Solved
Lambeth, Queen Elizabeth II, Millennium, London  
Chariot, Moon, Hermit, Tower  
Bottle, Bell, Stocking, Spider  
Print, Shore, Fiddler, Velvet
4/4 Groups Solved
Strudel, Knish, Bridie, Calzone  
Scar, Ursula, Stromboli, Hades  
Ned, Scratch, Nick, Harry  
Doll, Bird, Dame, Sheila
Ground-truth connection(s): Quarks
Ground-truth connection(s): Thames bridges, Blue
Ground-truth connection(s): Turnovers, Disney animated villains, "Old" names for the Devil, Nicknames for women
Figure 5: Examples of partially and fully solved walls predicted by GPT-4.
1 wall and correctly clustered 89 groups. Contextual embeddings had the lowest overall performance amongst all methods (Table 6 in Appendix §A). One explanation is that by concatenating all the clues in each wall, the resulting input may not adhere to the sentence structure that PLMs are accustomed to during training. This may disrupt the natural flow of information and, in turn, lead to less meaningful contextual embeddings. Moreover, the context may change abruptly when combining clues from different parts of the wall. This can introduce ambiguity and contextual shifts that the model may struggle to interpret accurately. Another possible explanation is the effect of positional encoding in the underlying models. Unlike other main components of PLMs, positional encoding is variant to sequence order [33]. Even though the addition of positional to word embeddings helps with learning the contextual representation of words at different positions, intrinsic similarities may be more important than contextual usage. The embedding dependence on neighboring clues may have hindered the clustering process by introducing noise and capturing irrelevant information that is specific to a particular context. For instance, in Figure 3, the contextual embedding model erroneously associated the clue "Shop" with the connection "Photo ______", resulting in the formation of the word "Photoshop"; however, this association is incorrect as it is an example of a red herring in the wall. In contrast, the static embedding model correctly mapped "Shop" to its British slang meaning connection "Betray". Please refer to Appendix §B for more examples.
Few-shot ICL with LLMs Performance of GPT-4 far surpassed the static (Table 3) and contextual embedding baselines (Table 6), particularly in terms of the number of solved walls and correct groups ( $>2\mathrm{X}$  the next most performant model, E5), but was still far below human performance ( $\S 4$ ; see  $\S 5$  for example predictions). Examining the predictions of the best-performing model (GPT-4,
WD ↓	FMS ↑	ARI ↑	AMI ↑	# Solved Walls	# Correct Groups
Classic Word Embeddings
GloVe	84.9 ± .4	31.5 ± .3	14.4 ± .3	17.6 ± .4	0 ± 0	68 ± 4
FastText (Crawl)	84.2 ± .5	32.1 ± .3	15.2 ± .3	18.4 ± .4	0 ± 0	80 ± 4
FastText (News)	85.5 ± .5	30.4 ± .2	13.0 ± .2	15.8 ± .3	0 ± 0	62 ± 3
Pre-trained Language Models (PLMs)
ELMoLARGE	86.3 ± .6	29.5 ± .3	11.8 ± .4	14.5 ± .4	0 ± 0	55 ± 4
DistilBERTBASE	86.7 ± .6	29.1 ± .2	11.3 ± .3	14.0 ± .3	0 ± 0	49 ± 4
BERTLARGE	88.3 ± .5	26.5 ± .2	8.2 ± .3	10.3 ± .3	0 ± 0	33 ± 2
BERTBASE	89.5 ± .4	25.1 ± .2	6.4 ± .3	8.1 ± .4	0 ± 0	22 ± 2
RoBERTaLARGE	88.4 ± .4	26.7 ± .2	8.4 ± .3	9.4 ± .4	0 ± 0	29 ± 3
Sentence Transformers
all-mpnetBASE	86.3 ± .4	29.4 ± .3	11.7 ± .4	14.3 ± .5	0 ± 0	50 ± 4
E5LARGE	84.4 ± .7	32.3 ± .4	15.4 ± .5	18.5 ± .6	0 ± 0	76 ± 5
E5BASE	83.8 ± .6	33.1 ± .3	16.3 ± .4	19.5 ± .4	1 ± 0	89 ± 6
Human Performance	-	-	-	-	285 / 494	1405 / 1976
Table 3: Results of selected models on Task 1 (Grouping) using static embeddings. WD: Wasserstein Distance. FMS: Fowlkes Mallows Score. ARI: Adjusted Rand Index. NMI: Normalized Mutual Information. Mean ± standard deviation over 16 random seeds is shown. Bold: best scores.
# In-context Examples	WD ↓	FMS ↑	ARI ↑	AMI ↑	# Solved Walls	# Correct Groups
GPT-3.5-turbo	0-shot	82.5	34.0	18.4	21.6	0	114
1-shot	82.3	34.4	18.2	21.2	0	123
3-shot	80.9	36.8	21.3	24.7	0	140
5-shot	80.6	37.3	22.0	25.4	2	149
10-shot	81.2	36.1	20.4	24.0	2	137
GPT-4	0-shot	75.8	41.5	27.2	30.7	6	239
1-shot	73.4	43.7	29.7	33.5	4	262
3-shot	73.7	43.9	29.9	33.6	5	272
5-shot	72.9	43.4	29.1	32.8	7	269
10-shot	73.6	42.8	28.5	32.3	3	249
Human Performance	-	-	-	-	285 / 494	1405 / 1976
Table 4: Results on Task 1 (Grouping) using Large Language Models. WD: Wasserstein Distance. FMS: Fowlkes Mallows Score. ARI: Adjusted Rand Index. NMI: Normalized Mutual Information. Bold: best scores.
3-shot), we found common sources of error to include misformatted outputs (4.4% of all predicted groups) and hallucinated clues (6.6%).
Surprisingly, more in-context examples (from 1 to 10 shot) did not improve performance. One possible explanation for this observation is that, due to the huge variety of possible connection types, the in-context examples' primary benefit is demonstrating the expected output format - as opposed to demonstrating how to perform the task - which likely requires only a single example. This is related to the concepts of task learning versus task recognition, which are thought to be the two distinct mechanisms through which ICL leverages demonstrations [50, 32]. Many clues require open-domain, arcane, cultural and intimate knowledge of niche subject areas (e.g., "Professional snooker players", "Female Radio 1 DJs") that, without prior memorization, are unlikely to help. The presence of orthographically similar clue words in the in-context examples could themselves act as red herrings and plausibly induce negative transfer learning. An interesting future direction would be the evaluation of retrieval augmented models [24, 37, 11, 29], which may be capable of solving groups about highly specific subject areas.
4.2 Task 2: Connections Results
In Figure 6, we present the results for Task 2 (Connections). In general, GPT-4 outperforms GPT-3.5-turbo, especially in the 0-shot regime. Performance for GPT-4 improves monotonically with an increasing number of in-context examples, although improvements are sometimes small (e.g.,
Figure 6: Results for Task 2 (Connections) with GPT-3.5-turbo and GPT-4. For reference, human performance is approximately  $80\%$  (fraction of correctly answered connections). We report  $\max (\text{BERTScore}, 0)$  in the case of GPT-3.5-turbo for readability.
$< 0.01$ ). As expected, the exact match score for both models is low ( $< 15\%$ ). This is explained by the fact that even insignificant differences between the model's predictions and the ground truth will result in a score of 0 (e.g., "Made of rubber" vs. "Made from rubber"). For this reason, we also report ROUGE-1 and BERTScore F1 scores ( $\S 2.2$ ). Although not a perfect comparison, we can contextualize these results with human performance, which we recorded as the fraction of correctly guessed connections:  $\sim 80\%$  on the test set. The quiz show Only Connect allows for some small deviations in guessed connections that will be accepted as correct, making the comparison to ROUGE and BERTScore more suitable than to exact match. Our results suggest that at  $41 - 45\%$  F1, the best performance achieved with few-shot ICL (GPT-4, 10-shot) is far below human performance. Lastly, we note that a common source of model error was the inclusion of clues in the predicted connection (occurring in  $8.2\%$  of all predicted connections for the best performing model), e.g., "Fireplace tools (Spade, Brush, Poker, Tongs)", even though (1) the model was not instructed to do so, and (2) the in-context examples were not formatted like this.
More complicated post-processing or prompting strategies (e.g., "Chain of Thought" [79], "Tree of Thoughts" [87]) could mitigate these issues and improve performance. However, applying these more complicated prompting strategies to the OCW dataset is non-trivial, as they require breaking down the problem into intermediate steps, and the number or nature these intermediate reasoning steps should take is unclear. We leave their application to the OCW dataset for future work.
4.3 Effects of Red-Herrings: Additional Datasets, Experiments and Analyses
To analyze our red-herring hypothesis on language models, we designed and performed additional ablative experiments. The original OCW dataset contains red-herrings as distractors by design. We generate two additional datasets from OCW to decrease the presence of red-herrings: OCW-Randomized and OCW-WordNet. The goals, construction and other details are presented in Appendix §C.1.
In OCW-Randomized, we diluted the presence of red herring by randomly swapping groups among the walls in the test set – thus negating the inherent deliberate distractor groups in each wall. We further simplify the grouping task in OCW-WordNet by removing red herring altogether. This is achieved by using subordinate-superlative (or hyponym-hypernym) word hierarchy and synonyms in the English lexical database WordNet [46, 20]. Thus the results in Table 5 present results on datasets with a decreasing proportion of red herring from left to right, and by our hypothesis, increasing task simplicity for LLMs. The results are aligned with our expectations, with GPT-3.5-turbo and GPT-4 performance increasing significantly with the reduction of red herring from the test set.
Various datasets and tasks have been proposed for evaluating language models against human-like linguistic capabilities. Earlier examples of such tasks include word sense disambiguation (WSD) [55],
OCW	OCW-Randomized	OCW-WordNet
# Solved Walls	# Correct Groups	# Solved Walls	# Correct Groups	# Solved Walls	# Correct Groups
GPT-3.5-turbo	0-shot	0	114	5	274	337	1522
1-shot	0	123	12	315	320	1400
3-shot	0	140	10	306	415	1748
5-shot	2	149	16	337	415	1759
10-shot	2	137	17	333	428	1800
GPT-4	0-shot	6	239	59	595	471	1926
1-shot	4	262	57	644	304	1581
3-shot	5	272	62	649	279	1537
5-shot	7	269	68	655	298	1584
10-shot	3	249	55	614	378	1742
Human Performance	285 / 494	1405 / 1976	-	-	-	-
Table 5: Coalesced results of LLMs performance on Task 1 (grouping) using two additional test datasets OCW-Randomized and OCW-WordNet with decreasing presence of red-herrings from left to right in the walls, juxtaposed against the original OCW test set (left-most column). Only the main metrics are shown (details and full results in Appendix §C). Bold: best scores.
Winograd schema challenge [35] and word sense induction (WSI) [80]. WSD aims to determine a word's correct meaning or sense within a specific context. WSI focuses on automatically clustering words into different senses or semantic categories based on their contextual usage patterns. Benchmarks like GLUE [74] and SuperGLUE [73] are aimed at aggregating and standardizing these classical NLP tasks to evaluate language models. The PLMs (e.g., BERT variants) and the first generation of LLMs, mostly solved or attained human-level performance on these tasks by 2020s [41].
In order to evaluate the human-imitative capabilities of modern LLMs, more challenging tasks have been proposed in recent benchmarks like BIG-bench [66] and HumanEval [15]. BIG-bench aims to address the limitations of existing benchmarks by providing a more comprehensive, open, and dynamic (tasks added on a rolling basis) evaluation benchmark. It covers a wide range of tasks, including a suite of tasks targeted specifically for human-like behavior. HumanEval is an evaluation set to measure the functional correctness of code synthesis from docstrings [15]. This benchmark includes 164 original programming problems that assess language comprehension, algorithms, and simple mathematics comparable to simple software interview questions. While these recent benchmarks include a wide net of complex tasks, evaluating a broad range of LLM capabilities, our work here is orthogonal to these since none of them aims to specifically measure creative problem-solving or creativity and their impediments in LLMs.
As with any machine learning dataset, especially one designed to evaluate the performance of LLMs, the OCW dataset has several limitations. First, we noticed that the performance of contextual approaches can vary significantly depending on the order that clues are provided to the model. To alleviate this (and where feasible), we evaluate models across 16 random sortings of the clues. Due to cost, we did not evaluate GPT-3.5-turbo and GPT-4's sensitivity to this ordering; future work should report performance across multiple random sorts. Second, due to the nature of the quiz show Only Connect, the clues, groups, and connections in the dataset tend to be Western- (and specifically UK-) centric (e.g. "Doctor Who companions", "English cricket captains", "Irish counties"). Therefore, performance on the OCW dataset may not extrapolate to languages or cultures outside of Western English. In fact, the US-centric bias of LLMs like GPT-3.5/4 [84] might partially explain their poor performance on the UK-centric OCW dataset. We hope to add additional Only Connect inspired walls in multiple languages and with clues derived from various cultures & subcultures in future work. Finally, given that the walls are publicly available as text on fan sites like ocdb.cc, there is always the possibility that they are included in the training sets of LLMs like GPT. However, we think this is unlikely, given the low performance on the grouping and connection tasks. Preventing the test sets of publicly available datasets like our OCW from "leaking" into the training sets of LLMs remains an interesting and open problem. We have taken basic steps against this leakage by distributing our dataset in a compressed format [30].
[80] L. White, R. Togneri, W. Liu, and M. Bennamoun. Finding word sense embeddings of known meaning. In Computational Linguistics and Intelligent Text Processing: 19th International Conference, CICLing 2018, Hanoi, Vietnam, March 18–24, 2018, Revised Selected Papers, Part II, pages 3–16. Springer, 2023.
[81] Wikipedia contributors. Only connect — Wikipedia, the free encyclopedia. https://en.wikipedia.org/w/index.php?title=0nly_Connect&oldid=1157929067, 2023. [Online; accessed 7-June-2023].
[82] J. Wiley. Expertise as mental set: The effects of domain knowledge in creative problem solving. Memory & cognition, 26:716-730, 1998.
[83] T. Wolf, L. Debut, V. Sanh, J. Chaumont, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-demos.6.
[84] R. Wolfe and A. Caliskan. American== white in multimodal language-and-image ai. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pages 800-812, 2022.
[85] G. Wood and J. Pennington. Encoding and retrieval from long-term storage. Journal of Experimental Psychology, 99(2):243, 1973.
[86] C.-L. Wu, S.-Y. Huang, P.-Z. Chen, and H.-C. Chen. A systematic review of creativity-related studies applying the remote associates test from 2000 to 2019. Frontiers in psychology, 11: 573432, 2020.
[87] S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. ArXiv preprint, abs/2305.10601, 2023.
[88] B. Yin, M. Zhao, L. Guo, and L. Qiao. Sentence-BERT and k-means based clustering technology for scientific and technical literature. In 2023 15th International Conference on Computer Research and Development (ICCRD), pages 15-20. IEEE, 2023.
[89] S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al. OPT: Open pre-trained transformer language models. ArXiv preprint, abs/2205.01068, 2022.
[90] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi. Bertscore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[91] W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, et al. A survey of large language models. ArXiv preprint, abs/2303.18223, 2023.
Task 1 - Grouping In addition to grouping clue words using token embeddings (discussed in the main paper §4), we also ran grouping the words by clustering on 'contextual' embeddings. We experimentally induce 'context' by joining the sixteen (16) word tokens (in a random order) into a single pseudo-sentence. The embeddings for each token were different based on the ordering of the tokens. We repeat the random ordering sixteen times and report the mean and variance of the results obtained in Table 6.
WD ↓	FMS ↑	ARI ↑	AMI ↑	# Solved Walls	# Correct Groups
ELMoLARGE	90.0 ± .3	23.6 ± .4	4.5 ± .5	5.6 ± .7	0 ± 0	19 ± 3
DistilBERTBASE	88.4 ± .7	26.7 ± .3	8.3 ± .4	10.4 ± .5	0 ± 0	30 ± 4
BERTLARGE	87.2 ± .6	28.3 ± .5	10.4 ± .6	12.8 ± .7	0 ± 0	46 ± 5
BERTBASE	87.7 ± .5	28.0 ± .2	10.0 ± .3	12.4 ± .4	0 ± 0	39 ± 2
RoBERTaLARGE	88.4 ± .5	25.9 ± .2	7.4 ± .3	9.3 ± .4	0 ± 0	30 ± 4
all-mpnetBASE	87.6 ± .5	28.0 ± .3	10.0 ± .4	12.4 ± .5	0 ± 0	38 ± 3
E5LARGE	87.7 ± .5	28.1 ± .3	10.2 ± .4	12.7 ± .5	0 ± 0	37 ± 4
E5BASE	87.2 ± .3	28.2 ± .2	10.2 ± .3	12.5 ± .4	0 ± 0	46 ± 5
Human Performace	-	-	-	-	285 / 494	1405 / 1976
Table 6: Results of selected models on Task 1 (Grouping) using contextual embeddings. WD: Wasserstein Distance. FMS: Fowlkes Mallows Score. ARI: Adjusted Rand Index. NMI: Normalized Mutual Information. Mean ± standard deviation over 16 random seeds is shown. Bold: best scores.
Task 2 – Connections In addition to prompting based results on GPT-4 (discussed in §4), we ran experiments on additional LLMs like LLaMa [67] (7B, 13B) using pre-trained configuration weights obtained by permission from Meta AI. However, without additional fine-tuning on the specific task, these LLMs were unable to solve the task in a meaningful manner. To elucidate, LLaMa generated a bunch of hallucinated words with unequal group sizes. We omit these unintelligible results for brevity.
B Additional Figures
In this section, we provide additional t-SNE projections of embeddings from various methods used.
Figure 7: Solved wall for Task 1 (Grouping) using GloVe. Left: (wall_id="7ed3"), the embedding model erroneously associated the clue "Suspension" with the connection "Bridges"; however, this association is an example of a red herring. "Suspension" is "a term used in musical harmony" in this context. Right: (wall_id="5e3c"), shows that clue "Lord" is close to "God, Heavens, and Grief" in the embedding space, which matches the "Good!" connection. However, this is another example of a red herring as, in this context, "Lord" refers to "Lord's cricket Ground", a cricket stadium named after "Thomas Lord".
Figure 8: Solved wall for Task 1 (Grouping) using FastText (Crawl). Left: (wall_id="d5e6"), the embedding model erroneously associated the clue "Tara" other girls' names; but here, "Tara" is short for "Hill of Tara" and belongs to the "national coronation sites" group. Right: (wall_id="4c22"), shows that clue "Pie" associated with the connection "Apple". Even though it is acceptable in general context, here it represents a homophone for the Greek letter "π".
Figure 9: Solved wall (wall_id="2d8f") for Task 1 (Grouping) using BERTLarge with both static and contextual embeddings. Left: contextual embedding solved 3/4 groups. Here the clue "Rambrandt" is placed near other Dutch painters. The correct grouping for this clue in this wall is "Toothpaste Brands". Right: static embedding solved 0/4 groups.
C Effects of Red-Herrings: Additional Experiments, Analysis and Results
C.1 Additional Datasets
Both of the additional datasets described in this section for ablation experiments have been made available via our code repositories on Github and HuggingFace.
C.1.1 OCW-Randomized Dataset
This test dataset generates a version of the test set where red herrings are removed or largely reduced in frequency. This is achieved by rebuilding every wall using a randomly selected group from different walls. We only applied the process to the (original OCW) test set, the train and validation sets are left untouched.
Method For each wall in the existing test set, we leave the first group untouched, and sample three new groups, each from a different wall, such that none of the groups share a word in common. The connections for each group are unmodified. The result is a new version of the test set where every wall is composed of 4 random groups from 4 different walls.
C.1.2 OCW-WordNet Dataset
WordNet [46, 20] is a large lexical database of English. Nouns, verbs, adjectives, and adverbs are grouped into sets of cognitive synonyms (synsets), each expressing a distinct concept. We use the hybernym/hyonym (or superlative/subordinative) hierarchical lexical structure aggregated in WordNet to generate an easy test set to further analyze the effects of red-herring in OCW.
Method We use the existing words in a wall to select synonyms from the word's synsets. We only consider synsets that have at least five synonymous lexical names, then randomly sample four words. The original test set word and its definition (ss.definition()) subsequently becomes the connection phrase for the group. Four groups were generated for each wall, and the easy wall generation process was repeated for the total number of walls (494) in the original test data set.
For the group connections, we concatenate the superlative parent word with a synset definition giving a description of the word. This allows for an ideal semantic similarity score to be calculated using BERTScore. For a few cases (approx. 70/494 walls in the test set), the number of generated groups per wall is less than four, due to the unavailability of direct synonyms from word synsets. In those edge cases, we generate and append groups using common hyphenym words like animal, mammal, furniture, etc. to ensure a wall is valid with four groups.
A sample generated easy group is shown below, where we prefix the group_id from the original OCW dataset with 'easy' to aid with mapping or identification.
{ ... "group_3": { "group_id": "easy_691a_3", "gt_words": ["gibe","shaft","jibe","barb"], "gt_connection": "Shaft: an aggressive remark directed at a person like a missile and intended to have a telling effect" }
Further, we generate easy to train and validation sets mimicking the original dataset, package and release these three additional easy sets, as OCW-WordNet as added contributions.
C.2 Results of Ablation Experiments
C.2.1 PLMs: Performance on Task 1 (Grouping)
We perform and present the results using 'static' embeddings due to the noted superior results and the word order related deficiency already shown by using contextual embeddings pertinent to our task setup.