SlowGuess/NeurIPS_2023_OCR huggingface dataset:
WD ↓	FMS ↑	ARI ↑	AMI ↑	# Solved Walls	# Correct Groups
Classic Word Embeddings
GloVe	76.8 ± .7	39.2 ± .3	24.0 ± .4	27.7 ± .4	7 ± 1	213 ± 8
FastText (Crawl)	76.1 ± .5	40.5 ± .3	25.0 ± .6	28.6 ± .7	13 ± 1	236 ± 7
FastText (News)	79.3 ± .5	36.8 ± .3	21.0 ± .3	24.5 ± .4	5 ± 1	176 ± 6
Pre-trained Language Models (PLMs)
ELMoLARGE	80.9 ± .4	35.2 ± .3	18.9 ± .3	22.2 ± .4	3 ± 1	154 ± 6
DistilBERTBASE	82.3 ± .6	34.2 ± .4	17.7 ± .5	21.1 ± .5	1 ± 1	124 ± 8
BERTLARGE	86.2 ± .4	29.2 ± .3	11.5 ± .3	14.2 ± .4	0 ± 0	66 ± 4
BERTBASE	87.5 ± .4	27.7 ± .3	9.6 ± .6	11.8 ± .5	0 ± 0	48 ± 4
RoBERTaLARGE	86.7 ± .5	28.6 ± .2	10.8 ± .3	13.4 ± .3	1 ± 0	56 ± 4
Sentence Transformers
all-mpnetBASE	81.4 ± .4	35.1 ± .4	18.9 ± .5	22.0 ± .6	8 ± 1	154 ± 7
E5LARGE	76.0 ± .5	40.7 ± .3	25.9 ± .4	29.7 ± .4	8 ± 1	230 ± 5
E5BASE	75.1 ± .8	41.8 ± .3	27.2 ± .3	31.1 ± .3	8 ± 1	249 ± 8
Human Performance	-	-	-	-	-	-
Table 7: Results of OCW-Randomized using static embeddings. WD: Wasserstein Distance. FMS: Fowlkes Mallows Score. ARI: Adjusted Rand Index. NMI: Normalized Mutual Information. Mean ± standard deviation over 16 random seeds is shown. Bold: best scores.
WD ↓	FMS ↑	ARI ↑	AMI ↑	# Solved Walls	# Correct Groups
Classic Word Embeddings
GloVe	43.0 ± 1.0	66.1 ± .4	57.4 ± .5	60.9 ± .5	118 ± 3	886 ± 1
FastText (Crawl)	30.6 ± 1.0	75.8 ± .6	69.6 ± .7	72.4 ± .7	195 ± 6	1173 ± 18
FastText (News)	44.9 ± 1.2	64.9 ± .5	55.9 ± .6	59.5 ± .6	105 ± 3	844 ± 12
Pre-trained Language Models (PLMs)
ELMoLARGE	52.5 ± 1.1	58.9 ± .3	48.2 ± .4	52.5 ± .4	67 ± 3	682 ± 9
DistilBERTBASE	45.5 ± 1.0	64.1 ± .4	55.0 ± .5	58.7 ± .5	105 ± 3	835 ± 13
BERTLARGE	76.9 ± 1.0	38.9 ± .2	23.4 ± .3	27.5 ± .3	7 ± 0	197 ± 6
BERTBASE	73.0 ± 1.3	42.5 ± .5	27.9 ± .6	32.5 ± .6	8 ± 2	268 ± 12
RoBERTaLARGE	57.4 ± 1.3	54.8 ± .3	43.3 ± .3	47.5 ± .3	48 ± 2	573 ± 8
Sentence Transformers
all-mpnetBASE	22.6 ± .7	81.9 ± .4	77.1 ± .5	79.4 ± .4	256 ± 4	1365 ± 12
E5LARGE	23.6 ± .8	80.9 ± .4	75.9 ± .5	78.3 ± .4	250 ± 4	1347 ± 12
E5BASE	26.9 ± .9	78.0 ± .4	72.3 ± .5	75.0 ± .5	224 ± 4	1259 ± 10
Human Performance	-	-	-	-	-	-
Table 8: Results of OCW-WordNet using static embeddings. WD: Wasserstein Distance. FMS: Fowlkes Mallows Score. ARI: Adjusted Rand Index. NMI: Normalized Mutual Information. Mean ± standard deviation over 16 random seeds is shown. Bold: best scores.
C.2.2 LLMs: Performance on Task 1 (Grouping) using GPT3.5/4
Here we present the results of repeating Task 1 (grouping) on the ablation datasets OCW-Randomized (C.1.1) and OCW-Wordnet (C.1.2) to analyze the effects of red-herrings in walls on LLM performance.
The results adhere to the expected results of superior performance with the dilution/removal of red-herrings from the walls.
# In-context Examples	WD ↓	FMS ↑	ARI ↑	AMI ↑	# Solved Walls	# Correct Groups
GPT-3.5-turbo	0-shot	74.3	40.4	26.4	29.8	5	274
1-shot	72.0	43.1	29.0	32.3	12	315
3-shot	72.7	43.4	29.4	32.9	10	306
5-shot	70.7	44.6	30.9	34.4	16	337
10-shot	70.5	43.8	30.0	33.5	17	333
GPT-4	0-shot	58.2	56.2	45.4	48.8	59	595
1-shot	55.1	58.0	47.5	51.0	57	644
3-shot	55.0	57.5	46.9	50.3	62	649
5-shot	54.1	58.0	47.5	50.9	68	655
10-shot	56.6	56.1	45.1	48.5	55	614
Human Performance	-	-	-	-	-	-
Table 9: Results of OCW-Randomized using Large Language Models. WD: Wasserstein Distance. FMS: Fowlkes Mallows Score. ARI: Adjusted Rand Index. NMI: Normalized Mutual Information. Bold: best scores.
# In-context Examples	WD ↓	FMS ↑	ARI ↑	AMI ↑	# Solved Walls	# Correct Groups
GPT-3.5-turbo	0-shot	15.9	86.3	83.4	84.9	337	1522
1-shot	24.8	76.4	74.4	75.4	320	1400
3-shot	8.65	92.7	91.2	91.8	415	1748
5-shot	8.09	94.0	92.4	93.1	415	1759
10-shot	6.55	95.3	94.0	94.7	428	1800
GPT-4	0-shot	1.51	98.5	98.0	98.2	471	1926
1-shot	19.2	87.9	84.3	83.7	304	1581
3-shot	21.5	86.6	82.5	81.8	279	1537
5-shot	19.1	88.1	84.5	83.8	298	1584
10-shot	11.2	92.9	90.7	90.4	378	1742
Human Performance	-	-	-	-	-	-
Table 10: Results of OCW-WordNet using Large Language Models. WD: Wasserstein Distance. FMS: Fowlkes Mallows Score. ARI: Adjusted Rand Index. NMI: Normalized Mutual Information. Bold: best scores.