SlowGuess/NAACL_2022_OCR huggingface dataset:
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dólar, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Prasanta Chandra Mahalanobis. 1936. On the generalized distance in statistics. National Institute of Science of India.
Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. On measuring social biases in sentence encoders. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 622-628, Minneapolis, Minnesota. Association for Computational Linguistics.
William Merrill, Yoav Goldberg, Roy Schwartz, and Noah A. Smith. 2021. Provable limitations of acquiring meaning from ungrounded form: What will future language models understand? Transactions of the Association for Computational Linguistics, 9:1047-1060.
Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in pre-training distributed word representations. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018).
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111-3119.
Kaichun Mo, Shilin Zhu, Angel X. Chang, Li Yi, Subarna Tripathi, Leonidas J. Guibas, and Hao Su. 2019. PartNet: A large-scale benchmark for fine-grained and hierarchical part-level 3D object understanding. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. 2020. How context affects language models' factual predictions. In *Automated Knowledge Base Construction*.
Fabio Petroni, Tim Rocttäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463-2473, Hong Kong, China. Association for Computational Linguistics.
Alec Radford, Jeff Wu, R. Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B. Viégas, Andy Coenen, Adam Pearce, and Been Kim. 2019. Visualizing and measuring the geometry of BERT. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 8592-8600.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842-866.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting knowledge from language models with automatically generated prompts. In Empirical Methods in Natural Language Processing (EMNLP).
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. 2012. Indoor segmentation and support inference from rgbd images. In European conference on computer vision, pages 746-760. Springer.
Damien Sileo. 2021. Visual grounding strategies for text-only natural language processing. In Proceedings of the Third Workshop on Beyond Vision and LANguage: inTEgrating Real-world kKnowledge (LANTERN), pages 19-29, Kyiv, Ukraine. Association for Computational Linguistics.
Shuran Song, Fisher Yu, Andy Zeng, Angel X. Chang, Manolis Savva, and Thomas Funkhouser. 2017. Semantic scene completion from a single depth image. Proceedings of 30th IEEE Conference on Computer Vision and Pattern Recognition.
Gábor J Székely, Maria L Rizzo, and Nail K Bakirov. 2007. Measuring and testing dependence by correlation of distances. The annals of statistics, 35(6):2769-2794.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. CoRR, abs/1905.06316.
François Torregrossa, Vincent Claveau, Nihel Kooli, Guillaume Gravier, and Robin Allesiardo. 2020. On the correlation of word embedding evaluation metrics. In Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 4789-4797.
Amos Tversky and Itamar Gati. 2004. Studies of similarity. In Eldar Shafir, editor, Preference, Belief, and Similarity. Selected Writing of Amos Tversky, pages 75-95. MIT Press.
Ekaterina Vylomova, Laura Rimell, Trevor Cohn, and Timothy Baldwin. 2016. Take and took, gaggle and goose, book and read: Evaluating the utility of vector differences for lexical relation learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1671-1682, Berlin, Germany. Association for Computational Linguistics.
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The benchmark of linguistic minimal pairs for English. Transactions of the Association for Computational Linguistics, 8:377-392.
Liang Yao, Chengsheng Mao, and Yuan Luo. 2019. Kgbert: Bert for knowledge graph completion. arXiv preprint arXiv:1909.03193.
Li Zhang, Jun Li, and Chao Wang. 2017. Automatic synonym extraction using word2vec and spectral clustering. In 2017 36th Chinese Control Conference (CCC), pages 5629-5632. IEEE.
Wenxuan Zhou, Junyi Du, and Xiang Ren. 2019. Improving bert fine-tuning with embedding normalization. arXiv preprint arXiv:1911.03918.
A tabular breakdown of the datasets used can be seen in Table 4, 5 and 6. The exact models used are listed in Table 7. The heatmap visualizations for the other two datasets are in Figure 3 and 4.
A.1 Word Frequency
We also conducted an experiment to correlate the scores with their frequency. For this purpose, the corresponding objects of each target were selected. And then the distance correlation between the scores and the corresponding word frequency was calculated based on the average of the last 10 years of Google Ngrams. The results are shown in Table 9. The correlations are not particularly significant (mostly  $p \geq 0.1$ ), but it is noticeable that especially the cosine score depends strongly on the word frequency. The classifiers are generally less sensitive.
bathroom	bedroom	kitchen	living room	office
object	score	object	score	object	score	object	score	object	score
toilet	1.00	dresser	1.00	drying rack	1.00	coffee table	0.94	whiteboard	1.00
bathtub	1.00	night stand	1.00	kitchen island	1.00	ottoman	0.93	room divider	0.94
toothbrush holder	1.00	headboard	1.00	pot	1.00	fireplace	0.87	stapler	0.92
toothpaste	1.00	bed	0.97	frying pan	1.00	dvd player	0.69	cork board	0.92
shower curtain	1.00	alarm clock	0.97	spice rack	1.00	sofa	0.68	file	0.88
toothbrush	0.97	laundry basket	0.86	cutting board	1.00	decorative plate	0.61	keyboard	0.85
towel rod	0.96	hat	0.74	blender	1.00	tv stand	0.57	mouse	0.84
toilet paper	0.96	doll	0.70	knife	1.00	blanket	0.55	pen	0.83
squeeze tube	0.95	stuffed animal	0.60	stove	0.98	television	0.53	computer	0.82
faucet handle	0.82	pillow	0.56	dishwasher	0.97	remote control	0.50	column	0.81
Table 4: Statistics generated from ScanNet using NYU categories: score is the conditional probability  $P(\text{room} \mid \text{object})$  of the room given the object based on the frequencies observable in NYU.
bed	dishwasher	door	mortise lock	refrigerator	toilet
object	score	object	score	object	score	object	score	object	score	object	score
pillow	1.00	drain hose	1.00	lock	1.00	ring	1.00	switch	1.00	valve seat shaft	1.00
bolster	1.00	overflow protection switch	1.00	cornice	1.00	keyway	1.00	refrigerator compa-ment	1.00	tank lid	1.00
mattress cover	1.00	tub	1.00	hanging stile	1.00	cotter pin	1.00	egg tray	1.00	conical washer	1.00
leg	1.00	pump	1.00	entablature	1.00	spring	1.00	shelf channel	1.00	lift chain	1.00
box spring	1.00	gasket	1.00	top rail	1.00	rotor	1.00	magnetic gasket	1.00	seat	1.00
headboard	1.00	water hose	1.00	middle panel	1.00	cylinder case	1.00	storage door	1.00	shutoff valve	1.00
mattress	1.00	heating element	1.00	bottom rail	1.00	key	1.00	freezer door	1.00	trip lever	1.00
pillow protector	1.00	rack	1.00	panel	1.00	faceplate	1.00	guard rail	1.00	ball-cock supply valve	1.00
elastic	1.00	cutlery basket	1.00	jamb	1.00	dead bolt	1.00	crisper	1.00	toilet bowl	1.00
footboard	1.00	wash tower	1.00	doorknob	1.00	cylinder	1.00	glass cover	1.00	flush handle	1.00
motor	1.00	threshold	1.00	stator	1.00	butter compartment	1.00	wax seal	1.00
detergent dispenser	1.00	weatherboard	1.00	strike plate	1.00	thermostat control	1.00	tank ball	1.00
slide	1.00	lock rail	1.00	freezer compartment	1.00	float ball	1.00
leveling foot	1.00	shutting stile	1.00	ice cube tray	1.00	filler tube	1.00
insulating material	1.00	header	1.00	meat keeper	1.00	waste pipe	1.00
spray arm	1.00	door stop	1.00	seat cover	1.00
rinse-aid dispenser	1.00	shelf	1.00	cold-water supply line	1.00
dairy compartment	1.00	overflow tube	1.00
door shelf	1.00	trap	1.00
refill tube	1.00
Table 5: A subset of part-whole relations extracted from Online-Bildwörterbuch. All parts have a value of 1.00 in our data set, because they only occur with this object.
eat	listen to	play	read	wash with	wear
object	score	object	score	object	score	object	score	object	score	object	score
food	0.13	music	0.22	game	0.27	book	0.08	soap	0.29	clothing	0.07
diet	0.08	song	0.03	music	0.06	label	0.06	water	0.29	glove	0.06
meal	0.07	body	0.03	note	0.04	instruction	0.05	vinegar	0.04	shoe	0.05
breakfast	0.04	side	0.02	sport	0.03	review	0.04	solution	0.03	clothes	0.05
balanced diet	0.03	partner	0.02	chord	0.02	body language	0.02	detergent	0.03	shirt	0.02
fruit	0.03	child	0.02	song	0.02	rule	0.02	baking soda	0.03	makeup	0.02
vegetable	0.03	perspective	0.02	video game	0.02	example	0.02	cream	0.02	gear	0.02
plenty	0.03	response	0.02	card	0.02	complaint	0.01	shampoo	0.02	boot	0.02
protein	0.03	parent	0.02	role	0.02	law	0.01	towel	0.02	dress	0.02
snack	0.02	people	0.02	video	0.02	story	0.01	cold water	0.02	sock	0.02
Table 6: A subset of verb-object relations extracted from an updated version of HowToKB.
Model	Specification	Dimension	Parameters	Dataset Size (T;S)	URL
word2vec	GoogleNews-vectors-negative300	300	-	100B; -	https://code.google.com/archive/p/word2vec/
Glove	Common Crawl - glove.840B.300d	300	-	840B; -	https://nlp.stanford.edu/projects/glove/
Levy	Dependency-Based Words	300	-	English Wiki (~2B tokens)	https://levyomer.wordpress.com/2014/04/25/ dependency-based-word-embeddings/
fastText	crawl-300d-2M-subword	300	-	600B; -	https://fasttext.cc/docs/en/ english-vectors.html
static-BERT	bert_12layer_sent	768	-	+1.28B; -	https://zenodo.org/record/5055755
BERT-Base	bert-base-uncased	768	~110M	3.3B; 16GB	https://huggingface.co/bert-base-uncased
BERT-Large	bert-large-uncased	1024	~336M	3.3B; 16GB	https://huggingface.co/bert-large-uncased
RoBERTa	roberta-large	1024	~336M	-; 160GB	https://huggingface.co/roberta-large
ELECTRA	electra-large-generator	256	~51M	https://huggingface.co/google/ electra-large-generator
ALBERT	albert-xxlarge-v2	4096	~223M	3.3B; 16GB	https://huggingface.co/albert-xxlarge-v2
GPT2	gpt2-large	1280	~774M	-; 40GB	https://huggingface.co/gpt2-large
GPT-Neo	gpt-neo-2.7B	2560	~2.7B	420B; -	https://huggingface.co/EleutherAI/ gpt-neo-2.7B
GPT-J	gpt-j-6B	4096	~6B	-; 825GB	https://huggingface.co/EleutherAI/gpt-j-6B
Table 7: Model overview. Mostly only the token quantity (T) or the dataset size (S) was given.
Task	Model	Data	Templates
Cosine Score & Classification	MLM & CLM	Room & Objects & Parts	This is a/an {x}. That is a/an {x}. There is a/an {x}. Here is a/an {x}. A/An {x} is here. A/An {x} is there.
Verbs	I {x} something. I {x} anything. I {x}. You {x} something. You {x} anything. You {x}.
Increased Log Probability	MLM	Room & Object	A/An {obj} is usually in the {room}.
Object & Part	A/An {part} is usually part of a/an {obj}.
Verb & Object	I usually {verb} this {obj}.
CLM	Room & Object	A/An {obj} is usually in the ... In the {room} is usually a/an ...
Object & Part	A/An {part} is usually part of a ... In the {obj} is usually a/an ...
Verb & Object	I usually {verb} this ...
Table 8: Templates for calculating scores regarding Masked Language Models (MLM) and Causal Language Models (CLM). For more details, see Sec. 4.
Word2Vec	GloVe	Levy	fastText	static-BERT
cos	dist	kend	knn	svm	ffn	cos	dist	kend	knn	svm	ffn	cos	dist	kend	knn	svm	ffn	cos	dist	kend	knn	svm	ffn	cos	dist	kend	knn	svm	ffn
Room	bathroom	0.73	0.75	0.78	0.31	0.31	0.22	0.53	0.56	0.57	0.23	0.00	0.23	0.65	0.67	0.68	0.25	0.00	0.32	0.65	0.67	0.70	0.00	0.00	0.00	0.74	0.75	0.76	0.56	0.41	0.47
bedroom	0.55	0.53	0.56	0.00	0.36	0.35	0.72	0.72	0.69	0.50	0.35	0.57	0.51	0.51	0.50	0.00	0.00	0.37	0.58	0.54	0.53	0.66	0.55	0.35	0.45	0.44	0.44	0.68	0.35	0.36
kitchen	0.51	0.52	0.51	0.35	0.49	0.42	0.37	0.34	0.34	0.29	0.35	0.33	0.46	0.46	0.46	0.20	0.00	0.20	0.33	0.36	0.40	0.30	0.20	0.37	0.46	0.46	0.46	0.20	0.31	0.42
living room	0.63	0.61	0.61	0.36	0.46	0.50	0.48	0.62	0.63	0.44	0.29	0.30	0.53	0.57	0.57	0.41	0.41	0.54	0.39	0.57	0.59	0.28	0.29	0.27	0.52	0.54	0.53	0.00	0.30	0.52
office	0.65	0.65	0.58	0.38	0.26	0.37	0.52	0.57	0.56	0.38	0.43	0.47	0.66	0.66	0.65	0.27	0.50	0.50	0.45	0.43	0.41	0.58	0.35	0.30	0.37	0.40	0.40	0.57	0.17	0.42
BERT-Base	BERT-Large	RoBERTa	ElectraGen	Albert
cos	m-s	m-t	knn	svm	ffn	cos	m-s	m-t	knn	svm	ffn	cos	m-s	m-t	knn	svm	ffn	cos	m-s	m-t	knn	svm	ffn	cos	m-s	m-t	knn	svm	ffn
Room	bathroom	0.40	0.35	0.30	0.23	0.23	0.23	0.52	0.34	0.50	0.23	0.23	0.24	0.61	0.47	0.42	0.43	0.23	0.35	0.58	0.66	0.39	0.35	0.35	0.40	0.69	0.56	0.36	0.34	0.36	0.39
bedroom	0.61	0.47	0.41	0.45	0.28	0.37	0.63	0.36	0.37	0.42	0.19	0.36	0.67	0.56	0.33	0.42	0.28	0.41	0.41	0.53	0.62	0.68	0.55	0.50	0.54	0.58	0.36	0.18	0.31	0.47
kitchen	0.34	0.60	0.35	0.30	0.23	0.43	0.38	0.73	0.45	0.75	0.75	0.62	0.65	0.38	0.49	0.46	0.34	0.24	0.48	0.37	0.37	0.25	0.44	0.43	0.43	0.75	0.38	0.35	0.22	0.45
living room	0.52	0.57	0.56	0.25	0.20	0.32	0.47	0.47	0.54	0.36	0.36	0.39	0.43	0.51	0.45	0.51	0.27	0.27	0.41	0.36	0.45	0.44	0.49	0.47	0.38	0.54	0.47	0.19	0.30	0.34
office	0.68	0.56	0.64	0.35	0.44	0.56	0.64	0.75	0.67	0.26	0.58	0.45	0.48	0.49	0.53	0.27	0.28	0.47	0.50	0.70	0.55	0.59	0.55	0.25	0.72	0.54	0.61	0.58	0.37	0.36
Table 9: Distance Correlation calculated on the word frequencies of Google Ngram. (A gray cell indicates significant at  $p < {0.1}$  )
(a) Cosine Score
(b) FFN Classify Score
(c) Mask Object Score
Figure 2: Heatmap of source-object associations based on BERT-Large and the room dataset. The objects (sources) on the y-axis are grouped by the room in which they are most likely to be located according to the NYU Depth V2 Dataset.
(d) Mask Room Score
(d) Mask Object Score
Figure 3: Association heatmap of BERT-Large on the part dataset. The parts (sources) on the y-axis are grouped by the room in which they are most likely to be located according to the Online-Bildwörterbuch Dataset.
(c) Mask Object Score
Figure 4: Association heatmap of BERT-Large on the verb dataset. The objects (sources) on the y-axis are grouped by the room in which they are most likely to be located according to the HowToKB Dataset.
(d) Mask Verb Score