Venue: NAACL 2022

# What do Toothbrushes do in the Kitchen?

How Transformers Think our World is Structured

Alexander Henlein

Alexander Mehler

Text Technology Lab, Goethe-University Frankfurt, Germany {henlein, mehler}@em.uni-frankfurt.de

###### Abstract

Transformer-based models are now predominant in NLP. They outperform approaches based on static models in many respects. This success has in turn prompted research that reveals a number of biases in the language models generated by transformers. In this paper we utilize this research on biases to investigate to what extent transformer-based language models allow for extracting knowledge about object relations (\(X\)_occurs in \(Y;\)\(X\) consists of \(Z;\) action \(A\) involves using \(X\)_). To this end, we compare contextualized models with their static counterparts. We make this comparison dependent on the application of a number of similarity measures and classifiers. Our results are threefold: Firstly, we show that the models combined with the different similarity measures differ greatly in terms of the amount of knowledge they allow for extracting. Secondly, our results suggest that similarity measures perform much worse than classifier-based approaches. Thirdly, we show that, surprisingly, static models perform almost as well as contextualized models - in some cases even better.

+
Footnote †: _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 5791 - 5807

## 1 Introduction

Few models have recently influenced NLP as much as transformers Vaswani et al. (2017). Hardly any new NLP system today is introduced without a transformer-based model such as BERT Devlin et al. (2019) or GPT Radford et al. (2019). As a result, static models such as word2vec Mikolov et al. (2013) are increasingly being substituted. Nevertheless, transformers are still far from being fully understood. Thus, research studies are being conducted to find out how they work and what properties the language models they generate have.

During training, transformers seem to capture both syntactic and semantic features Rogers et al. (2020). For example, dependency trees can be reconstructed from trained attention heads Clark et al. (2019), syntactic trees can be reconstructed from word encodings Hewitt and Manning (2019), and these encodings can be clustered into representations of word senses Reif et al. (2019). BERT also seems to encode information about entity types and semantic roles Tenney et al. (2019). For an overview of this research see Rogers et al. (2020).

Since BERT and other transformers are trained on various data crawled from the internet, they are sensitive to biases Caliskan et al. (2017); May et al. (2019); Bender et al. (2021). In practice, instead of reproducing negative biases, they are expected to allow for the derivation of statements, such as that toothbrushes are spatially associated with bathrooms rather than living rooms. In this line of thinking, approaches such as the popularization of knowledge graphs can be located Yao et al. (2019); Petroni et al. (2019); Heinzerling and Inui (2021). Our paper is situated in this context. More specifically, we examine the extent to which knowledge about spatial objects and their relations is implicitly encoded in these models. Since the underlying texts are rather implicit regarding such information, it can be assumed that the object relations derivable from transformers are weakly encoded cf. Landau and Jackendoff (1993); Hayward and Tarr (1995). Reading, for example, the sentence:

"After getting up, I ate an apple"

one may assume that the narrator got up from his bed in the bedroom, went to the kitchen, took an apple, washed it in the sink, and finally ate it. The apple is also likely to have been peeled and cut. Equally, however, nothing is said in the sentence about a bedroom or a kitchen. Nevertheless, it is a well known approach to explore the usage regularities of words, currently most efficiently represented by neural networks, as a source for knowledge extraction (see, e.g. Zhang et al., 2017; Bouraoui et al., 2020; Shin et al., 2020; Petroni et al., 2019).

In this work, we use a number of methods to identify biases in contextualized models and ask to what extent they can be used to extract object

based knowledge from these models. To this end, we consider three relations:

1. _Spatial containment of (source) objects in (target) rooms:_ e.g. a fridge probably belongs in a kitchen, but not in a living room;
2. _Parts (source) in relation to composite objects (target):_ e.g. a refrigerator compartment is probably a part of a fridge;
3. _Objects (source) in relation to actions (target) that involve them:_ e.g. reading involves something being read, e.g., a book.

Regarding these relations, we examine a set of pretrained contextualized and static word representation models. This is done to answer the question to what extent they allow the extraction of instances of these relations when trained on very large datasets. We focus on rather common terms (_kitchen_, _to read_ etc.) as part of the general language.

It is assumed that (static or contextualized) models implicitly represent such relations, so that it is possible to identify probable targets starting from certain sources. That is, for a word like _fridge_ (source), we expect it to be semantically more strongly associated with _kitchen_ (target) than with words naming other rooms, since fridges are more likely to be found in kitchens than in other rooms, and that certain word representation models reflect this association. We also assume that this association is asymmetric and exists to a lesser extent from target to source (cf. Tversky and Gati (2004)).

The paper is organized as follows: Related work is reported in Sec. 2. The datasets we use are represented in Sec. 3 and our method in Sec. 4. Our experiments are presented in Sec. 5 and discussed in Sec. 6. Sec. 7 provides a conclusion. All used data, scripts and results are open source on GitHub1.

Footnote 1: [https://github.com/texttechnologylab/SpatialAssociationsInLM](https://github.com/texttechnologylab/SpatialAssociationsInLM)

## 2 Related Work

Biases in NLP models are not a new problem that appeared with BERT, but affect almost all models trained on language datasets Caliskan et al. (2017). As such, there are methods for measuring social biases in static models such as word2vec Mikolov et al. (2013). One of the best known approaches is WEAT Caliskan et al. (2017). Here, two groups of concepts are compared with two groups of attributes based on the difference between the sums of their cosine similarities (see Section 4). This approach already points to a methodological premise that also guides our work: Relations of entities are tentatively determined by similarity analyses of vectorial word representations.

However, a direct comparison of word vectors is not possible with contextualized methods such as BERT, where the vector representation of a word varies with the context of its occurrence (cf. Ethayarajh (2019)). Efforts to transfer the cosine-based approach from static to contextualized models have not been able to recreate similar performances May et al. (2019). Therefore, new approaches have been developed based on the specifics of contextualized models. For example, BERT is trained using masked language modeling, where the model estimates the probability of masked words in sentences Devlin et al. (2019). The probability distribution for a masked word in a given context can then be used as information to characterize candidate words Kurita et al. (2019). Sec. 4.3 describes this approach in more detail. An alternative approach is to examine the interpretability of models Belinkov and Glass (2019); Jiang et al. (2020); Petroni et al. (2019, 2020); Bommasani et al. (2020); Hupkes et al. (2020), which goes beyond the scope of this paper. In any event, both approaches share the same basic ideas, e.g., the probability prediction of mask tokens cf. Kurita et al. (2019); Belinkov and Glass (2019).

Work has also been done on how BERT represents information about spatial objects. For example, BERT has problems with certain object properties (e.g. _cheap_ or _cube_) or implicit visual properties that are rarely expressed Da and Kasai (2019). Problems are also encountered with extracting numerical commonsense knowledge, such as the typical number of tires on a car or the feet on a bird Lin et al. (2020). More than that, the models seem to allow for extracting some object knowledge, but not with respect to properties based on their affordance (e.g. objects through which one can see are transparent Forbes et al. (2019)). Even though these results seem to question the use of BERT and its competitors for knowledge extraction, these models still perform better in downstream tasks than their static competitors Devlin et al. (2019); Liu et al. (2019); Brown et al. (2020); Da and Kasai (2019). Bouraoui et al. (2020) compared these models using different datasets and lexical relations. These include relations similar to those examined here (e.g. a pot is usually found in a kitchen), but beyond the level of detail achieved in 

our study.

What will become increasingly important is the so-called grounding of language models Merrill et al. (2021): Here, the models are trained not only on increasingly large text data, but also, for example, on images thus enabling better "understanding" of spatial relations Sileo (2021); Li et al. (2020). In this paper, we focus on models without grounding.

## 3 Datasets Used for Evaluation

### Spatial Containment

The _NYU Depth V2 Dataset_Silberman et al. (2012) consists of video sequences of numerous indoor scenes. It features 464 labeled scenes using a rich category set. We use this dataset as a basis for evaluating the probability of occurrence of objects in rooms (e.g. kitchen, living room, etc.). That is, we estimate the conditional probability \(P(r\mid o)\) of a room \(r\) (target) given an object \(o\) (source). In this way, we aim to measure the strength of an object's association with a particular room as reflecting the corresponding spatial containment relation. At the same time, we want to filter out objects such as _window_ that are evenly distributed among the rooms studied here. In our experiments, we consider the ten most frequently mentioned objects in NYU to associate with the five most frequently mentioned spaces. This data is shown in the Table 4 (appendix).

The advantage of NYU over other scene datasets such as 3D-Front Fu et al. (2020) is that it deals with real spaces and not artificially created ones. In addition, NYU's object category set is relatively fine-grained (we counted 895 different object names) and uses colloquial terms. This is in contrast to, for example, SUNCG Song et al. (2017) (with categories like "slot machine with chair","range hood with cabinet", "food processor") and ShapeNetCore Chang et al. (2015) with only 55 object categories or COCO Lin et al. (2014) with 80 object categories. This makes NYU more suitable for our task of evaluating word representation models as resources for knowledge extraction starting from general language.

### Part-whole Relations

We use a subset of the object descriptions from _Online-Bildworterbruch_2. This resource describes very fine-grained part-whole relations of objects expressed by colloquial names, in contrast to, e.g., PartNet Mo et al. (2019) where one finds labels such as _seat single surface_ or _arm near vertical bar_. The list of objects from _Online-Bildworterbruch_ used in our study and their subdivisions is shown in Table 5. The selected objects were chosen by hand, provided that the chosen examples are general enough and the subdivision is sufficiently fine.

Footnote 2: [http://www.bildwoerterbuch.com/en/home](http://www.bildwoerterbuch.com/en/home)

### Action-object Relations

To study entities as typical objects of certain actions, we derive a dataset from HowToKB Chu et al. (2017) which is based on WikiHow3. In HowToKB, task frames, temporal sequences of subtasks, and attributes for involved objects were extracted from WikiHow articles. Some changes were made to the knowledge database, including a newly crawled version of WikiHow. In addition, the pre-processing tools have been updated and partially extended (see Table 6).

Footnote 3: [https://www.wikihow.com/](https://www.wikihow.com/)

#### 3.3.1 Related Datasets

For evaluating static models, there are datasets and approaches to measuring lexical relations, such as DiffVecVylomova et al. (2016), BATS Gladkova et al. (2016) or BLiMP Warstadt et al. (2020). Although these datasets are also used to evaluate BERT Bouraoui et al. (2020), they represent only an unstructured subset of the data we used and are thus not appropriate for our study.

## 4 Approach

We now present the static and contextualized models used in our study. Table 7 in the appendix lists these models and their sources. We also specify the measures used to compute word associations as a source of knowledge extraction, and describe how to use classifiers as an alternative to them.

### Static Models

Probably the best known static model is word2vec Mikolov et al. (2013). Its CBOW variant is trained to predict words in the context of their surrounding words. The word representations trained in this way partially encode semantic relations Mikolov et al. (2013), making them a suitable candidate for comparison with the corresponding information values of contextualized word representations. In addition to word2vec, we consider GloVe Pennington et al. (2014), Levy Levy and Goldberg (2014), 

fastText Mikolov et al. (2018) and a static BERT model Gupta and Jaggi (2021). Unlike window-based approaches to static embeddings, Levy embeddings are trained on dependency trees.

### Contextualized Models

Unlike static models, the vector representations of (sub-)word (units) in contextualized models depend on the context in which they occur so that tokens of the same type may each be represented differently in different contexts. All contextual models we evaluate here are pre-trained and come from the _huggingface models repository4_. We evaluate two types of contextualized models:

Footnote 4: [https://huggingface.co/models](https://huggingface.co/models)

**Masked Language Models (MLM)** are trained to reconstruct randomly masked words in input sequences. We experiment with BERT Devlin et al. (2019), RoBERTa Liu et al. (2019), ELECTRA Clark et al. (2020) and ALBERT Lan et al. (2019). The models differ in training, training data, and model size. BERT is trained using masked language modeling and next sentence prediction. RoBERTa omits the second task, but uses much more training data. Two models are trained for ELECTRA: one on masked language modeling (generator) and a second one that recognizes just these replaced tokens (discriminator). Since many of our evaluations need mask tokens, we only use the generator model for the evaluations. Finally, ALBERT is trained to predict the order of pairs of consecutive text segments in addition to masked language modeling.

**Causal Language Models (CLM)** are trained to predict the next word for a given input text. From this class we experiment with GPT-2 Radford et al. (2019), GPT-Neo Gao et al. (2021); Black et al. (2021) and GPT-J Wang and Komatsuzaki (2021). GPT-Neeo and GPT-J are re-implementations of GPT-3 Brown et al. (2020) where GPT-J was trained on a significantly larger data set named _The Pile_Gao et al. (2021) (cf. Table 7 in the appendix).

### Similarity Measures

To compute similarities of word associations based on the models studied here, we make use of research on biases in such models. These approaches calculate biases between two groups of concepts with respect to candidate groups of attributes. To this end, associations are evaluated by computing the similarities of vector representations of concepts and attributes. We adopt this approach to investigate our research question. However, as we consider knowledge extraction starting from source words (e.g. _toaster_, _shower_) in relation to target words (e.g. _kitchen_, _bathroom_), we modify it as described below.

#### 4.3.1 Cosine and Correlation Measures

Based on the human implicit association test Greenwald et al. (1998), WEAT Caliskan et al. (2017) is originally designed to compare the association between two sets of concepts (\(X\) and \(Y\)) and two sets of attributes (\(A\) and \(B\)). The degree of bias is calculated as follows:

\[s(X,Y,A,B)=\sum_{x\in X}s(x,A,B)-\sum_{y\in Y}s(y,A,B) \tag{1}\]

\[s(w,A,B)=\sum_{a\in A}\cos{(w,a)}-\sum_{b\in B}\cos{(w,b)} \tag{2}\]

Since we are considering source words in relation to target words, we use the following variant:

\[s(X,A)=\frac{1}{|X||A|}\sum_{x\in X}\sum_{a\in A}\cos{(x,a)} \tag{3}\]

For contextualized models, we adopt the approach of May et al. (2019), that is, we generate sentences such as "This is a {x}:" or "A {x} is here". All templates used in our study are listed in the appendix Table 8. However, instead of using the BERT token [CLS] (the default token at the beginning of an input sequence, which often serves as the default representation of the entire sequence), we use the maximum of the vector representations of all subwords of the expression. This approach is suitable for models like RoBERTa that do not use the [CLS] token for training, or the GPT models that do not have this token at all. In addition, we also achieved slightly better results on regular BERT models using this approach. We explain this with the fact that our focus is actually only on single tokens and that the vector representation of the [CLS] token often focuses only on a few dimensions Zhou et al. (2019). Our approach results in a set of contextualized representations for each source and target word, which are then compared using formula 3. We were able to obtain better results in our experiments with this representation than with those generated via the [CLS] token. For static models, if there is no vector representation 

for a potential multiword expressions (MWE)5, the average of the vectors of their components is used. This representation yielded the largest bias in the work of Azarpanah and Farhadloo (2021). For the static models, we also experimented with _distance correlation_(Szekely et al., 2007), _Pearson correlation_(Benesty et al., 2009), _Spearman correlation_(Kokoska and Zwillinger, 2000), _Kendall's tau_(Kendall, 1938) and _Mahalanobis distance_(Mahalanobis, 1936) - cf. Torregrossa et al. (2020); Azarpanah and Farhadloo (2021) - of the word vectors. Due to space limitations, only the values of the distance correlation and Kendall's tau are shown (see Table 1); the other correlation measures behave similarly. Moreover, the values for these measures tend to perform worse for contextualized models. This observation is consistent with findings of Azarpanah and Farhadloo (2021) where the Mahalanobis distance measure performed worst.

Footnote 5: Word2Vec contains vectors for MWE’s.

#### 4.3.2 Increased Log Probability

The cosine measure has shown to be problematic for assessing bias in contextualized models such as BERT (May et al., 2019; Kurita et al., 2019). Kurita et al. (2019) have therefore developed a new approach for models trained using masked language modeling. They weight the probability of a target word in a simple sentence template, assuming that an attribute is given or not:

\[{\rm score}(target,attribute)=\] \[\log\frac{P(|{\bf MASK}|=[target]\mid|{\bf MASK}|is~{}a~{}[ attribute])}{P(|{\bf MASK}_{i}|=[target]\mid|{\bf MASK}_{i}|is~{}a~{}[{\bf MASK}_{i}])}\]

Experiments show that the values of this measure correlate significantly better with human biases.

Since this measure is based on the context sensitivity of models, it cannot be applied to static models. For contextualized models, we use the probability of the last token (e.g. _curtain_ in the case of _shower curtain_) for source-forming MWEs and the first token (e.g. _living_ in the case of _living room_) for target-forming MWEs. We also performed experiments with multiple masks, one for each of the components of a MWE. However, this did not produce better results. We adapt this approach for causal language models as follows: Instead of a complete sentence, we use incomplete sentence templates such as "A(n) {object} is usually in the ..." or "In the {room} is usually a/an ...". The model should then predict the next token. Instead of masking the seed word, a neutral equivalent is used for calculation:

\[\begin{array}{c}\textit{A(n) {\{object\}} is usually in the ...}\\ \Downarrow\\ \textit{This is usually in the ....}\end{array}\]

Instead of performing the analysis in only one direction, we determine the score for both the target and the source given the other.

#### 4.3.3 Classifier-based Measures

In addition to the previously described measures, we experiment with classifiers. To this end, we train three classifiers on the model representations of the source word to determine the associated target word as a class label (e.g. predict _kitchen_, given the vector of _frying pan_). We generate the set of source word representations \(X\) in the same way as in the case of the cosine measure (see Section 4.3.1) and average them before classification:

\[target=\textit{Classifier}\left(\frac{1}{|X|}\sum_{\vec{x}\in X}\vec{x}\right)\]

The training runs on a leave-one-out cross-validation repeated 100 times. The target vector was then generated from the counted predicted classes (see Figure 1(b) in Appendix). We trained a \(k\)-nearest neighbors classifier with \(k=5\) (KNN), an SVM with a linear kernel and a feed-forward network (FFN). A small hyperparameter optimization was performed for the FFN, which resulted in the following parameters: Adam Optimizer (Kingma and Ba, 2014) with a learning rate of \(0.01\) over \(100\) epochs and one hidden layer of size \(100\) and ReLU as activation function.

### Scoring Measures and Classifiers

Given a word representation model, we compute the final score for the measures and classifiers to estimate how well they reconstruct the original probability distribution of the source entities relative to the target entities (see Table 4, 5, and 6). This is computed by the distance correlation (Szekely et al., 2007) between the target-source probability distributions and the corresponding association distributions of the respective measure or classifier. The advantage of the distance correlation over the Pearson correlation is that it can also measure non-linear relations. This was calculated both for all targets individually (correlation of all sources to one target) and then _concatenated_ for all targets 

together; we denote this variant by _CONC_. Therefore, _CONC_ does not correspond to the average of the individual distance correlations.

## 5 Experiments

Using the apparatus of Section 4, we now evaluate the classes of word representation models (static, MLMs and CLMs) in conjunction with the similarity measures and classifiers. The results for the static models are shown in Table 1, for the MLMs in Table 2 and for the CLMs in Table 3. Figure 2, 3 and 4 in Appendix show a visualization of the associations computed by means of cosine, masked-target & masked-source increased log similarity measures and the FFN classifier based on BERT-Large using the different datasets.

An experiment was also conducted with word frequencies via _Google Ngram6_ (see Section A.1 in the appendix).

Footnote 6: [https://books.google.com/ngrams](https://books.google.com/ngrams)

### Model-related Observations

The basic expectation that the cosine measure would generally perform the worst and the FFN classifier the best was met (see Tables 1-3). Interestingly, cosine is also outperformed by distance correlation in almost all cases.

Among the static models, GloVe and fastText performed best in most cases, especially on the room and part dataset (Table 1). Although Levy performs by far the worst in the room dataset, it keeps up with all classification results in the verb dataset. One reason for this could be the dependency-based learning strategy, which seems to work very well for verb associations, even though it was trained on a much smaller data set.

Among the masked-language models, BERT-Base surprisingly performed the best (Table 2). BERT-Large achieved the better Increased Log Probabilities, but the FFN classifier still worked better with the vector representations of the Base variant. This suggests that although associations are represented in a more fine-grained manner in BERT-Large, they are more difficult to retrieve due to the size of this model.

Among the masked-language models, GPT-J (which was trained with by far the largest training data) performs best (Table 3). Context-based models generally seem to determine the target given the source (\(P(\textit{targer}~{}|~{}\textit{source})\)) more easily than the reverse (\(P(\textit{source}~{}|~{}\textit{target})\)). With verbs, on the other hand, the reverse effect occurs. The GPT models show that the results for sources are better when weighted, while for targets the results are better without weighting.

In general, the SVM performed surprisingly well, even though only a linear kernel was used. But also the KNN method mostly performed better than the similarity measures. However, FFN performs best in all cases, outperforming cosine (worst case) by increases in the interval \([6\%,52\%]\) and outperforming the KNN approach (worst classifier) in each case by increases in the interval \([2\%,43\%]\).

### Dataset-related Observations

In terms of rooms, _bathroom_ scores the best, while _living room_ or _office_ usually score the worst. This may be because many bathroom objects are related to specific bathroom activities (e.g., toothbrush, bathtub), while objects that used to be located in the living room are increasingly found in other rooms (e.g., television in the bedroom). This would also explain why the results for _kitchen_ are also better.

On the part dataset, the static models actually performed significantly better than the contextualized models. This relates especially to GloVe and fastText which outperformed almost all contextualized models. Thus, static models are in some cases a good alternative to their contextualized counterparts. However, the more technical the objects become (here _mortise lock_ and _dishwasher_), the worse the results become.

On the verb dataset, the contextualized models perform minimally better. As mentioned earlier, these models can associate objects with verbs more easily than the other way around. Here, the largest difference in performance is observed in the case of Levy, where the results are almost equal to those of the other models, probably due to the learning strategy based on dependency trees.

In summary, knowledge extraction using language models, whether static or contextualized, is more effective using classifiers than using similarity measures commonly used in the field of bias research: there is potential for this type of knowledge extraction, but at the price of training classifiers - if one uses similarity measures instead, this knowledge is mostly out of reach.

### Relation Observation

All previous evaluations only examined associations between instances and concepts, but not whether the models represent their true relations.

 



\begin{table}
\begin{tabular}{l|c c

To fill this gap, we repeated the experimental setup of Kurita et al. (2019) for the room and part dataset on BERT-large, but this time masked the relation. The results are shown in Figure 1. Our selection of relations does not claim to be exhaustive, but serves as an illustration. It shows that while BERT-large is still very good at assigning objects _in_ rooms, the dominant relation predicted for parts is _used by_. This suggests that BERT has problems correctly assigning object parts, an observation that could explain its poorer results while being consistent with findings of Lin et al. (2020) (e.g., regarding counting parts).

## 6 Discussion

As good as the results obtained using classifiers are, they must be viewed with caution. One can attribute their success to the fine-tuning of numerous parameters (and ultimately to overfitting); however, one can also attribute this success to nonlinear structuring of the information encoded in language models. In other words, these models appear to encode object knowledge, but require a sophisticated apparatus to retrieve it. Thus, they should not be considered as an alternative to unsupervised approaches.

Another issue is that our experiments do not yet allow for a comparison of model _architectures_, as the models studied differ significantly in terms of the size of their parameter spaces and training data. Our experiments do suggest that certain smaller models come close to or even outperform the results of larger models. However, a comparison of model architectures would require controlling for these parameters. Nevertheless, the results we have obtained are, in part, promising enough to encourage such research.

Finally, our experiments show that static models can perform better than contextualized models to some extent. This finding is conditioned by our experiments and their context of application. These observations that _older_ models perform better on certain tasks are consistent with other work (e.g. LSTMs on small datasets for intent classification Ezen-Can (2020) or definiteness prediction Kabbara and Cheung (2021). At this point, a much broader analysis is needed (considering more areas and object relations), which also exploits contextual knowledge represented in contextualized models more than has been done here and in related work. Nevertheless, it is generally difficult to obtain data for such a broader analysis, and our experiments are already broader in scope and consider finer relationships than similar approaches.

## 7 Conclusion

We evaluated static and contextualized models as potential resources for object-related knowledge extraction. To this end, we examined three datasets (to identify typical artifacts in rooms, objects of actions, or parts of objects). We also experimented with different similarity measures and classifiers to extract the information contained in the language models. In doing so, we have shown that the models in combination with the measures differ greatly in terms of the amount of knowledge they allow for extracting. There is a weak trend that BERT-Base is the best performer among contextualized models, and GloVe and fastText among static models. Secondly, our results suggest that approaches based on classifiers perform significantly better than similarity measures. Thirdly, we have shown that static models perform almost as well as contextualized models - in some cases even better. This result shows that research on these models needs to be advanced. In future work we will also investigate

Figure 1: Small relation evaluation of BERT-large after the method of Kurita et al. (2019).

 

how grounded language models perform on such datasets. However, as noted above, this requires a significant expansion of bias research, such as that conducted here to enable knowledge extraction.

## Acknowledgements

The support by the _Stiftung Polytechnische Gesellschaft_ (SPTG) is gratefully acknowledged.

## References

* A. Azarpanah and M. Farhadloo (2021)Measuring biases of word embeddings: what similarity measures and descriptive statistics to use? In Proceedings of the First Workshop on Trustworthy Natural Language Processing, pp. 8-14. Cited by: SS3.
* Y. Belinkov and J. Glass (2019)Analysis methods in neural language processing: a survey. Transactions of the Association for Computational Linguistics7, pp. 49-72. External Links: Document, ISSN 0056-455X Cited by: SS1.
* E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell (2021)On the dangers of stochastic partots: can language models be too big?. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT '21, New York, NY, USA, pp. 610-623. External Links: ISBN 978-1-4503-3213-1, Link, Document Cited by: SS1.
* J. Benesty, J. Chen, Y. Huang, and I. Cohen (2009)Pearson correlation coefficient. In Noise reduction in speech processing, pp. 1-4. Cited by: SS1.
* S. Black, G. Leo, P. Wang, C. Leahy, and S. Biderman (2021)GPT-Neo: large scale autoregressive language modeling with mesh-tensorflow. If you use this software, please cite it using these metadata. Cited by: SS1.
* R. Bommasani, K. Davis, and C. Cardie (2020)Interpreting pretrained contextualized representations via reductions to static Embeddings. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 4758-4781. External Links: Link, Document Cited by: SS1.
* Z. Bouraoui, J. Camacho-Collados, and S. Schockaert (2020)Inducing relational knowledge from bert. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34, pp. 7456-7463. External Links: Link, Document Cited by: SS1.
* T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, Arvind Neelakantan, P. Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei (2020)Language models are few-shot learners. In Advances in Neural Information Processing Systems, Vol. 33, pp. 1877-1901. External Links: Link, Document Cited by: SS1.
* A. Caliskan, J. J. Bryson, and A. Narayanan (2017)Semantics derived automatically from language corpora contain human-like biases. Science356 (6334), pp. 183-186. External Links: ISSN 0037-8716, Link, Document Cited by: SS1.
* A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu (2015)ShapeNet: an information-rich 3d model repository. Technical report arXiv:1512.03012 [cs.GR]. Note: Stanford University -- Princeton University -- Toyota Technological Institute at Chicago External Links: Link Cited by: SS1.
* C. X. Chu, N. Tandon, and G. Weikum (2017)Distilling task knowledge from how-to communities. In Proceedings of the 26th International Conference on World Wide Web, pp. 805-814. Cited by: SS1.
* K. Clark, U. Khandelwal, O. Levy, and C. D. Manning (2019)What does BERT look at? an analysis of BERT's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Florence, Italy, pp. 276-286. External Links: Link, Document Cited by: SS1.
* K. Clark, M. Luong, Quoc V. Le, and C. D. Manning (2020)ElectRA: pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555. Cited by: SS1.
* J. Da and J. Kasai (2019)Cracking the contextual commonsense code: understanding commonsense reasoning aptitude of deep contextual representations. CoRRabs/1910.01157. External Links: Link, 1910.01157 Cited by: SS1.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pp. 4171-4186. External Links: Link, Document Cited by: SS1.
* K. Ethayarajh (2019)How contextual are contextualized word representations? comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 55-65. External Links: Link, Document Cited by: SS1.

 

* Ezen-Can (2020) Aysu Ezen-Can. 2020. A comparison of LSTM and BERT for small corpus. _CoRR_, abs/2009.05451.
* Forbes et al. (2019) Maxwell Forbes, Ari Holtzman, and Yejin Choi. 2019. Do neural language representations learn physical commonsense? In _CogSci_.
* Fu et al. (2020) Huan Fu, Bowen Cai, Lin Gao, Lingxiao Zhang, Cao Li, Qixun Zeng, Chengyue Sun, Yiyun Fei, Yu Zheng, Ying Li, Yi Liu, Peng Liu, Lin Ma, Le Weng, Xiaohang Hu, Xin Ma, Qian Qian, Rongfei Jia, Binqiang Zhao, and Hao Zhang. 2020. 3d-front: 3d furnished rooms with layouts and semantics. _arXiv preprint arXiv:2011.09127_.
* Gao et al. (2021) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2021. The pile: An 800gb dataset of diverse text for language modeling. _CoRR_, abs/2101.00027.
* Gladkova et al. (2016) Anna Gladkova, Aleksandr Drozd, and Satoshi Matsuoka. 2016. Analogy-based detection of morphological and semantic relations with word embeddings: What works and what doesn't. In _Proceedings of the NAACL-HLT SRW_, pages 47-54, San Diego, California, June 12-17, 2016. ACL.
* Greenwald et al. (1998) Anthony G Greenwald, Debbie E McGhee, and Jordan LK Schwartz. 1998. Measuring individual differences in implicit cognition: the implicit association test. _Journal of personality and social psychology_, 74(6):1464.
* Gupta and Jaggi (2021) Prakhar Gupta and Martin Jaggi. 2021. Obtaining better static word embeddings using contextual embedding models. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 5241-5253, Online. Association for Computational Linguistics.
* Hayward and Tarr (1995) William G Hayward and Michael J Tarr. 1995. Spatial language and spatial representation. _Cognition_, 55(1):39-84.
* Heinzerling and Inui (2021) Benjamin Heinzerling and Kentaro Inui. 2021. Language models as knowledge bases: On entity representations, storage capacity, and paraphrased queries. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 1772-1791, Online. Association for Computational Linguistics.
* Hewitt and Manning (2019) John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4129-4138, Minneapolis, Minnesota. Association for Computational Linguistics.
* Hupkes et al. (2020) Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. 2020. Compositionality decomposed: How do neural networks generalise? (extended abstract). In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20_, pages 5065-5069. International Joint Conferences on Artificial Intelligence Organization. Journal track.
* Jiang et al. (2020) Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How Can We Know What Language Models Know? _Transactions of the Association for Computational Linguistics_, 8:423-438.
* Kabbara and Cheung (2021) Jad Kabbara and Jackie Chi Kit Cheung. 2021. Post-editing extractive summaries by definiteness prediction. In _Findings of the Association for Computational Linguistics: EMNLP 2021_, pages 3682-3692, Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Kendall (1938) Maurice G Kendall. 1938. A new measure of rank correlation. _Biometrika_, 30(1/2):81-93.
* Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6980_.
* Kokoska and Zwillinger (2000) Stephen Kokoska and Daniel Zwillinger. 2000. _CRC standard probability and statistics tables and formulae_. Crc Press.
* Kurita et al. (2019) Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. 2019. Measuring bias in contextualized word representations. In _Proceedings of the First Workshop on Gender Bias in Natural Language Processing_, pages 166-172, Florence, Italy. Association for Computational Linguistics.
* Lan et al. (2019) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. ALBERT: A lite BERT for self-supervised learning of language representations. _CoRR_, abs/1909.11942.
* Landau and Jackendoff (1993) Barbara Landau and Ray Jackendoff. 1993. "what" and "where" in spatial language and spatial cognition. _Behavioral and Brain Sciences_, 16(2):217-238.
* Levy and Goldberg (2014) Omer Levy and Yoav Goldberg. 2014. Neural word embedding as implicit matrix factorization. _Advances in neural information processing systems_, 27:2177-2185.
* Li et al. (2020) Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2020. What does BERT with vision look at? In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 5265-5275, Online. Association for Computational Linguistics.
* Lin et al. (2020) Bill Yuchen Lin, Seyeon Lee, Rahul Khanna, and Xiang Ren. 2020. Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-Trained Language Models. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 6862-6868, Online. Association for Computational Linguistics 

Tsuang-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In _European conference on computer vision_, pages 740-755. Springer.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. _CoRR_, abs/1907.11692.
* Mahalanobis (1936) Prasanta Chandra Mahalanobis. 1936. On the generalized distance in statistics. National Institute of Science of India.
* May et al. (2019) Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. On measuring social biases in sentence encoders. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 622-628, Minneapolis, Minnesota. Association for Computational Linguistics.
* Merrill et al. (2021) William Merrill, Yoav Goldberg, Roy Schwartz, and Noah A. Smith. 2021. Provable limitations of acquiring meaning from ungrounded form: What will future language models understand? _Transactions of the Association for Computational Linguistics_, 9:1047-1060.
* Mikolov et al. (2018) Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in pre-training distributed word representations. In _Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018)_.
* Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In _Advances in neural information processing systems_, pages 3111-3119.
* Mo et al. (2019) Kaichun Mo, Shilin Zhu, Angel X. Chang, Li Yi, Subarina Tripathi, Leonidas J. Guibas, and Hao Su. 2019. PartNet: A large-scale benchmark for fine-grained and hierarchical part-level 3D object understanding. In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_, pages 1532-1543.
* Petroni et al. (2020) Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rocktaschel, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. 2020. How context affects language models' factual predictions. In _Automated Knowledge Base Construction_.
* Petroni et al. (2019) Fabio Petroni, Tim Rocktaschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 2463-2473, Hong Kong, China. Association for Computational Linguistics.
* Radford et al. (2019) Alec Radford, Jeff Wu, R. Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
* Reif et al. (2019) Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B. Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019. Visualizing and measuring the geometry of BERT. In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_, pages 8592-8600.
* Rogers et al. (2020) Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTokogy: What we know about how BERT works. _Transactions of the Association for Computational Linguistics_, 8:842-866.
* Shin et al. (2020) Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting knowledge from language models with automatically generated prompts. In _Empirical Methods in Natural Language Processing (EMNLP)_.
* Silberman et al. (2012) Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. 2012. Indoor segmentation and support inference from rgbd images. In _European conference on computer vision_, pages 746-760. Springer.
* Sileo (2021) Damien Sileo. 2021. Visual grounding strategies for text-only natural language processing. In _Proceedings of the Third Workshop on Beyond Vision and LANguage: intFgrating Real-world kNowledge (LANTERN)_, pages 19-29, Kyiv, Ukraine. Association for Computational Linguistics.
* Song et al. (2017) Shuran Song, Fisher Yu, Andy Zeng, Angel X. Chang, Manolis Savva, and Thomas Funkhouser. 2017. Semantic scene completion from a single depth image. _Proceedings of 30th IEEE Conference on Computer Vision and Pattern Recognition_.
* Szekely et al. (2007) Gabor J Szekely, Maria L Rizzo, and Nail K Bakirov. 2007. Measuring and testing dependence by correlation of distances. _The annals of statistics_, 35(6):2769-2794.
* Tenney et al. (2019) Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durre, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. _CoRR_, abs/1905.06316 

Francois Torregrossa, Vincent Claveau, Nihel Kooli, Guillaume Gravier, and Robin Allesiardo. 2020. On the correlation of word embedding evaluation metrics. In _Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020)_, pages 4789-4797.
* Tversky and Gati [2004] Amos Tversky and Itamar Gati. 2004. Studies of similarity. In Eldar Shafir, editor, _Preference, Belief, and Similarity. Selected Writing of Amos Tversky_, pages 75-95. MIT Press.
* Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Proceedings of the 31st International Conference on Neural Information Processing Systems_, NIPS'17, page 6000-6010, Red Hook, NY, USA. Curran Associates Inc.
* Vylomova et al. [2016] Ekaterina Vylomova, Laura Rimell, Trevor Cohn, and Timothy Baldwin. 2016. Take and took, gaggle and goose, book and read: Evaluating the utility of vector differences for lexical relation learning. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1671-1682, Berlin, Germany. Association for Computational Linguistics.
* Wang and Komatsuzaki [2021] Ben Wang and Aram Komatsuzaki. 2021. GPTJ-GB: A 6 Billion Parameter Autoregressive Language Model. [https://github.com/kingoflolz/mesh-transformer-jax](https://github.com/kingoflolz/mesh-transformer-jax).
* Warstadt et al. [2020] Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohannaney, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The benchmark of linguistic minimal pairs for English. _Transactions of the Association for Computational Linguistics_, 8:377-392.
* Yao et al. [2019] Liang Yao, Chengsheng Mao, and Yuan Luo. 2019. Kgbert: Bert for knowledge graph completion. _arXiv preprint arXiv:1909.03193_.
* Zhang et al. [2017] Li Zhang, Jun Li, and Chao Wang. 2017. Automatic synonym extraction using word2vec and spectral clustering. In _2017 36th Chinese Control Conference (CCC)_, pages 5629-5632. IEEE.
* Zhou et al. [2019] Wenxuan Zhou, Junyi Du, and Xiang Ren. 2019. Improving bert fine-tuning with embedding normalization. _arXiv preprint arXiv:1911.03918_.

## Appendix A Appendix

A tabular breakdown of the datasets used can be seen in Table 4, 5 and 6. The exact models used are listed in Table 7. The heatmap visualizations for the other two datasets are in Figure 3 and 4.

 



\begin{table}
\begin{tabular}{l c|c c|c c|c c|c c|c c|c c} \hline \hline  & **eat** & & \multicolumn{2}{c|}{**listen to**} & \multicolumn{2}{c|}{**play**} & \multicolumn{3}{c|}{**read**} & \multicolumn{2}{c|}{**wash with**} & \multicolumn{2}{c}{**wear**} \\ object & score & object & score & object & score & object & score & object & score & object & score \\ \hline food & 0.13 & music & 0.22 & game & 0.27 & book & 0.08 & soap & 0.29 & clothing & 0.07 \\ diet & 0.08 & song & 0.03 & music & 0.06 & label & 0.06 & water & 0.29 & glove & 0.06 \\ meal & 0.07 & body & 0.03 & note & 0.04 & instruction & 0.05 & vinegar & 0.04 & shoe & 0.05 \\ breakfast & 0.04 & side & 0.02 & sport & 0.03 & review & 0.04 & solution & 0.03 & clothes & 0.05 \\ balanced diet & 0.03 & partner & 0.02 & chord & 0.02 & body language & 0.02 & detergent & 0.03 & shirt & 0.02 \\ fruit & 0.03 & child & 0.02 & song & 0.02 & rule & 0.02 & baking soda & 0.03 & makeup & 0.02 \\ vegetable & 0.03 & perspective & 0.02 & video game & 0.02 & example & 0.02 & cream & 0.02 & gear & 0.02 \\ plenty & 0.03 & response & 0.02 & card & 0.02 & complaint & 0.01 & shampoo & 0.02 & boot & 0.02 \\ protein & 0.03 & parent & 0.02 & role & 0.02 & law & 0.01 & towel & 0.02 & dress & 0.02 \\ snack & 0.02 & people & 0.02 & video & 0.02 & story & 0.01 & cold water & 0.02 & socket & 0.02 \\ \hline \hline \end{tabular}
\end{table}
Table 6: A subset of verb-object relations extracted from an updated version of HowToKB.

\begin{table}
\begin{tabular}{l l|l l r|l l l|l l l|l l} \hline \hline  & **bathroom** & \multirow{2}{*}{score} & \multicolumn{2}{c|}{**bedroom**} & \multicolumn{2}{c|}{**kitchen**} & \multicolumn{2}{c|}{**living room**} & \multicolumn{2}{c}{**office**} \\ object & & & & & & & & & & & & & \\ score & object & score & object & score & object & score & object & score & object & score \\ \hline toilet & 1.00 & dresser & 1.00 & drying rack & 1.00 & coffee table & 0.94 & whiteboard & 1.00 \\ bathtub & 1.00 & night stand & 1.00 & kitchen island & 1.00 & ottoman & 0.93 & room divider & 0.94 \\ toothwalk holder & 1.00 & headboard & 1.00 & pot & 1.00 &replace & 0.87 & stapler & 0.92 \\ toothspaste & 1.00 & bed & 0.97 & frying pan & 1.00 & dvd player & 0.69 & cook board & 0.92 \\ shower certain & 1.00 & alarm clock & 0.97 & sprice rack & 1.00 & sofa & lei & 0.88 \\ toothbrush & 0.97 & laundry basket & 0.86 & cutting board & 1.00 & decorative plate & 0.61 & keyboard & 0.85 \\ towed rod & 0.96 & hat & 0.74 & blender & 1.00 & vt standard & 0.57 & mouse & 0.84 \\ loilet paper & 0.96 & doll & 0.70 & knife & 1.00 & blanket & 0.55 & pen & 0.83 \\ squeeze tube & 0.95 & stuffed animal & 0.60 & stove & 0.98 & television & 0.53 & computer & 0.82 \\ faucet handle & 0.82 & pillow & 0.56 & dishwasher & 0.97 & remote control & 0.50 & column & 0.81 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Statistics generated from ScanNet using NYU categories: _score_ is the conditional probability \(P(\mathit{room}\mid object)\) of the room given the object based on the frequencies observable in NYU.

\begin{table}
\begin{tabular}{l l|l l l|l l l|l l l|l l} \hline \hline  & **bed** & & \multicolumn{2}{c|}{**dishwather**} & \multicolumn{2}{c|}{**door**} & \multicolumn{2}{c|}{**mortise lock**} & \multicolumn{2}{c}{**refrigator**} & \multicolumn{2}{c}{**tollet**} \\ object & score & object & score & object & score & object & score & object & score & object & score \\ \hline pillow & 1.00 & drain house & 1.00 & lock & 1.00 & img & 1.00 & switch & 1.00 & valve seat shaft & 1.00 \\ better & 1.00 & overflow protection & 1.00 & corneric & 1.00 & keypay & 1.00 & refrigerator compartment & 1.00 & talk lid & 1.00 \\ manness cover & 1.00 & uh & 1.00 & hanging single & 1.00 & center pin & 1.00 & egg stay & 1.00 & conical weather & 1.00 \\ leg & 1.00 & pump & 1.00 & ophthalmic & 1.00 & spring & 1.00 & shelf demand & 1.00 & bit chain & 1.00 \\ best string & 1.00 & gasket & 1.00 & top rail & 1.00 & river & 1.00 & mungraphic parket & 1.00 & sweat & 1.00 \\ handboard & 1.00 & water base & 1.00 & middle panel & 1.00 & cylinder case & 1.00 & storage door & 1.00 & shutorf valve & 1.00 \\ matines & 1.00 & hahening element & 1.00 & bottom rail & 1.00 & key & floorcard & 1.00 & up level & 1.00 \\ pillow protector & 1.00 & rack & 1.00 & pand & 1.00 & flexemble & 1.00 & guard rail & 1.00 & ball-clock & supply & 1.00 \\ elastic & 1.00 & calvery habet & 1.00 & jamb & 1.00 & dead bus & 1.00 & crimger & 1.00 & table bowl & 1.00 \\ footboard & 1.00 & wash tower & 1.00 & downloads & 1.00 & cylinder & 1.00 & glass server & 1.00 & flush handler & 1.00 \\ motor & 1.00 & threaded & 1.00 & visitor & 1.00 & buffer compartment & 1.00 & walk sed & 1.00 \\ adergent dispense & 1.00 & authentichand & 1.00 & strike plane & 1.00 & thermostat control & 1.00 & tank hall & 1.00 \\ slide & 1.00 & lock rail & 1.00 & frezer compartment & 1.00 & float ball & 1.00 \\ levering foot & 1.00 & showing safe & 1.00 & & ice cube tray & 1.00 & filter bete & 1.00 \\ insulating material & 1.00 & herafer & 1.00 & and more kepper & 1.00 & waste pipe & 1.00 \\ spray arm & 1.00 & & door stop & 1.00 & seat over & 1.00 \\ infinue-aid dipremer & 1.00 & & and hold & 1.00 & cold-water & supply & 1.00 \\ line & & line & line & & & & & & \\ voegthae & & & & & & & & & & \\ toilet & 1.00 & turnp & 1.00 & runp & 1.00 \\ plenty & 1.00 & response & 1.00 & card & 0.02 & complaint & 0.01 & shampoo & 0.02 & boost & 0.02 \\ protein & 0.03 & parent & 0.02 & role & 0.02 & law & 0.01 & towel & 0.02 & dress & 0.02 \\ snack & 0.02 & people & 0.02 & video & 0.02 & story & 0.01 & cold water & 0.02 & socket & 0.02 \\ \hline \hline \end{tabular}
\end{table}
Table 5: A subset of part-whole relations extracted from _Online-Bildwörterbuch_. All parts have a value of _1.00_ in our data set, because they only occur with this object.





\begin{table}
\begin{tabular}{l|c|l|r|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l l|l|l|l|l|l|l} \hline Model & \multicolumn{12}{l|}{Specification} & \multicolumn{14}{l|}{Dimension} & \multicolumn{16}{l|}{Parameters} & \multicolumn{17}{l|}{Dataset Size} & URL \\  & \multicolumn{18}{l|}{(T: S)} & \multicolumn{19}{l|}{URL} \\ \hline word2vec & GoogleNews-vectors-negative300 & 300 & - & - & \multicolumn{1}{l|}{100B -} & [https://code.google.com/archive/pword2vec/Glove](https://code.google.com/archive/pword2vec/Glove) & Common Crawal - glove,8d08.30od & 300 & - & \multicolumn{1}{l|}{840B -} & [https://mlp.stanford.edu/projects/glovew/Levy](https://mlp.stanford.edu/projects/glovew/Levy) & Dependency-Based Words & 300 & - & English Wiki & [https://levyomer.wordpress.com/2014/04/25/](https://levyomer.wordpress.com/2014/04/25/)

Figure 2: Heatmap of source-object associations based on BERT-Large and the room dataset. The objects (sources) on the y-axis are grouped by the room in which they are most likely to be located according to the _NYU Depth V2 Dataset_.

 

* [14] M. C. Bender, J. R. Friedman, and A. S. Gheorghi, "Theoretical Population of the Universe: The Role of Life in the Early Universe," _Astrophys. J._**630**(2), 579-804

[MISSING_PAGE_POST] 

* [14] M. C. Bader, A. D. Krizhevsky, and J. L. Friedman. The impact of the deep learning model on the performance of a large dataset. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 235-240, June 2016.
* [15] S. G. Hinton, R. H. Harrell, and P. I. Jordan. Learning with low variance. _Journal of Machine Learning Research_**78**:199-202, 2000.
* [16] T. Hastie, N. Kaiser, and E. Silverstein. Deep residual learning for image recognition. _arXiv preprint arXiv:1412.6090_**, 2014.
* [17] Y. Bengio et al. Generative adversarial nets. _Advances in neural information processing systems_**46**:1553-1562, 2015.
* [18] W. Dong, X. Liu, Z. Wang, and Q. Weinberger. Unsupervised learning by jointly optimizing generative models. In _NIPS_, volume 1, pages 1099-1109, 2016.
* [19] Y. Bengio, V. LeCun, and O. Vitells. Learning from incomplete data via latent space. _CoRR_**100**, 2014.
* [20] Y. Bengio, A. Courville, and J. Schmidhuber. Learning representations using convolutional networks. In _NIPS_, volume 1, pages 1099-1110, 2016.

[MISSING_PAGE_POST]