SlowGuess/NAACL_2022_OCR huggingface dataset:
Alexander Henlein and Alexander Mehler  
Text Technology Lab, Goethe-University Frankfurt, Germany  
{henlein, mehler}@em.uni-frankfurt.de
Transformer-based models are now predominant in NLP. They outperform approaches based on static models in many respects. This success has in turn prompted research that reveals a number of biases in the language models generated by transformers. In this paper we utilize this research on biases to investigate to what extent transformer-based language models allow for extracting knowledge about object relations ( $X$  occurs in  $Y$ ;  $X$  consists of  $Z$ ; action  $A$  involves using  $X$ ). To this end, we compare contextualized models with their static counterparts. We make this comparison dependent on the application of a number of similarity measures and classifiers. Our results are threefold: Firstly, we show that the models combined with the different similarity measures differ greatly in terms of the amount of knowledge they allow for extracting. Secondly, our results suggest that similarity measures perform much worse than classifier-based approaches. Thirdly, we show that, surprisingly, static models perform almost as well as contextualized models – in some cases even better.
Few models have recently influenced NLP as much as transformers (Vaswani et al., 2017). Hardly any new NLP system today is introduced without a transformer-based model such as BERT (Devlin et al., 2019) or GPT (Radford et al., 2019). As a result, static models such as word2vec (Mikolov et al., 2013) are increasingly being substituted. Nevertheless, transformers are still far from being fully understood. Thus, research studies are being conducted to find out how they work and what properties the language models they generate have.
During training, transformers seem to capture both syntactic and semantic features (Rogers et al., 2020). For example, dependency trees can be reconstructed from trained attention heads (Clark et al., 2019), syntactic trees can be reconstructed
from word encodings (Hewitt and Manning, 2019), and these encodings can be clustered into representations of word senses (Reif et al., 2019). BERT also seems to encode information about entity types and semantic roles (Tenney et al., 2019). For an overview of this research see Rogers et al. (2020).
Since BERT and other transformers are trained on various data crawled from the internet, they are sensitive to biases (Caliskan et al., 2017; May et al., 2019; Bender et al., 2021). In practice, instead of reproducing negative biases, they are expected to allow for the derivation of statements, such as that toothbrushes are spatially associated with bathrooms rather than living rooms. In this line of thinking, approaches such as the popularization of knowledge graphs can be located (Yao et al., 2019; Petroni et al., 2019; Heinzerling and Inui, 2021). Our paper is situated in this context. More specifically, we examine the extent to which knowledge about spatial objects and their relations is implicitly encoded in these models. Since the underlying texts are rather implicit regarding such information, it can be assumed that the object relations derivable from transformers are weakly encoded (cf. Landau and Jackendoff, 1993; Hayward and Tarr, 1995). Reading, for example, the sentence:
"After getting up, I ate an apple"
one may assume that the narrator got up from his bed in the bedroom, went to the kitchen, took an apple, washed it in the sink, and finally ate it. The apple is also likely to have been peeled and cut. Equally, however, nothing is said in the sentence about a bedroom or a kitchen. Nevertheless, it is a well known approach to explore the usage regularities of words, currently most efficiently represented by neural networks, as a source for knowledge extraction (see, e.g. Zhang et al., 2017; Bouraoui et al., 2020; Shin et al., 2020; Petroni et al., 2019).
In this work, we use a number of methods to identify biases in contextualized models and ask to what extent they can be used to extract object
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5791 - 5807 July 10-15, 2022 ©2022 Association for Computational Linguistics
based knowledge from these models. To this end, we consider three relations:
1. Spatial containment of (source) objects in (target) rooms: e.g. a fridge probably belongs in a kitchen, but not in a living room;
2. Parts (source) in relation to composite objects (target): e.g. a refrigerator compartment is probably a part of a fridge;
3. Objects (source) in relation to actions (target) that involve them: e.g. reading involves something being read, e.g., a book.
Regarding these relations, we examine a set of pretrained contextualized and static word representation models. This is done to answer the question to what extent they allow the extraction of instances of these relations when trained on very large datasets. We focus on rather common terms (kitchen, to read etc.) as part of the general language.
It is assumed that (static or contextualized) models implicitly represent such relations, so that it is possible to identify probable targets starting from certain sources. That is, for a word like fridge (source), we expect it to be semantically more strongly associated with kitchen (target) than with words naming other rooms, since fridges are more likely to be found in kitchens than in other rooms, and that certain word representation models reflect this association. We also assume that this association is asymmetric and exists to a lesser extent from target to source (cf. Tversky and Gati (2004)).
The paper is organized as follows: Related work is reported in Sec. 2. The datasets we use are represented in Sec. 3 and our method in Sec. 4. Our experiments are presented in Sec. 5 and discussed in Sec. 6. Sec. 7 provides a conclusion. All used data, scripts and results are open source on GitHub<sup>1</sup>.
Biases in NLP models are not a new problem that appeared with BERT, but affect almost all models trained on language datasets (Caliskan et al., 2017). As such, there are methods for measuring social biases in static models such as word2vec (Mikolov et al., 2013). One of the best known approaches is WEAT (Caliskan et al., 2017). Here, two groups of concepts are compared with two groups of attributes based on the difference between the sums of their cosine similarities (see Section 4). This approach already points to a methodological premise
that also guides our work: Relations of entities are tentatively determined by similarity analyses of vectorial word representations.
However, a direct comparison of word vectors is not possible with contextualized methods such as BERT, where the vector representation of a word varies with the context of its occurrence (cf. Ethayarajh, 2019). Efforts to transfer the cosine-based approach from static to contextualized models have not been able to recreate similar performances (May et al., 2019). Therefore, new approaches have been developed based on the specifics of contextualized models. For example, BERT is trained using masked language modeling, where the model estimates the probability of masked words in sentences (Devlin et al., 2019). The probability distribution for a masked word in a given context can then be used as information to characterize candidate words (Kurita et al., 2019). Sec. 4.3 describes this approach in more detail. An alternative approach is to examine the interpretability of models (Belinkov and Glass, 2019; Jiang et al., 2020; Petroni et al., 2019, 2020; Bommasani et al., 2020; Hupkes et al., 2020), which goes beyond the scope of this paper. In any event, both approaches share the same basic ideas, e.g., the probability prediction of mask tokens (cf. Kurita et al., 2019; Belinkov and Glass, 2019).
Work has also been done on how BERT represents information about spatial objects. For example, BERT has problems with certain object properties (e.g. cheap or cute) or implicit visual properties that are rarely expressed (Da and Kasai, 2019). Problems are also encountered with extracting numerical commonsense knowledge, such as the typical number of tires on a car or the feet on a bird (Lin et al., 2020). More than that, the models seem to allow for extracting some object knowledge, but not with respect to properties based on their affordance (e.g. objects through which one can see are transparent (Forbes et al., 2019)). Even though these results seem to question the use of BERT and its competitors for knowledge extraction, these models still perform better in downstream tasks than their static competitors (Devlin et al., 2019; Liu et al., 2019; Brown et al., 2020; Da and Kasai, 2019). Bouraoui et al. (2020) compared these models using different datasets and lexical relations. These include relations similar to those examined here (e.g. a pot is usually found in a kitchen), but beyond the level of detail achieved in
<https://github.com/texttechnologylab/SpatialAssociationsInLM>
our study.
What will become increasingly important is the so-called grounding of language models (Merrill et al., 2021): Here, the models are trained not only on increasingly large text data, but also, for example, on images thus enabling better "understanding" of spatial relations (Sileo, 2021; Li et al., 2020). In this paper, we focus on models without grounding.
3 Datasets Used for Evaluation
3.1 Spatial Containment
The NYU Depth V2 Dataset (Silberman et al., 2012) consists of video sequences of numerous indoor scenes. It features 464 labeled scenes using a rich category set. We use this dataset as a basis for evaluating the probability of occurrence of objects in rooms (e.g. kitchen, living room, etc.). That is, we estimate the conditional probability  $P(r \mid o)$  of a room  $r$  (target) given an object  $o$  (source). In this way, we aim to measure the strength of an object's association with a particular room as reflecting the corresponding spatial containment relation. At the same time, we want to filter out objects such as window that are evenly distributed among the rooms studied here. In our experiments, we consider the ten most frequently mentioned objects in NYU to associate with the five most frequently mentioned spaces. This data is shown in the Table 4 (appendix).
The advantage of NYU over other scene datasets such as 3D-Front (Fu et al., 2020) is that it deals with real spaces and not artificially created ones. In addition, NYU's object category set is relatively fine-grained (we counted 895 different object names) and uses colloquial terms. This is in contrast to, for example, SUNCG (Song et al., 2017) (with categories like "slot machine with chair", "range hood with cabinet", "food processor") and ShapeNetCore (Chang et al., 2015) with only 55 object categories or COCO (Lin et al., 2014) with 80 object categories. This makes NYU more suitable for our task of evaluating word representation models as resources for knowledge extraction starting from general language.
3.2 Part-whole Relations
We use a subset of the object descriptions from Online-Bildwörterbuch<sup>2</sup>. This resource describes very fine-grained part-whole relations of objects
expressed by colloquial names, in contrast to, e.g., PartNet (Mo et al., 2019) where one finds labels such as seat single surface or arm near vertical bar. The list of objects from Online-Bildwörterbuch used in our study and their subdivisions is shown in Table 5. The selected objects were chosen by hand, provided that the chosen examples are general enough and the subdivision is sufficiently fine.
3.3 Action-object Relations
To study entities as typical objects of certain actions, we derive a dataset from HowToKB (Chu et al., 2017) which is based on WikiHow<sup>3</sup>. In HowToKB, task frames, temporal sequences of subtasks, and attributes for involved objects were extracted from WikiHow articles. Some changes were made to the knowledge database, including a newly crawled version of WikiHow. In addition, the pre-processing tools have been updated and partially extended (see Table 6).
3.3.1 Related Datasets
For evaluating static models, there are datasets and approaches to measuring lexical relations, such as DiffVec (Vylomova et al., 2016), BATS (Gladkova et al., 2016) or BLiMP (Warstadt et al., 2020). Although these datasets are also used to evaluate BERT (Bouraoui et al., 2020), they represent only an unstructured subset of the data we used and are thus not appropriate for our study.
We now present the static and contextualized models used in our study. Table 7 in the appendix lists these models and their sources. We also specify the measures used to compute word associations as a source of knowledge extraction, and describe how to use classifiers as an alternative to them.
4.1 Static Models
Probably the best known static model is word2vec (Mikolov et al., 2013). Its CBOW variant is trained to predict words in the context of their surrounding words. The word representations trained in this way partially encode semantic relations (Mikolov et al., 2013), making them a suitable candidate for comparison with the corresponding information values of contextualized word representations. In addition to word2vec, we consider GloVe (Pennington et al., 2014), Levy (Levy and Goldberg, 2014),
$^{2}$ http://www.bildwoerterbuch.com/en/home
<sup>3</sup>https://www.wikihow.com/
fastText (Mikolov et al., 2018) and a static BERT model (Gupta and Jaggi, 2021). Unlike window-based approaches to static embeddings, Levy embeddings are trained on dependency trees.
4.2 Contextualized Models
Unlike static models, the vector representations of (sub-)word (units) in contextualized models depend on the context in which they occur so that tokens of the same type may each be represented differently in different contexts. All contextual models we evaluate here are pre-trained and come from the huggingface models repository<sup>4</sup>. We evaluate two types of contextualized models:
Masked Language Models (MLM) are trained to reconstruct randomly masked words in input sequences. We experiment with BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), ELECTRA (Clark et al., 2020) and ALBERT (Lan et al., 2019). The models differ in training, training data, and model size. BERT is trained using masked language modeling and next sentence prediction. RoBERTa omits the second task, but uses much more training data. Two models are trained for ELECTRA: one on masked language modeling (generator) and a second one that recognizes just these replaced tokens (discriminator). Since many of our evaluations need mask tokens, we only use the generator model for the evaluations. Finally, ALBERT is trained to predict the order of pairs of consecutive text segments in addition to masked language modeling.
Causal Language Models (CLM) are trained to predict the next word for a given input text. From this class we experiment with GPT-2 (Radford et al., 2019), GPT-Neo (Gao et al., 2021; Black et al., 2021) and GPT-J (Wang and Komatsuzaki, 2021). GPT-Neo and GPT-J are re-implementations of GPT-3 (Brown et al., 2020) where GPT-J was trained on a significantly larger data set named The Pile (Gao et al., 2021) (cf. Table 7 in the appendix).
4.3 Similarity Measures
To compute similarities of word associations based on the models studied here, we make use of research on biases in such models. These approaches calculate biases between two groups of concepts with respect to candidate groups of attributes. To this end, associations are evaluated by computing the similarities of vector representations of con
cepts and attributes. We adopt this approach to investigate our research question. However, as we consider knowledge extraction starting from source words (e.g. toaster, shower) in relation to target words (e.g. kitchen, bathroom), we modify it as described below.
4.3.1 Cosine and Correlation Measures
Based on the human implicit association test (Greenwald et al., 1998), WEAT (Caliskan et al., 2017) is originally designed to compare the association between two sets of concepts ( $X$  and  $Y$ ) and two sets of attributes ( $A$  and  $B$ ). The degree of bias is calculated as follows:
$$
s (w, A, B) = \sum_ {a \in A} \cos (w, a) - \sum_ {b \in B} \cos (w, b) \tag {2}
$$
Since we are considering source words in relation to target words, we use the following variant:
$$
s (X, A) = \frac {1}{| X | | A |} \sum_ {x \in X} \sum_ {a \in A} \cos (x, a) \tag {3}
$$
For contextualized models, we adopt the approach of May et al. (2019), that is, we generate sentences such as "This is a  $\{\mathbf{x}\}$ " or "A  $\{\mathbf{x}\}$  is here". All templates used in our study are listed in the appendix Table 8. However, instead of using the BERT token [CLS] (the default token at the beginning of an input sequence, which often serves as the default representation of the entire sequence), we use the maximum of the vector representations of all subwords of the expression. This approach is suitable for models like RoBERTa that do not use the [CLS] token for training, or the GPT models that do not have this token at all. In addition, we also achieved slightly better results on regular BERT models using this approach. We explain this with the fact that our focus is actually only on single tokens and that the vector representation of the [CLS] token often focuses only on a few dimensions (Zhou et al., 2019). Our approach results in a set of contextualized representations for each source and target word, which are then compared using formula 3. We were able to obtain better results in our experiments with this representation than with those generated via the [CLS] token. For static models, if there is no vector representation
4https://huggingface.co/models
for a potential multiword expressions (MWE) $^5$ , the average of the vectors of their components is used. This representation yielded the largest bias in the work of Azarpanah and Farhadloo (2021). For the static models, we also experimented with distance correlation (Székely et al., 2007), Pearson correlation (Benesty et al., 2009), Spearman correlation (Kokoska and Zwillinger, 2000), Kendall's tau (Kendall, 1938) and Mahalanobis distance (Mahalanobis, 1936) - cf. Torregrossa et al. (2020); Azarpanah and Farhadloo (2021) - of the word vectors. Due to space limitations, only the values of the distance correlation and Kendall's tau are shown (see Table 1); the other correlation measures behave similarly. Moreover, the values for these measures tend to perform worse for contextualized models. This observation is consistent with findings of Azarpanah and Farhadloo (2021) where the Mahalanobis distance measure performed worst.
4.3.2 Increased Log Probability
The cosine measure has shown to be problematic for assessing bias in contextualized models such as BERT (May et al., 2019; Kurita et al., 2019). Kurita et al. (2019) have therefore developed a new approach for models trained using masked language modeling. They weight the probability of a target word in a simple sentence template, assuming that an attribute is given or not:
$$
\begin{array}{l} \operatorname {s c o r e} (\text {t a r g e t}, \text {a t t r i b u t e}) = \\ \log \frac {P ([ \mathrm {M A S K} ] = [ t a r g e t ] \mid [ \mathrm {M A S K} ] i s a [ a t t r i b u t e ])}{P ([ \mathrm {M A S K} _ {1} ] = [ t a r g e t ] \mid [ \mathrm {M A S K} _ {1} ] i s a [ \mathrm {M A S K} _ {2} ]} \\ \end{array}
$$
Experiments show that the values of this measure correlate significantly better with human biases.
Since this measure is based on the context sensitivity of models, it cannot be applied to static models. For contextualized models, we use the probability of the last token (e.g. curtain in the case of shower curtain) for source-forming MWEs and the first token (e.g. living in the case of living room) for target-forming MWEs. We also performed experiments with multiple masks, one for each of the components of a MWE. However, this did not produce better results. We adapt this approach for causal language models as follows: Instead of a complete sentence, we use incomplete sentence templates such as "A(n) {object} is usually in the ..." or "In the {room} is usually a/an ..." The model should then predict the next token. Instead
of masking the seed word, a neutral equivalent is used for calculation:
$$
A (n) \left\{\text {o b j e c t} \right\} \text {i s u s u a l l y i n t h e} \dots
$$
This is usually in the ...
Instead of performing the analysis in only one direction, we determine the score for both the target and the source given the other.
4.3.3 Classifier-based Measures
In addition to the previously described measures, we experiment with classifiers. To this end, we train three classifiers on the model representations of the source word to determine the associated target word as a class label (e.g. predict kitchen, given the vector of frying pan). We generate the set of source word representations  $X$  in the same way as in the case of the cosine measure (see Section 4.3.1) and average them before classification:
$$
t a r g e t = \text {C l a s s i f i e r} \left(\frac {1}{| X |} \sum_ {\vec {x} \in X} \vec {x}\right)
$$
The training runs on a leave-one-out cross-validation repeated 100 times. The target vector was then generated from the counted predicted classes (see Figure 2b in Appendix). We trained a  $k$ -nearest neighbors classifier with  $k = 5$  (KNN), an SVM with a linear kernel and a feed-forward network (FFN). A small hyperparameter optimization was performed for the FFN, which resulted in the following parameters: Adam Optimizer (Kingma and Ba, 2014) with a learning rate of 0.01 over 100 epochs and one hidden layer of size 100 and ReLU as activation function.
4.4 Scoring Measures and Classifiers
Given a word representation model, we compute the final score for the measures and classifiers to estimate how well they reconstruct the original probability distribution of the source entities relative to the target entities (see Table 4, 5, and 6). This is computed by the distance correlation (Székely et al., 2007) between the target-source probability distributions and the corresponding association distributions of the respective measure or classifier. The advantage of the distance correlation over the Pearson correlation is that it can also measure nonlinear relations. This was calculated both for all targets individually (correlation of all sources to one target) and then concatenated for all targets
Word2Vec contains vectors for MWE's.
together; we denote this variant by  $CONC$ . Therefore,  $CONC$  does not correspond to the average of the individual distance correlations.
Using the apparatus of Section 4, we now evaluate the classes of word representation models (static, MLMs and CLMs) in conjunction with the similarity measures and classifiers. The results for the static models are shown in Table 1, for the MLMs in Table 2 and for the CLMs in Table 3. Figure 2, 3 and 4 in Appendix show a visualization of the associations computed by means of cosine, masked-target & masked-source increased log similarity measures and the FFN classifier based on BERT-Large using the different datasets.
An experiment was also conducted with word frequencies via Google Ngram $^6$  (see Section A.1 in the appendix).
5.1 Model-related Observations
The basic expectation that the cosine measure would generally perform the worst and the FFN classifier the best was met (see Tables 1-3). Interestingly, cosine is also outperformed by distance correlation in almost all cases.
Among the static models, GloVe and fastText performed best in most cases, especially on the room and part dataset (Table 1). Although Levy performs by far the worst in the room dataset, it keeps up with all classification results in the verb dataset. One reason for this could be the dependency-based learning strategy, which seems to work very well for verb associations, even though it was trained on a much smaller data set.