SlowGuess/NAACL_2021_OCR huggingface dataset:
Philipp Dufter*, Nora Kassner*, Hinrich Schütze
Center for Information and Language Processing (CIS), LMU Munich, Germany
{philipp,kassner}@cis.lmu.de
Recent research investigates factual knowledge stored in large pretrained language models (PLMs). Instead of structural knowledge base (KB) queries, masked sentences such as "Paris is the capital of [MASK]" are used as probes. The good performance on this analysis task has been interpreted as PLMs becoming potential repositories of factual knowledge. In experiments across ten linguistically diverse languages, we study knowledge contained in static embeddings. We show that, when restricting the output space to a candidate set, simple nearest neighbor matching using static embeddings performs better than PLMs. E.g., static embeddings perform  $1.6\%$  points better than BERT while just using  $0.3\%$  of energy for training. One important factor in their good comparative performance is that static embeddings are standardly learned for a large vocabulary. In contrast, BERT exploits its more sophisticated, but expensive ability to compose meaningful representations from a much smaller subword vocabulary.
Pretrained language models (PLMs) (Peters et al., 2018; Howard and Ruder, 2018; Devlin et al., 2019) can be finetuned to a variety of natural language processing (NLP) tasks and then generally yield high performance. Increasingly, these models and their generative variants (e.g., GPT, Brown et al., 2020) are used to solve tasks by simple text generation, without any finetuning. This motivated research on how much knowledge is contained in PLMs: Petroni et al. (2019) used models pretrained with a masked language objective to answer cloze-style templates such as:
(Ex1) Paris is the capital of [MASK].
Using this methodology, Petroni et al. (2019) showed that PLMs capture some knowledge implicitly. This has been interpreted as suggesting
Model	Vocabulary Size	LAMA	p1 LAMA-UHN
Oracle	22.0	23.7
BERT	30k	39.6	30.7
mBERT	110k	36.3	27.4
fastText	BERT-30k	26.9	16.8
mBERT-110k	27.5	17.8
30k	16.4	5.8
120k	34.3	25.0
250k	37.7	29.0
500k	39.9	31.8
1000k	41.2	33.4
Table 1: Results for majority oracle, BERT, mBERT and fastText. Static fastText embeddings are competitive and outperform BERT for large vocabularies. BERT and mBERT use their subword vocabularies. For fastText, we use BERT/mBERT's vocabularies and newly trained wordpiece vocabularies on Wikipedia.
that PLMs are promising as repositories of factual knowledge. In this paper, we present evidence that simple static embeddings like fastText perform as well as PLMs in the context of answering knowledge base (KB) queries. Answering KB queries can be decomposed into two subproblems, typing and ranking. Typing refers to the problem of predicting the correct type of the answer entity; e.g., "country" is the correct type for [MASK] in (Ex1), a task that PLMs seem to be good at. Ranking consists of finding the entity of the correct type that is the best fit ("France" in (Ex1)). By restricting the output space to the correct type we disentangle the two subproblems and only evaluate ranking. We do this for three reasons. (i) Ranking is the knowledge-intensive step and thus the key research question. (ii)Typed querying reduces PLMs' dependency on the template. (iii) It allows a direct comparison between static word embeddings and PLMs. Prior work has adopted a similar approach (Xiong et al., 2020; Kassner et al., 2021).
For a PLM like BERT, ranking amounts to finding the entity whose embedding is most similar
* Equal contribution - random order.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2353-2363 June 6-11, 2021. ©2021 Association for Computational Linguistics
to the output embedding for [MASK]. For static embeddings, we rank entities (e.g., entities of type country) with respect to similarity to the query entity (e.g., "Paris" in (Ex1)). In experiments across ten linguistically diverse languages, we show that this simple nearest neighbor matching with fastText embeddings performs comparably to or even better than BERT. For example for English, fastText embeddings perform  $1.6\%$  points better than BERT  $(41.2\%)$  vs.  $39.6\%$ , see Table 1, column "LAMA". This suggests that BERT's core mechanism for answering factual queries is not more effective than simple nearest neighbor matching using fastText embeddings.
We believe this means that claims that PLMs are KBs have to be treated with caution. Advantages of BERT are that it composes meaningful representations from a small subword vocabulary and handles typing implicitly (Petroni et al., 2019). In contrast, answering queries without restricting the answer space to a list of candidates is hard to achieve with static word embeddings. On the other hand, static embeddings are cheap to obtain, even for large vocabulary sizes. This has important implications for green NLP. PLMs require tremendous computational resources, whereas static embeddings have only  $0.3\%$  of the carbon footprint of BERT (see Table 4). This argues for proponents of resource-hungry deep learning models to try harder to find cheap "green" baselines or to combine the best of both worlds (cf. Poerner et al., 2020).
i) We propose an experimental setup that allows a direct comparison between PLMs and static word embeddings. We find that static word embeddings show performance similar to BERT on the modified LAMA analysis task across ten languages.
ii) We provide evidence that there is a trade-off between composing meaningful representations from subwords and increasing the vocabulary size. Storing information through composition in a network seems to be more expensive and challenging than simply increasing the number of atomic representations.
iii) Our findings may point to a general problem: baselines that are simpler and "greener" are not given enough attention in deep learning.
Code and embeddings are available online.
Language	Code	Family	Script
Arabic	AR	Afro-Asiatic	Arabic
German	DE	Indo-European	Latin
English	EN	Indo-European	Latin
Spanish	ES	Indo-European	Latin
Finnish	FI	Uralic	Latin
Hebrew	HE	Afro-Asiatic	Hebrew
Japanese	JA	Japonic	Japanese
Korean	KO	Koreanic	Korean
Turkish	TR	Turkic	Latin
Thai	TH	Tai-Kadai	Thai
Table 2: Overview of the ten languages in our experiments, including language family and script.
We follow the LAMA setup introduced by Petroni et al. (2019). More specifically, we use data from TREx (Elsahar et al., 2018). TREx consists of triples of the form (object, relation, subject). The underlying idea of LAMA is to query knowledge from PLMs using templates without any finetuning: the triple (Paris, capital-of, France) is queried with the template "Paris is the capital of [MASK]." TREx covers 41 relations. Templates for each relation were manually created by Petroni et al. (2019). LAMA has been found to contain many "easy-to-guess" triples; e.g., it is easy to guess that a person with an Italian sounding name is Italian. LAMA-UHN is a subset of triples that are "hard-to-guess" created by Poerner et al. (2020).
Beyond English, we run experiments on nine additional languages using mLAMA, a multilingual version of TREx (Kassner et al., 2021). For an overview of languages and language families see Table 2. For training static embeddings, we use Wikipedia dumps from October 2020.
We describe our proposed setup, which allows to compare PLMs with static embeddings.
3.1 PLMs
We use the following two PLMs: (i) BERT for English (BERT-base-cased, Devlin et al. (2019)), (ii) mBERT for all ten languages (the multilingual version BERT-base-multilingual-cased).
Petroni et al. (2019) use templates like "Paris is the capital of [MASK]" and give  $\arg \max_{w\in \mathcal{V}}p(w|t)$  as an answer where  $\mathcal{V}$  is the vocabulary of the PLM and  $p(w|t)$  is the probability that word  $w$  gets predicted in the template  $t$ .
We follow the same setup as (Kassner et al.,
https://github.com/pdufter/staticclama
Model	Vocab. Size	AR	DE	ES	FI	p1 HE	JA	KO	TH	TR
Oracle	21.9	22.3	21.6	21.3	22.9	21.3	21.7	23.7	23.5
mBERT	110k	17.2	31.5	33.6	20.6	17.5	15.1	18.9	13.5	33.8
fastText	mB-110k	16.4	20.9	24.6	21.4	14.5	12.9	16.1	12.9	26.0
30k	20.8	16.2	17.1	16.7	21.4	14.6	17.3	21.3	22.1
120k	27.9	25.2	31.0	24.2	28.3	22.4	28.2	28.0	33.2
250k	30.1	30.3	34.2	28.8	32.8	24.9	30.5	31.6	35.6
500k	31.7	32.5	36.6	30.9	33.7	27.0	31.5	31.8	36.1
1000k	31.3	33.6	36.5	31.8	33.9	27.2	29.8	30.5	36.6
Table 3: p1 for mBERT and fastText on mLAMA. fastText clearly outperforms mBERT for large vocabularies. Numbers across languages are not comparable as the number of triples varies.
Model	Power (W)	h	kWh · PUE	CO2e
BERT	12,041	79	1,507	1,438
fastText-en	618	5	5	5
ratio-en	0.05	0.06	0.003	0.003
Table 4: Power consumption (Power), hours of computation (h), energy consumption (kWh · PUE) and carbon emissions  $(\mathrm{CO}_{2}\mathrm{e})$  of BERT vs. fastText. Training embeddings for all languages takes around 4 times the resources as training English. BERT numbers from (Strubell et al., 2019). We use our server's peak power consumption. See appendix for details.
2021) and use typed querying: for each relation, we create a candidate set  $\mathcal{C}$  and then predict  $\arg \max_{c\in \mathcal{C}}p(c|t)$ . For most templates, there is only one valid entity type, e.g., country for (Ex1). We choose as  $\mathcal{C}$  the set of objects across all triples for a single relation. The candidate set could also be obtained from an entity typing system (e.g., Yaghoobzadeh et al., 2018), but this is beyond the scope of this paper. Variants of typed prediction have been used before (Xiong et al., 2020).
We accommodate multi-token objects, i.e., objects that are not contained in the vocabulary, by including multiple [MASK] tokens in the templates. We then compute an object's score as the average of the log probabilities for its individual tokens. Note that we do not perform any finetuning.
3.2 Vocabulary
The vocabulary  $\mathcal{V}$  of the wordpiece tokenizer is of central importance for static embeddings as well as PLMs. BERT models come with fixed vocabularies. It would be prohibitive to retrain the models with a new vocabulary. It would also be too expensive to increase the vocabulary by a large factor: the embedding matrix is responsible for the majority of the memory consumption of these models.
In contrast, increasing the vocabulary size is
cheap for static embeddings. We thus experiment with different vocabulary sizes for static embeddings. To this end, we train new vocabularies for each language on Wikipedia using the wordpiece tokenizer (Schuster and Nakajima, 2012).
3.3 Static Embeddings
Using either newly trained vocabularies or existing BERT vocabularies, we tokenize Wikipedia. We then train fastText embeddings (Bojanowski et al., 2017) with default parameters (http://fasttext.cc). We consider the same candidate set  $\mathcal{C}$  as for PLMs. Let  $c\in \mathcal{C}$  be a candidate that gets split into tokens  $t_1,\ldots ,t_k$  by the wordpiece tokenizer. We then assign to  $c$  the embedding vector
$$
\bar {e} _ {c} = \frac {1}{k} \sum_ {i = 1} ^ {k} e _ {t _ {i}}
$$
where  $e_{t_i}$  is the fastText vector for token  $t_i$ . We compute the representations for a query  $q$  analogously. For a query  $q$  (the subject of a triple), we then compute the prediction as:
$$
\operatorname *{arg  max}_{c\in \mathcal{C}}\operatorname {cosine - sim}\bigl(\bar{e}_{q},\bar{e}_{c}\bigr),
$$
i.e., we perform simple nearest neighbor matching. Note that the static embedding method does not get any signal about the relation. The method's only input is the subject of a triple, and we leave incorporating a relation vector to future work.
We compute precision at one for each relation, i.e.,  $1 / |T|\sum_{t\in T}\mathbb{1}\{\hat{t}_{object} = t_{object}\}$  where  $T$  is the set of all triples and  $\hat{t}_{object}$  the object predicted using contextualized/static embeddings. Note that  $T$  is different for each language. Our final measure (p1) is then the precision at one (macro-)averaged over relations. As a consistency check we provide an Oracle baseline: it always predicts the most frequent object across triples based on the gold candidate sets.
In this section, we compare the performance of BERT and fastText, analyze their resource consumption, and give evidence that BERT composes meaningful representations from subwords.
4.1 BERT vs. fastText
Results for English are in Table 1. The table shows that when increasing the vocabulary size, static embeddings and BERT exhibit similar performance on LAMA. The Oracle baseline is mostly outperformed. Only for small vocabulary sizes, fastText is worse. Performance of fastText increases with larger vocabulary sizes and with a vocabulary size of  $1000\mathrm{k}$  we observe a  $1.6\%$  absolute performance increase of fastText embeddings compared to BERT (41.2% vs.  $39.6\%$ ). The performance gap between fastText and BERT increases to  $2.7\%$  points on LAMA-UHN, indicating that fastText is less vulnerable to misleading clues about the subject.
Only providing results on English can be prone to unexpected biases. Thus, we verify our results for nine additional languages. Results are shown in Table 3 and the conclusions are similar: for large enough vocabularies, static embeddings consistently have better performance. For languages outside the Indo-European family, the performance gap between mBERT and fastText is much larger (e.g., 31.7 vs. 17.2 for Arabic) and mBERT is sometimes worse than the Oracle.
Our fastText method is quite primitive: it is a type-restricted search for entities similar to what is most prominent in the context (whose central element is the query entity, e.g., "Paris" in (Ex1)). The fact that fastText outperforms BERT raises the question: Does BERT simply use associations between entities (like fastText) or has it captured factual knowledge beyond this?
4.2 BERT vs fastText: Diversity of Predictions
The entropy of the distribution of predicted objects is 6.5 for BERT vs. 7.3 for fastText. So BERT's predictions are less diverse. Of 151 possible objects on average, BERT predicts (on average) 85, fastText 119. For a given relation, BERT's prediction tends to be dominated by one object, which is often the most frequent correct object – possibly because these objects are frequent in Wikipedia/Wikidata. When filtering out triples whose correct answer is the most frequent object, BERT's performance drops to 35.7 whereas fastText's increases to 42.5. See Table 7 in the appendix for full results on diversity. We leave investigating why BERT has these narrower object preferences for future work.
Figure 1: p1 as a function of the tokenization length of the triples' subjects. BERT and fastText use the same vocabulary here, ensuring comparability. BERT based models exhibit a stable performance independent of the number of tokens a subject gets split into. In contrast, fastText's performance drops.
4.3 Contextualization in BERT
BERT's attention mechanism should be able to handle long subjects – in contrast to fastText, for which we use simple averaging. Figure 1 shows that fastText's performance indeed drops when the query gets tokenized into multiple tokens. In contrast, BERT's performance remains stable. We conclude that token averaging harms fastText's performance and that the attention mechanism in BERT composes meaningful representations from subwords.
We try to induce static embeddings from BERT by feeding object and subject surface forms to BERT without any context and then averaging the hidden representations for each layer. Figure 2 analyzes whether a nearest neighbor matching over this static embedding space extracted from BERT's representations is effective in extracting knowledge from it. We find that performance on LAMA is significantly lower across all hidden layers with the first two layers performing best. That simple averaging does not work as well as contextualization indicates that BERT is great at composing meaningful representations through attention. In future work, it would be interesting to extract better static representations from BERT, for example by extracting the representations of entities in real sentences.
4.4 Resource Consumption
Table 4 compares resource consumption of BERT vs. fastText following Strubell et al. (2019). fastText can be efficiently computed on CPUs with a drastically lower power consumption and computation time. Overall, fastText has only  $0.3\%$  of the
Figure 2: Contextualization in BERT. The dashed lines are p1 when querying with templates like "Paris is the capital of [MASK]." and a candidate set. The solid lines reflect performance of nearest neighbor matching with cosine similarity when inducing a static embedding space from the representations at these layers. This shows that extracting high quality static embeddings is not trivial, and BERT's contextualization is essential for getting good performance.
carbon emissions compared to BERT. In a recent study, Zhang et al. (2020) showed that capturing factual knowledge inside PLMs is an especially resource hungry task.
These big differences demonstrate that fastText, in addition to performing better than BERT, is the environmentally better model to "encode knowledge" of Wikipedia in an unsupervised fashion. This calls into question the use of large PLMs as knowledge bases, particularly in light of the recent surge of knowledge augmented LMs, e.g., (Lewis et al., 2020; Guu et al., 2020).
Petroni et al. (2019) first asked: can PLMs function as KBs? Subsequent analysis focused on different aspects, such as negation (Kassner and Schütze, 2020; Ettinger, 2020), paraphrases (Elazar et al., 2021), easy to guess names (Poerner et al., 2020), finding alternatives to a cloze-style approach (Bouraoui et al., 2020; Heinzerling and Inui, 2020; Jiang et al., 2020) or analyzing different model sizes (Roberts et al., 2020).
There is a recent surge of work that tries to improve PLMs' ability to harvest factual knowledge: Zhang et al. (2019), Peters et al. (2019) and Wang et al. (2020) inject factual knowledge into PLMs. Guu et al. (2020), Lewis et al. (2020), Izacard and Grave (2020), Kassner and Schütze (2020) and Petroni et al. (2020) combine PLMs with information retrieval and Bosselut et al. (2019), Liu et al. (2020) and Yu et al. (2020) with knowledge bases.
In contrast, we provide evidence that BERT's ability to answer factual queries is not more effective than capturing "knowledge" with simple
traditional static embeddings. This suggests that learning associations between entities and type-restricted similarity search over these associations may be at the core of BERT's ability to answer cloze-style KB queries, a new insight into BERT's working mechanism.
We have shown that, when restricting cloze-style questions to a candidate set, static word embeddings outperform BERT. To explain this puzzling superiority of a much simpler model, we put forward a new characterization of factual knowledge learned by BERT: BERT seems to be able to complete cloze-style queries based on similarity assessments on a type-restricted vocabulary much like a nearest neighbor search for static embeddings.
However, BERT may still be the better model for the task: we assume perfect typing (for BERT and fastText) and only evaluate ranking. Typing is much harder with static embeddings and BERT has been shown to perform well at guessing the expected entity type based on a template. BERT also works well with small vocabularies, storing most of its "knowledge" in the parameterization of subword composition. Our results suggest that increasing the vocabulary size and computing more atomic entity representations with fastText is a cheap and environmentally friendly method of storing knowledge. In contrast, learning high quality composition of smaller units requires many more resources.
fastText is a simple cheap baseline that outperforms BERT on LAMA, but was not considered in the original research. This may be an example of a general problem: "green" baselines are often ignored, but should be considered when evaluating resource-hungry deep learning models. A promising way forward would be to combine the best of both worlds, e.g., by building on work that incorporates large vocabularies into PLMs after pretraining.
Acknowledgements. This work was supported by the European Research Council (# 740516) and the German Federal Ministry of Education and Research (BMBF) under Grant No. 01IS18036A. The authors of this work take full responsibility for its content. The first author was supported by the Bavarian research institute for digital transformation (bidt) through their fellowship program. We thank Yanai Elazar and the anonymous reviewers for valuable comments.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomás Mikolov. 2017. Enriching word vectors with subword information. Trans. Assoc. Comput. Linguistics, 5:135-146.
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: commonsense transformers for automatic knowledge graph construction. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4762-4779. Association for Computational Linguistics.
Zied Bouraoui, José Camacho-Collados, and Steven Schockaert. 2020. Inducing relational knowledge from BERT. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7456-7463. AAAI Press.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, E. Hovy, H. Schutze, and Yoav Goldberg. 2021. Measuring and improving consistency in pretrained language models. *ArXiv*, abs/2102.01017.
Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. 2018. T-REx: A large scale alignment of natural language with knowledge base triples. In Proceedings of the Eleventh International Conference on Language Resources and