SlowGuess/ACL_2020_OCR huggingface dataset:
compensate for this as follows: In our downstream task experiments, we gather the set of contexts  $\mathcal{C}$  for each word from WWC+BookCorpus during inference.5
5.2 WNLaMPro
We evaluate BERTRAM on the WNLaMPro dataset (Schick and Schütze, 2020). This dataset consists of cloze-style phrases like "A lingonberry is a ____" and the task is to correctly fill the slot (____) with one of several acceptable target words (e.g., "fruit", "bush" or "berry"), which requires understanding of the meaning of the phrase's keyword ("lingonberry" in the example). As the goal of this dataset is to probe a language model's ability to understand rare words without any task-specific finetuning, Schick and Schütze (2020) do not provide a training set. The dataset is partitioned into three subsets based on the keyword's frequency in WWC: RARE (occurring fewer than 10 times) MEDIUM (occurring between 10 and 100 times), and FREQUENT (all remaining words).
For our evaluation, we compare the performance of a standalone BERT (or RoBERTa) model with one that uses BERTRAM as shown in Figure 1 (bottom). As our focus is to improve representations for rare words, we evaluate our model only on WNLaMPro RARE and MEDIUM. Table 1 gives results; our measure is mean reciprocal rank (MRR). We see that supplementing BERT with any of the proposed methods results in noticeable improvements for the RARE subset, with ADD clearly outperforming SHALLOW and REPLACE. Moreover, ADD performs surprisingly well for more frequent words, improving the score for WNLaMPro-MEDIUM by
5We recreate BookCorpus with the script at github. com/soskek/bookcorpus. We refer to the joined corpus of WwC and BookCorpus as WwC+BookCorpus.
Model	MNLI	AG's News	DBPedia
All	Msp	WN	All	Msp	WN	All	Msp	WN
BERT (base)	50.5	49.1	53.4	56.5	54.8	61.9	49.3	46.0	57.6
+ Mimick (Pinter et al., 2017)	37.2	38.2	38.7	45.3	43.9	50.5	36.5	35.8	41.1
+ A La Carte (Khodak et al., 2018)	44.6	45.7	46.1	52.4	53.7	56.1	51.1	48.7	59.3
+ AM (Schick and Schütze, 2020)	50.9	50.7	53.6	58.9	59.8	62.6	60.7	63.1	62.8
+ BERTRAM	53.3	52.5	55.6	62.1	63.1	65.3	64.2	67.9	64.1
+ BERTRAM-SLASH	56.4	55.3	58.6	62.9	63.3	65.3	65.7	67.3	67.2
+ BERTRAM-SLASH + INDomain	59.8	57.3	62.7	62.5	62.1	66.6	74.2	74.8	76.7
RoBERTa (large)	67.3	68.7	68.4	63.7	68.1	65.7	65.5	67.3	66.6
+ BERTRAM-SLASH	70.1	71.5	70.9	64.6	68.4	64.9	71.9	73.8	73.9
+ BERTRAM-SLASH + INDomain	71.7	71.9	73.2	68.1	71.9	69.0	76.0	78.8	77.3
Table 3: Accuracy of standalone BERT and RoBERTa, various baselines and BERTRAM on rarified MNLI, AG's News and DBPedia. The five BERTRAM instances are BERTRAM-ADD. Best results per baseline model are underlined, results that do not differ significantly from the best results in a two-sided binomial test  $(p < 0.05)$  are bold. Msp/WN: subset of instances containing at least one misspelling/synonym. All: all instances.
$58\%$  compared to  $\mathrm{BERT}_{\mathrm{base}}$  and  $37\%$  compared to Attentive Mimicking. This makes sense considering that the key enhancement of BERTRAM over AM lies in improving context representations and interconnection of form and context; the more contexts are given, the more this comes into play. Noticeably, despite being both based on and integrated into a  $\mathrm{BERT}_{\mathrm{base}}$  model, our architecture even outperforms  $\mathrm{BERT}_{\mathrm{large}}$  by a large margin. While RoBERTa performs much better than BERT on WNLaMPro, BERTRAM still significantly improves results for both rare and medium frequency words. As it performs best for both the RARE and MEDIUM subset, we always use the ADD configuration of BERTRAM in the following experiments.
5.3 Downstream Task Datasets
To measure the effect of adding BERTRAM to a pretrained deep language model on downstream tasks, we rarify (cf. §4) the following three datasets:
- MNLI (Williams et al., 2018), a natural language inference dataset where given two sentences  $a$  and  $b$ , the task is to decide whether  $a$  entails  $b$ ,  $a$  and  $b$  contradict each other or neither;
- AG's News (Zhang et al., 2015), a news classification dataset with four different categories (world, sports, business and science/tech);
DBPedia (Lehmann et al., 2015), an ontology dataset with 14 classes (e.g., company, artist) that have to be identified from text snippets.
For all three datasets, we create rarified instances both using  $\mathrm{BERT}_{\mathrm{base}}$  and  $\mathrm{RoBERTa}_{\mathrm{large}}$  as a baseline model and build the substitution dictionary  $S$
using the synonym relation of WordNet (Miller, 1995) and the pattern library (Smedt and Daelemans, 2012) to make sure that all synonyms have consistent parts of speech. Furthermore, we only consider synonyms for each word's most frequent sense; this filters out much noise and improves the quality of the created sentences. In addition to WordNet, we use the misspelling dataset of Pik-tus et al. (2019). To prevent misspellings from dominating the resulting datasets, we only assign misspelling-based substitutes to randomly selected  $10\%$  of the words contained in each sentence. Motivated by the results on WNLaMPro-MEDIUM, we consider every word that occurs less than 100 times in WWC+BookCorpus as being rare. Example entries from the rarified datasets obtained using  $\mathrm{BERT}_{\mathrm{base}}$  as a baseline model can be seen in Table 2. The average number of words replaced with synonyms or misspellings is 1.38, 1.82 and 2.34 for MNLI, AG's News and DBPedia, respectively.
Our default way of injecting BERTRAM embeddings into the baseline model is to replace the sequence of uncontextualized subword token embeddings for a given rare word with its BERTRAM-based embedding (Figure 1, bottom). That is, given a sequence of uncontextualized token embeddings  $\mathbf{e} = e_1,\ldots ,e_n$  where  $e_i,\dots ,e_j$  with  $1\leq i\leq j\leq n$  is the sequence of embeddings for a single rare word  $w$  with BERTRAM-based embedding  $v_{(w,\mathcal{C})}$ , we replace  $\mathbf{e}$  with
$$
\mathbf {e} ^ {\prime} = e _ {1}, \dots , e _ {i - 1}, v _ {(w, \mathcal {C})}, e _ {j + 1}, \dots , e _ {n}.
$$
As an alternative to replacing the original sequence of subword embeddings for a given rare word, we also consider BERTRAM-SLASH, a con
Figuration where the BERTRAM-based embedding is simply added and both representations are separated using a single slash:
$$
\mathbf {e} _ {\text {S L A S H}} = e _ {1}, \dots , e _ {j}, e _ {/}, v _ {(w, \mathcal {C})}, e _ {j + 1}, \dots , e _ {n}.
$$
The intuition behind this variant is that in BERT's pretraining corpus, a slash is often used to separate two variants of the same word (e.g., "useable / usable") or two closely related concepts (e.g., "company / organization", "web-based / cloud") and thus, BERT should be able to understand that both  $e_i, \ldots, e_j$  and  $v_{(w, \mathcal{C})}$  refer to the same entity. We therefore surmise that whenever some information is encoded in one representation but not in the other, giving BERT both representations is helpful.
By default, the set of contexts  $\mathcal{C}$  for each word is obtained by collecting all sentences from WWC+BookCorpus in which it occurs. We also try a variant where we add in-domain contexts by giving BERTRAM access to all texts (but not labels) found in the test set; we refer to this variant as INDOMAIN.<sup>6</sup> Our motivation for including this variant is as follows: Moving from the training stage of a model to its production use often causes a slight domain shift. This is turn leads to an increased number of input sentences containing words that did not – or only very rarely – appear in the training data. However, such input sentences can easily be collected as additional unlabeled examples during production use. While there is no straightforward way to leverage these unlabeled examples with an already finetuned BERT model, BERTRAM can easily make use of them without requiring any labels or any further training: They can simply be included as additional contexts during inference. As this gives BERTRAM a slight advantage, we also report results for all configurations without using indomain data. Importantly, adding indomain data increases the number of contexts for more than  $90\%$  of all rare words by at most 3, meaning that they can still be considered rare despite the additional indomain contexts.
Table 3 reports, for each task, the accuracy on the entire dataset (All) as well as scores obtained considering only instances where at least one word was replaced by a misspelling (Msp) or a WordNet synonym (WN), respectively.7 Consistent with results
Figure 3: BERT vs. BERT combined with BERTRAM-SLASH (BERT+BSL) on three downstream tasks for varying maximum numbers of contexts  $c_{\mathrm{max}}$
on WNLaMPro, combining BERT with BERTRAM consistently outperforms both a standalone BERT model and one combined with various baseline models. Using the SLASH variant brings improvements across all datasets as does adding INDOMAIN contexts (exception: BERT/AG's News). This makes sense considering that for a rare word, every single additional context can be crucial for gaining a deeper understanding. Correspondingly, it is not surprising that the benefit of adding BERTRAM to RoBERTa is less pronounced, because BERTRAM uses only a fraction of the contexts available to RoBERTa during pretraining. Nonetheless, adding BERTRAM significantly improves RoBERTa's accuracy for all three datasets both with and without adding INDOMAIN contexts.
To further understand for which words using BERTRAM is helpful, Figure 3 looks at the accuracy of  $\mathrm{BERT}_{\mathrm{base}}$  both with and without BERTRAM as a function of word frequency. That is, we compute the accuracy scores for both models when considering only entries  $(\mathbf{x}_{w_{i_1} = \hat{w}_{i_1},\dots,w_{i_k} = \hat{w}_{i_k}},y)$  where each substituted word  $\hat{w}_{ij}$  occurs less than  $c_{\mathrm{max}}$  times in WWC+BookCorpus, for different values of  $c_{\mathrm{max}}$ . As one would expect,  $c_{\mathrm{max}}$  is positively correlated with the accuracies of both models, showing that the rarer a word is, the harder it is to understand. Interestingly, the gap between standalone BERT and BERT with BERTRAM remains more or less constant regardless of  $c_{\mathrm{max}}$ . This suggests that using BERTRAM may even be helpful for more frequent words.
To investigate this hypothesis, we perform another rerification of MNLI that differs from the
For the MNLI dataset, which consists of text pairs  $(a,b)$  we treat  $a$  and  $b$  as separate contexts.
Note that results for BERT and RoBERTa are only loosely comparable because the datasets generated from both baseline models through rarification are different.
Figure 4: Improvements for BERT (base) and RoBERTa (large) when adding BERTRAM-SLASH (+BSL) or BERTRAM-SLASH + INDomain (+BSL+ID) on MNLI-1000
previous rarification in two respects. First, we increase the threshold for a word to count as rare from 100 to 1000. Second, as this means that we have more WordNet synonyms available, we do not use the misspelling dictionary (Piktus et al., 2019) for substitution. We refer to the resulting datasets for  $\mathrm{BERT}_{\mathrm{base}}$  and RoBERTa large as MNLI-1000.
Figure 4 shows results on MNLI-1000 for various rare word frequency ranges. For each value  $[c_0, c_1)$  on the  $x$ -axis, the  $y$ -axis shows improvement in accuracy compared to standalone BERT or RoBERTa when only dataset entries are considered for which each rarified word occurs between  $c_0$  (inclusively) and  $c_1$  (exclusively) times in WWC+BooksCorpus. We see that for words with frequency less than 125, the improvement in accuracy remains similar even without using misspellings as another source of substitutions. Interestingly, for every single interval of rare word counts considered, adding BERTRAM-SLASH to BERT considerably improves its accuracy. For RoBERTa, adding BERTRAM brings improvements only for words occurring less than 500 times. While using INDOMAIN data is beneficial for rare words - simply because it gives us additional contexts for these words -, when considering only words that occur at least 250 times in WWC+BookCorpus, adding INDOMAIN contexts does not help.
We have introduced BERTRAM, a novel architecture for inducing high-quality representations for
rare words in BERT's and RoBERTa's embedding spaces. This is achieved by employing a powerful pretrained language model and deeply integrating surface-form and context information. By replacing important words with rare synonyms, we created downstream task datasets that are more challenging and support the evaluation of NLP models on the task of understanding rare words, a capability that human speakers have. On all of these datasets, BERTRAM improves over standard BERT and RoBERTa, demonstrating the usefulness of our method.
Our analysis showed that BERTRAM is beneficial not only for rare words (our main target in this paper), but also for frequent words. In future work, we want to investigate BERTRAM's potential benefits for such frequent words. Furthermore, it would be interesting to explore more complex ways of incorporating surface-form information – e.g., by using a character-level CNN similar to the one of Kim et al. (2016) – to balance out the potency of BERTRAM's form and context parts.
This work was funded by the European Research Council (ERC #740516). We would like to thank the anonymous reviewers for their helpful comments.
Alexei Baevski, Sergey Edunov, Yinhan Liu, Luke Zettlemoyer, and Michael Auli. 2019. Cloze-driven pretraining of self-attention networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5359-5368, Hong Kong, China. Association for Computational Linguistics.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146.
Sam Bowman. 2019. Google T5 explores the limits of transfer learning.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers),
pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018. HotFlip: White-box adversarial examples for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 31-36, Melbourne, Australia. Association for Computational Linguistics.
Jeroen Van Hautte, Guy Emerson, and Marek Rei. 2019. Bad form: Comparing context-based and form-based few-shot learning in distributional semantic models. Computing Research Repository, arXiv:1910.00275.
Aurelie Herbelot and Marco Baroni. 2017. High-risk learning: acquiring new word vectors from tiny data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 304-309. Association for Computational Linguistics.
Mikhail Khodak, Nikunj Saunshi, Yingyu Liang, Tengyu Ma, Brandon Stewart, and Sanjeev Arora. 2018. A la carte embedding: Cheap but effective induction of semantic feature vectors. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12-22. Association for Computational Linguistics.
Yoon Kim, Yacine Jernite, David Sontag, and Alexander M. Rush. 2016. Character-aware neural language models. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI'16, pages 2741-2749. AAAI Press.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations (ICLR).
Angeliki Lazaridou, Marco Marelli, and Marco Baroni. 2017. Multimodal word meaning induction from minimal exposure to natural text. Cognitive Science, 41(S4):677-705.
Jaejun Lee, Raphael Tang, and Jimmy Lin. 2019. What would Elsa do? Freezing layers during transformer fine-tuning. Computing Research Repository, arXiv:1911.03090.
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N. Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick van Kleef, Soren Auer, and Christian Bizer. 2015. DBpedia - a large-scale, multilingual knowledge base extracted from wikipedia. Semantic Web Journal, 6(2):167-195.
Qianchu Liu, Diana McCarthy, and Anna Korhonen. 2019a. Second-order contexts from lexical substitutes for few-shot learning of word representations. In Proceedings of the Eighth Joint Conference on
Lexical and Computational Semantics (*SEM 2019), pages 61-67, Minneapolis, Minnesota. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. RoBERTa: A robustly optimized BERT pretraining approach. Computing Research Repository, arXiv:1907.11692.
Thang Luong, Richard Socher, and Christopher Manning. 2013. Better word representations with recursive neural networks for morphology. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pages 104-113, Sofia, Bulgaria. Association for Computational Linguistics.
Oren Melamud, Jacob Goldberger, and Ido Dagan. 2016. context2vec: Learning generic context embedding with bidirectional LSTM. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 51-61, Berlin, Germany. Association for Computational Linguistics.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. Computing Research Repository, arXiv:1301.3781.
George A. Miller. 1995. Wordnet: A lexical database for english. Commun. ACM, 38(11):39-41.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
Aleksandra Piktus, Necati Bora Edizel, Piotr Bojanowski, Edouard Grave, Rui Ferreira, and Fabrizio Silvestri. 2019. Misspelling oblivious word embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3226-3234, Minneapolis, Minnesota. Association for Computational Linguistics.
Yuval Pinter, Robert Guthrie, and Jacob Eisenstein. 2017. Mimicking word embeddings using subword RNNs. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing,
pages 102-112. Association for Computational Linguistics.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training.
Alexandre Salle and Aline Villavicencio. 2018. Incorporating subword information into matrix factorization word embeddings. In Proceedings of the Second Workshop on Subword/Character Level Models, pages 66-71, New Orleans. Association for Computational Linguistics.
Timo Schick and Hinrich Schütze. 2019a. Attentive mimicking: Better word embeddings by attending to informative contexts. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 489-494, Minneapolis, Minnesota. Association for Computational Linguistics.
Timo Schick and Hinrich Schütze. 2019b. Learning semantic representations for novel words: Leveraging both form and context. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence.
Timo Schick and Hinrich Schütze. 2020. Rare words: A major problem for contextualized embeddings and how to fix it by attentive mimicking. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
Cyrus Shaoul and Chris Westbury. 2010. The westbury lab wikipedia corpus.
Tom De Smedt and Walter Daelemans. 2012. Pattern for python. Journal of Machine Learning Research, 13(Jun):2063-2067.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Transformers: State-of-the-art natural language processing.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyls, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. Computing Research Repository, arXiv:1609.08144.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. Computing Research Repository, arXiv:1906.08237.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 649-657. Curran Associates, Inc.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19-27.
Our implementation of BERTRAM is based on PyTorch (Paszke et al., 2017) and the Transformers library (Wolf et al., 2019). To obtain target embeddings for frequent multi-token words (i.e., words that occur at least 100 times in WWC+BookCorpus) during training, we use one-token approximation (OTA) (Schick and Schütze, 2020). For RoBERTaLarge, we found increasing the number of iterations per word from 4,000 to 8,000 to produce better OTA embeddings using the same evaluation setup as Schick and Schütze (2020). For all stages of training, we use Adam (Kingma and Ba, 2015) as optimizer.