SlowGuess/ACL_2020_OCR huggingface dataset:
Timo Schick
Sulzer GmbH
timo.schick@sulzer.de
Hinrich Schütze
Center for Information and Language Processing
LMU Munich, Germany
inquiries@cislmu.org
Pretraining deep language models has led to large performance gains in NLP. Despite this success, Schick and Schütze (2020) recently showed that these models struggle to understand rare words. For static word embeddings, this problem has been addressed by separately learning representations for rare words. In this work, we transfer this idea to pretrained language models: We introduce BERTRAM, a powerful architecture based on BERT that is capable of inferring high-quality embeddings for rare words that are suitable as input representations for deep language models. This is achieved by enabling the surface form and contexts of a word to interact with each other in a deep architecture. Integrating BERTRAM into BERT leads to large performance increases due to improved representations of rare and medium frequency words on both a rare word probing task and three downstream tasks. $^{1}$
As word embedding algorithms (e.g. Mikolov et al., 2013) are known to struggle with rare words, several techniques for improving their representations have been proposed. These approaches exploit either the contexts in which rare words occur (Lazari-dou et al., 2017; Herbelot and Baroni, 2017; Khodak et al., 2018; Liu et al., 2019a), their surfaceform (Luong et al., 2013; Bojanowski et al., 2017; Pinter et al., 2017), or both (Schick and Schütze, 2019a,b; Hautte et al., 2019). However, all of this prior work is designed for and evaluated on uncontextualized word embeddings.
Contextualized representations obtained from pretrained deep language models (e.g. Peters et al., 2018; Radford et al., 2018; Devlin et al., 2019; Liu et al., 2019b) already handle rare words implicitly
using methods such as byte-pair encoding (Sennrich et al., 2016), WordPiece embeddings (Wu et al., 2016) and character-level CNNs (Baevski et al., 2019). Nevertheless, Schick and Schütze (2020) recently showed that BERT's (Devlin et al., 2019) performance on a rare word probing task can be significantly improved by explicitly learning representations of rare words using Attentive Mimicking (AM) (Schick and Schütze, 2019a). However, AM is limited in two important respects:
- For processing contexts, it uses a simple bag-of-words model, making poor use of the available information.
- It combines form and context in a shallow fashion, preventing both input signals from interacting in a complex manner.
These limitations apply not only to AM, but to all previous work on obtaining representations for rare words by leveraging form and context. While using bag-of-words models is a reasonable choice for static embeddings, which are often themselves bag-of-words (e.g. Mikolov et al., 2013; Bojanowski et al., 2017), it stands to reason that they are not the best choice to generate input representations for position-aware, deep language models.
To overcome these limitations, we introduce BERTRAM (BERT for Attentive Mimicking), a novel architecture for learning rare word representations that combines a pretrained BERT model with AM. As shown in Figure 1, the learned rare word representations can then be used as an improved input representation for another BERT model. By giving BERTRAM access to both surface form and contexts starting at the lowest layer, a deep integration of both input signals becomes possible.
Assessing the effectiveness of methods like BERTRAM in a contextualized setting is challenging: While most previous work on rare words was
<sup>1</sup>Our implementation of BERTRAM is publicly available at https://github.com/timoschick/bertram.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3996-4007 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
evaluated on datasets explicitly focusing on rare words (e.g Luong et al., 2013; Herbelot and Baroni, 2017; Khodak et al., 2018; Liu et al., 2019a), these datasets are tailored to uncontextualized embeddings and thus not suitable for evaluating our model. Furthermore, rare words are not well represented in commonly used downstream task datasets. We therefore introduce rarification, a procedure to automatically convert evaluation datasets into ones for which rare words are guaranteed to be important. This is achieved by replacing task-relevant frequent words with rare synonyms obtained using semantic resources such as WordNet (Miller, 1995). We rarify three common text (or text pair) classification datasets: MNLI (Williams et al., 2018), AG's News (Zhang et al., 2015) and DBPedia (Lehmann et al., 2015). BERTRAM outperforms previous work on four English datasets by a large margin: on the three rarified datasets and on WNLaMPro (Schick and Schütze, 2020).
- We introduce BERTRAM, a model that integrates BERT into Attentive Mimicking, enabling a deep integration of surface-form and contexts and much better representations for rare words.
- We devise rarification, a method that transforms evaluation datasets into ones for which rare words are guaranteed to be important.
- We show that adding BERTRAM to BERT achieves a new state-of-the-art on WNLaM-Pro (Schick and Schütze, 2020) and beats all baselines on rarified AG's News, MNLI and DBPedia, resulting in an absolute improvement of up to  $25\%$  over BERT.
Surface-form information (e.g., morphemes, characters or character  $n$ -grams) is commonly used to improve word representations. For static word embeddings, this information can either be injected into a given embedding space (Luong et al., 2013; Pinter et al., 2017), or a model can directly be given access to it during training (Bojanowski et al., 2017; Salle and Villavicencio, 2018; Piktus et al., 2019). In the area of contextualized representations, many architectures employ subword segmentation methods (e.g. Radford et al., 2018; Devlin et al., 2019; Yang et al., 2019; Liu et al., 2019b). Others use
Figure 1: Top: Standard use of BERT. Bottom: Our proposal; first BERTRAM learns an embedding for "unicycle" that replaces the WordPiece sequence. BERT is then run on this improved input representation.
convolutional neural networks to directly access character-level information (Kim et al., 2016; Peters et al., 2018; Baevski et al., 2019).
Complementary to surface form, another useful source of information for understanding rare words are the contexts in which they occur (Lazaridou et al., 2017; Herbelot and Baroni, 2017; Khodak et al., 2018). Schick and Schütze (2019a,b) show that combining form and context leads to significantly better results than using just one of the two. While all of these methods are bag-of-words models, Liu et al. (2019a) recently proposed an architecture based on context2vec (Melamud et al., 2016). However, in contrast to our work, they (i) do not incorporate surface-form information and (ii) do not directly access the hidden states of context2vec, but instead simply use its output distribution.
Several datasets focus on rare words, e.g., Stanford Rare Word (Luong et al., 2013), Definitional Nonce (Herbelot and Baroni, 2017), and Contextual Rare Word (Khodak et al., 2018). However, unlike our rarified datasets, they are only suitable for evaluating uncontextualized word representations. Rarification is related to adversarial example generation (e.g. Ebrahimi et al., 2018), which manipulates the input to change a model's prediction. We use a similar mechanism to determine which words in a given sentence are most important and replace them with rare synonyms.
3.1 Form-Context Model
We first review the basis for our new model, the form-context model (FCM) (Schick and Schütze, 2019b). Given a set of  $d$ -dimensional high-quality embeddings for frequent words, FCM induces embeddings for rare words that are appropriate for
the given embedding space. This is done as follows: Given a word  $w$  and a context  $C$  in which it occurs, a surface-form embedding  $v_{(w,C)}^{\mathrm{form}} \in \mathbb{R}^{d}$  is obtained by averaging over embeddings of all character  $n$ -grams in  $w$ ; the  $n$ -gram embeddings are learned during training. Similarly, a context embedding  $v_{(w,C)}^{\mathrm{context}} \in \mathbb{R}^{d}$  is obtained by averaging over the embeddings of all words in  $C$ . Finally, both embeddings are combined using a gate
$$
g \left(v _ {(w, C)} ^ {\text {f o r m}}, v _ {(w, C)} ^ {\text {c o n t e x t}}\right) = \sigma \left(x ^ {\top} \left[ v _ {(w, C)} ^ {\text {f o r m}}, v _ {(w, C)} ^ {\text {c o n t e x t}} \right] + y\right)
$$
with parameters  $x\in \mathbb{R}^{2d},y\in \mathbb{R}$  and  $\sigma$  denoting the sigmoid function, allowing the model to decide how to weight surface-form and context. The final representation of  $w$  is then a weighted combination of form and context embeddings:
$$
v _ {(w, C)} = \alpha \cdot (A v _ {(w, C)} ^ {\text {c o n t e x t}} + b) + (1 - \alpha) \cdot v _ {(w, C)} ^ {\text {f o r m}}
$$
where  $\alpha = g(v_{(w,C)}^{\mathrm{form}},v_{(w,C)}^{\mathrm{context}})$  and  $A\in \mathbb{R}^{d\times d}$ ,  $b\in \mathbb{R}^d$  are parameters learned during training.
The context part of FCM is able to capture the broad topic of rare words, but since it is a bag-of-words model, it is not capable of obtaining a more concrete or detailed understanding (see Schick and Schütze, 2019b). Furthermore, the simple gating mechanism results in only a shallow combination of form and context. That is, the model is not able to combine form and context until the very last step: While it can learn to weight form and context components, the two embeddings (form and context) do not share any information and thus do not influence each other.
3.2 BERTRAM
To overcome these limitations, we introduce BERTRAM, a model that combines a pretrained BERT language model (Devlin et al., 2019) with Attentive Mimicking (Schick and Schütze, 2019a). We denote with  $e_t$  the (uncontextualized, i.e., first-layer) embedding assigned to a (wordpiece) token  $t$  by BERT. Given a sequence of such uncontextualized embeddings  $\mathbf{e} = e_1, \ldots, e_n$ , we denote by  $\mathbf{h}_j(\mathbf{e})$  the contextualized representation of the  $j$ -th token at the final layer when the model is given  $\mathbf{e}$  as input.
Given a word  $w$  and a context  $C$  in which it occurs, let  $\mathbf{t} = t_1, \ldots, t_m$  be the sequence obtained from  $C$  by (i) replacing  $w$  with a [MASK] token and (ii) tokenization (matching BERT's vocabulary); furthermore, let  $i$  denote the index for which
$t_i = [\mathrm{MASK}]$ . We experiment with three variants of BERTRAM: BERTRAM-SHALLOW, BERTRAM-REPLACE and BERTRAM-ADD.
SHALLOW. Perhaps the simplest approach for obtaining a context embedding from  $C$  using BERT is to define
$$
v _ {(w, C)} ^ {\text {c o n t e x t}} = \mathbf {h} _ {i} (e _ {t _ {1}}, \ldots , e _ {t _ {m}}).
$$
This approach aligns well with BERT's pretraining objective of predicting likely substitutes for [MASK] tokens from their contexts. The context embedding  $v_{(w,C)}^{\text{context}}$  is then combined with its form counterpart as in FCM.
While this achieves our first goal of using a more sophisticated context model that goes beyond bag-of-words, it still only combines form and context in a shallow fashion.
REPLACE. Before computing the context embedding, we replace the uncontextualized embedding of the [MASK] token with the word's surface-form embedding:
$$
v _ {(w, C)} ^ {\mathrm {c o n t e x t}} = \mathbf {h} _ {i} (e _ {t _ {1}}, \ldots , e _ {t _ {i - 1}}, v _ {(w, C)} ^ {\mathrm {f o r m}}, e _ {t _ {i + 1}}, \ldots , e _ {t _ {m}}).
$$
Our rationale for this is as follows: During regular BERT pretraining, words chosen for prediction are replaced with [MASK] tokens only  $80\%$  of the time and kept unchanged  $10\%$  of the time. Thus, standard pretrained BERT should be able to make use of form embeddings presented this way as they provide a strong signal with regards to how the "correct" embedding of  $w$  may look like.
ADD. Before computing the context embedding, we prepaid the input with the surface-form embedding of  $w$ , followed by a colon  $(e_{:})^{3}$
$$
v _ {(w, C)} ^ {\text {c o n t e x t}} = \mathbf {h} _ {i + 2} \big (v _ {(w, C)} ^ {\text {f o r m}}, e _ {:}, e _ {t _ {1}}, \dots , e _ {t _ {m}} \big).
$$
The intuition behind this third variant is that lexical definitions and explanations of a word  $w$  are occasionally prefixed by "  $w :$  " (e.g., in some online dictionaries). We assume that BERT has seen many definitional sentences of this kind during pretraining and is thus able to leverage surface-form information about  $w$  presented this way.
For both REPLACE and ADD, surface-form information is directly and deeply integrated into the
2We refer to these three BERTRAM configurations simply as SHALLOW, REPLACE and ADD.
3We experimented with other prefixes, but found that this variant is best capable of recovering  $w$  at the masked position.
Figure 2: Schematic representation of BERTRAM-ADD processing the input word  $w =$  "washables" given a single context  $C_1 =$  "other washables such as trousers . . ." (left) and given multiple contexts  $\mathcal{C} = \{C_1,\dots ,C_m\}$  (right)
computation of the context embedding; thus, we do not require any gating mechanism and directly set  $v_{(w,C)} = A \cdot v_{(w,C)}^{\mathrm{context}} + b$ . Figure 2 (left) shows how a single context is processed using ADD.
To exploit multiple contexts of a word if available, we follow the approach of Schick and Schütze (2019a) and add an AM layer on top of our model; see Figure 2 (right). Given a set of contexts  $\mathcal{C} = \{C_1,\dots ,C_m\}$  and the corresponding embeddings  $v_{(w,C_1)},\ldots ,v_{(w,C_m)}$  ,AM applies a selfattention mechanism to all embeddings, allowing the model to distinguish informative from uninformative contexts. The final embedding  $v_{(w,\mathcal{C})}$  is then a weighted combination of all embeddings:
$$
v _ {(w, \mathcal {C})} = \sum_ {i = 1} ^ {m} \rho_ {i} \cdot v _ {(w, C _ {i})}
$$
where the self-attention layer determines the weights  $\rho_{i}$  subject to  $\sum_{i = 1}^{m}\rho_{i} = 1$ . For further details, see Schick and Schütze (2019a).
Like previous work, we use mimicking (Pinter et al., 2017) as a training objective. That is, given a frequent word  $w$  with known embedding  $e_w$  and a set of corresponding contexts  $\mathcal{C}$ , BERTRAM is trained to minimize  $\| e_w - v_{(w,\mathcal{C})} \|^2$ .
Training BERTRAM end-to-end is costly: the cost of processing a single training instance  $(w, \mathcal{C})$  with  $\mathcal{C} = \{C_1, \dots, C_m\}$  is the same as processing an entire batch of  $m$  examples in standard BERT. Therefore, we resort to the following three-stage training process:
1. We train only the context part, minimizing  $\| e_w - A\cdot (\sum_{i = 1}^m\rho_i\cdot v_{(w,C_i)}^{\mathrm{context}}) + b\|^2$  where  $\rho_{i}$  is the weight assigned to each context  $C_i$
through the AM layer. Regardless of the selected BERTRAM variant, the context embedding is always obtained using SHALLOW in this stage. Furthermore, only  $A$ ,  $b$  and all parameters of the AM layer are optimized.
2. We train only the form part (i.e., only the  $n$ -gram embeddings); our loss for a single example  $(w, \mathcal{C})$  is  $\| e_w - v_{(w, \mathcal{C})}^{\mathrm{form}} \|^2$ . Training in this stage is completely detached from the underlying BERT model.
3. In the third stage, we combine the pretrained form-only and context-only models and train all parameters. The first two stages are only run once and then used for all three BERTRAM variants because context and form are trained in isolation. The third stage must be run for each variant separately.
We freeze all of BERT's parameters during training as we – somewhat surprisingly – found that this slightly improves the model's performance while speeding up training. For ADD, we additionally found it helpful to freeze the form part in the third training stage. Importantly, for the first two stages of our training procedure, we do not have to backpropagate through BERT to obtain all required gradients, drastically increasing the training speed.
4 Dataset Rarification
The ideal dataset for measuring the quality of rare word representations would be one for which the accuracy of a model with no understanding of rare words is  $0\%$  whereas the accuracy of a model that perfectly understands rare words is  $100\%$ . Unfortunately, existing datasets do not satisfy this desidera
3999
tum, not least because rare words – by their nature – occur rarely.
This does not mean that rare words are not important: As we shift our focus in NLP from words and sentences as the main unit of processing to larger units like paragraphs and documents, rare words will occur in a high proportion of such larger "evaluation units". Rare words are also clearly a hallmark of human language competence, which should be the ultimate goal of NLP. Our work is part of a trend that sees a need for evaluation tasks in NLP that are more ambitious than what we have now. $^4$
To create more challenging datasets, we use rarification, a procedure that automatically transforms existing text classification datasets in such a way that rare words become important. We require a pretrained language model  $M$  as a baseline, an arbitrary text classification dataset  $\mathcal{D}$  containing labeled instances  $(\mathbf{x},y)$  and a substitution dictionary  $S$ , mapping each word  $w$  to a set of rare synonyms  $S(w)$ . Given these ingredients, our procedure consists of three steps: (i) splitting the dataset into a train set and a set of test candidates, (ii) training the baseline model on the train set and (iii) modifying a subset of the test candidates to generate the final test set.
Dataset Splitting. We partition  $\mathcal{D}$  into a training set  $\mathcal{D}_{\mathrm{train}}$  and a set of test candidates,  $\mathcal{D}_{\mathrm{cand}}$ .  $\mathcal{D}_{\mathrm{cand}}$  contains all instances  $(\mathbf{x},y)\in \mathcal{D}$  such that for at least one word  $w$  in  $\mathbf{x}$ ,  $S(w)\neq \emptyset -$  subject to the constraint that the training set contains at least one third of the entire data.
Baseline Training. We finetune  $M$  on  $\mathcal{D}_{\mathrm{train}}$ . Let  $(\mathbf{x},y)\in \mathcal{D}_{\mathrm{train}}$  where  $\mathbf{x} = w_{1},\ldots ,w_{n}$  is a sequence of words. We deviate from the finetuning procedure of Devlin et al. (2019) in three respects:
- We randomly replace  $5\%$  of all words in  $\mathbf{x}$  with a [MASK] token. This allows the model to cope with missing or unknown words, a prerequisite for our final test set generation.
- As an alternative to overwriting the language model's uncontextualized embeddings for rare words, we also want to allow models to add an alternative representation during test time, in
which case we simply separate both representations by a slash (cf. §5.3). To accustom the language model to this duplication of words, we replace each word  $w_{i}$  with  $\frac{w_i}{w_i}$  with a probability of  $10\%$ . To make sure that the model does not simply learn to always focus on the first instance during training, we randomly mask each of the two repetitions with probability  $25\%$ .
- We do not finetune the model's embedding layer. We found that this does not hurt performance, an observation in line with recent findings of Lee et al. (2019).
Test Set Generation. Let  $p(y \mid x)$  be the probability that the finetuned model  $M$  assigns to class  $y$  given input  $x$ , and  $M(x) = \arg \max_{y \in \mathcal{Y}} p(y \mid x)$  be the model's prediction for input  $x$  where  $\mathcal{Y}$  denotes the set of all labels. For generating our test set, we only consider candidates that are classified correctly by the baseline model, i.e., candidates  $(x, y) \in \mathcal{D}_{\text{cand}}$  with  $M(x) = y$ . For each such entry, let  $x = w_1, \ldots, w_n$  and let  $x_{w_i = t}$  be the sequence obtained from  $x$  by replacing  $w_i$  with  $t$ . We compute
$$
w_{i} = \operatorname *{arg  min}_{w_{j}:S(w_{j})\neq \emptyset}p(y\mid \mathbf{x}_{w_{j} = [\text{MASK}]}),
$$
i.e., we select the word  $w_{i}$  whose masking pushes the model's prediction the farthest away from the correct label. If removing this word already changes the model's prediction - that is,  $M(\mathbf{x}_{w_i = [\mathrm{MASK}]})\neq y-$ , we select a random rare synonym  $\hat{w}_i\in S(w_i)$  and add  $(\mathbf{x}_{w_i = \hat{w}_i},y)$  to the test set. Otherwise, we repeat the above procedure; if the label still has not changed after masking up to 5 words, we discard the candidate. Each instance  $(\mathbf{x}_{w_{i_1}} = \hat{w}_{i_1},\dots ,w_{i_k} = \hat{w}_{i_k},y)$  of the resulting test set has the following properties:
- If each  $w_{i_j}$  is replaced by [MASK], the entry is classified incorrectly by  $M$ . In other words, understanding the words  $w_{i_j}$  is necessary for  $M$  to determine the correct label.
- If the model's internal representation of each  $\hat{w}_{i_j}$  is sufficiently similar to its representation of  $w_{i_j}$ , the entry is classified correctly by  $M$ . That is, if the model is able to understand the rare words  $\hat{w}_{i_j}$  and to identify them as synonyms of  $w_{i_j}$ , it will predict the correct label.
$^{4}$ Cf. (Bowman, 2019): "If we want to be able to establish fair benchmarks that encourage future progress toward robust, human-like language understanding, we'll need to get better at creating clean, challenging, and realistic test datasets."
Model	RARE	MEDIUM
BERT (base)	0.112	0.234
+ AM (Schick and Schütze, 2020)	0.251	0.267
+ BERTRAM-SHALLOW	0.250	0.246
+ BERTRAM-REPLACE	0.155	0.216
+ BERTRAM-ADD	0.269	0.367
BERT (large)	0.143	0.264
RoBERTa (large)	0.270	0.275
+ BERTRAM-ADD	0.306	0.323
Note that the test set is closely coupled to the baseline model  $M$  because we select the words to be replaced based on  $M$ 's predictions. Importantly, however, the model is never queried with any rare synonym during test set generation, so its representations of rare words are not taken into account for creating the test set. Thus, while the test set is not suitable for comparing  $M$  with an entirely different model  $M'$ , it allows us to compare various strategies for representing rare words in the embedding space of  $M$ . Definitional Nonce (Herbelot and Baroni, 2017) is subject to a similar constraint: it is tied to a specific (uncontextualized) embedding space based on Word2Vec (Mikolov et al., 2013).
For our evaluation of BERTRAM, we follow the experimental setup of Schick and Schütze (2020). We experiment with integrating BERTRAM both into  $\mathrm{BERT}_{\mathrm{base}}$  and RoBERTa large (Liu et al., 2019b). Throughout our experiments, when BERTRAM is used to provide input representations for one of the two models, we use the same model as BERTRAM's underlying language model. Further training specifications can be found in Appendix A.
While BERT was trained on BookCorpus (Zhu et al., 2015) and a large Wikipedia dump, we follow previous work and train BERTRAM only on the much smaller Westbury Wikipedia Corpus (WWC) (Shaoul and Westbury, 2010); this of course gives BERT a clear advantage over BERTRAM. This advantage is even more pronounced when comparing BERTRAM with RoBERTa, which is trained on a corpus that is an order of magnitude larger than the original BERT corpus. We try to at least partially
Table 1: MRR on WNLaMPro test for baseline models and various BERTRAN configurations. Best results per base model are underlined, results that do not differ significantly from the best results in a paired t-test  $(p < 0.05)$  are bold.
Task	Entry
MNLI	i think i will go finish up my laundry wash-ables.
AG's	[...] stake will improve meliorate syman-tec's consulting contacts [...]
DBPedia	yukijiro hotaru [...] is a japanese nipponese aeter histrion.
MNLI	a smart person is often oftimes correct in their answers answers.
MNLI	the southwest has a lot of farming and vineyards wineries that make excellent fantastulous merlot.
Table 2: Examples from rarified datasets. Crossed out: replaced words. Bold: replacements.