Evaluation ::: Downstream Task Datasets	To measure the effect of adding Bertram to BERT on downstream tasks, we apply the procedure described in Section SECREF4 to a commonly used textual entailment dataset as well as two text classification datasets: MNLI , AG's News  and DBPedia . For all three datasets, we use BERT$_\text{base}$ as a baseline model and create the substitution dictionary $S$ using the synonym relation of WordNet  and the pattern library  to make sure that all synonyms have consistent parts of speech. As an additional source of word substitutions, we make use of the misspellings dataset of , which is based on query logs of a search engine. To prevent misspellings from dominating the resulting dataset, we only assign misspelling-based substitutes to randomly selected 10% of the words contained in each sentence. Motivated by the results on WNLaMPro-medium, we consider every word that occurs less than 100 times in the WWC and our BooksCorpus replica combined as being rare. Some examples of entries in the resulting datasets can be seen in Table TABREF35.

BERTRAM: Improved Word Embeddings Have Big Impact on Contextualized Model Performance	Pretraining deep contextualized representations using an unsupervised language modeling objective has led to large performance gains for a variety of NLP tasks. Despite this success, recent work by Schick and Schutze (2019) suggests that these architectures struggle to understand rare words. For context-independent word embeddings, this problem can be addressed by separately learning representations for infrequent words. In this work, we show that the same idea can also be applied to contextualized models and clearly improves their downstream task performance. Most approaches for inducing word embeddings into existing embedding spaces are based on simple bag-of-words models; hence they are not a suitable counterpart for deep neural network language models. To overcome this problem, we introduce BERTRAM, a powerful architecture based on a pretrained BERT language model and capable of inferring high-quality representations for rare words. In BERTRAM, surface form and contexts of a word directly interact with each other in a deep architecture. Both on a rare word probing task and on three downstream task datasets, BERTRAM considerably improves representations for rare and medium frequency words compared to both a standalone BERT model and previous work.

For various reasons, assessing the effectiveness of methods like Bertram in a contextualized setting poses a huge difficulty: While most previous work on rare words was evaluated on datasets explicitly focusing on such words , , , , , all of these datasets are tailored towards context-independent embeddings and thus not suitable for evaluating our proposed model. Furthermore, understanding rare words is of negligible importance for most commonly used downstream task datasets. To evaluate our proposed model, we therefore introduce a novel procedure that allows us to automatically turn arbitrary text classification datasets into ones where rare words are guaranteed to be important. This is achieved by replacing classification-relevant frequent words with rare synonyms obtained using semantic resources such as WordNet . Using this procedure, we extract rare word datasets from three commonly used text (or text pair) classification datasets: MNLI , AG's News  and DBPedia . On both the WNLaMPro dataset of  and all three so-obtained datasets, our proposed Bertram model outperforms previous work by a large margin. In summary, our contributions are as follows: We show that a pretrained BERT instance can be integrated into Attentive Mimicking, resulting in much better context representations and a deeper connection of form and context.

Evaluation ::: WNLaMPro	We evalute Bertram on the WNLaMPro dataset of . This dataset consists of cloze-style phrases like and the task is to correctly fill the slot (____) with one of several acceptable target words (e.g., “fruit”, “bush” and “berry”), which requires knowledge of the phrase's keyword (“lingonberry” in the above example). As the goal of this dataset is to probe a language model's ability to understand rare words without any task-specific finetuning,  do not provide a training set. Furthermore, the dataset is partitioned into three subsets; this partition is based on the frequency of the keyword, with keywords occurring less than 10 times in the WWC forming the rare subset, those occurring between 10 and 100 times forming the medium subset, and all remaining words forming the frequent subset. As our focus is on improving representations for rare words, we evaluate our model only on the former two sets. Results on WNLaMPro rare and medium are shown in Table TABREF34, where the mean reciprocal rank (MRR) is reported for BERT, Attentive Mimicking and Bertram.

Conclusion	We have introduced Bertram, a novel architecture for relearning high-quality representations of rare words. This is achieved by employing a powerful pretrained language model and deeply connecting surface-form and context information. By replacing important words with rare synonyms, we have created various downstream task datasets focusing on rare words; on all of these datasets, Bertram improves over a BERT model without special handling of rare words, demonstrating the usefulness of our proposed method. As our analysis has shown that even for the most frequent words considered, using Bertram is still beneficial, future work might further investigate the limits of our proposed method. Furthermore, it would be interesting to explore more complex ways of incorporating surface-form information – e.g., by using a character-level CNN similar to the one of  – to balance out the potency of Bertram's form and context parts.

In addition to the procedure described above, we also try a variant where instead of replacing the original WordPiece embeddings for a given rare word, we merely add the Bertram-based embedding, separating both representations using a single slash: As it performs best on the rare and medium subsets of WNLaMPro combined, we use only the add-gated variant of Bertram for all datasets. Results can be seen in Table TABREF37, where for each task, we report the accuracy on the entire dataset as well as scores obtained considering only instances where at least one word was replaced by a misspelling or a WordNet synonym, respectively. Consistent with results on WNLaMPro, combining BERT with Bertram outperforms both a standalone BERT model and one combined with Attentive Mimicking across all tasks. While keeping the original BERT embeddings in addition to Bertram's representation brings no benefit, adding in-domain data clearly helps for two out of three datasets. This makes sense as for rare words, every single additional context can be crucial for gaining a deeper understanding.

To further understand for which words using Bertram is helpful, in Figure FIGREF39 we look at the accuracy of BERT both with and without Bertram on all three tasks as a function of word frequency. That is, we compute the accuracy scores for both models when considering only entries $(\mathbf {x}_{w_{i_1} = \hat{w}_{i_1}, \ldots , w_{i_k} = \hat{w}_{i_k} }, y)$ where each substituted word $\hat{w}_{i_j}$ occurs less than $c_\text{max}$ times in WWC and BooksCorpus, for various values of $c_\text{max}$. As one would expect, $c_\text{max}$ is positively correlated with the accuracies of both models, showing that the rarer a word is, the harder it is to understand. Perhaps more interestingly, for all three datasets the gap between Bertram and BERT remains more or less constant regardless of $c_\text{max}$. This indicates that using Bertram might also be useful for even more frequent words than the ones considered.

We design a procedure that allows us to automatically transform text classification datasets into datasets for which rare words are guaranteed to be important. We show that Bertram achieves a new state-of-the-art on the WNLaMPro probing task  and beats all baselines on rare word instances of AG's News, MNLI and DBPedia, resulting in an absolute improvement of up to 24% over a BERT baseline.

Generation of Rare Word Datasets	To measure the quality of rare word representations in a contextualized setting, we would ideally need text classification datasets with the following two properties: A model that has no understanding of rare words at all should perform close to 0%. A model that perfectly understands rare words should be able to classify every instance correctly. Unfortunately, this requirement is not even remotely fulfilled by most commonly used datasets, simply because rare words occur in only a few entries and when they do, they are often of negligible importance. To solve this problem, we devise a procedure to automatically transform existing text classification datasets such that rare words become important. For this procedure, we require a pretrained language model $M$ as a baseline, an arbitrary text classification dataset $\mathcal {D}$ containing labelled instances $(\mathbf {x}, y)$ and a substitution dictionary $S$, mapping each word $w$ to a set of rare synonyms $S(w)$. Given these ingredients, our procedure consists of three steps: (i) splitting the dataset into a train set and a set of test candidates, (ii) training the baseline model on the train set and (iii) modifying a subset of the test candidates to generate the final test set.

Evaluation ::: Setup	For our evaluation of Bertram, we largely follow the experimental setup of . Our implementation of Bertram is based on PyTorch  and the Transformers library of . Throughout all of our experiments, we use BERT$_\text{base}$ as the underlying language model for Bertram. To obtain embeddings for frequent multi-token words during training, we use one-token approximation . Somewhat surprisingly, we found in preliminary experiments that excluding BERT's parameters from the finetuning procedure outlined in Section SECREF17 improves performance while speeding up training; we thus exclude them in the third step of our training procedure. While BERT was trained on BooksCorpus  and a large Wikipedia dump, we follow previous work and train Bertram on only the much smaller Westbury Wikipedia Corpus (WWC) ; this of course gives BERT a clear advantage over our proposed method. In order to at least partially compensate for this, in our downstream task experiments we gather the set of contexts $\mathcal {C}$ for a given rare word from both the WWC and BooksCorpus during inference.

Generation of Rare Word Datasets ::: Baseline Training	We finetune $M$ on $\mathcal {D}_\text{train}$. Let $(\mathbf {x}, y) \in \mathcal {D}_\text{train}$ where $\mathbf {x} = w_1, \ldots , w_n$ is a sequence of words. We deviate from the standard finetuning procedure of  in three respects: We randomly replace 5% of all words in $\mathbf {x}$ with a [MASK] token. This allows the model to cope with missing or unknown words, a prerequisite for our final test set generation. As an alternative to overwriting the language model's uncontextualized embeddings for rare words, we also want to allow models to simply add an alternative representation during test time, in which case we simply separate both representations by a slash. To accustom the language model to this duplication of words, we replace each word $w_i$ with “$w_i$ / $w_i$” with a probability of 10%.

the Stanford Rare Word dataset of , the Definitional Nonce dataset of  and the Contextual Rare Word dataset . However, all of these datasets are only suitable for evaluating context-independent word representations. Our proposed method of generating rare word datasets is loosely related to adversarial example generation methods such as HotFlip , which manipulate the input to change a model's prediction. We use a similar mechanism to determine which words in a given sentence are most important and replace these words with rare synonyms.

Related Work	Incorporating surface-form information (e.g., morphemes, characters or character $n$-grams) is a commonly used technique for improving word representations. For context-independent word embeddings, this information can either be injected into a given embedding space , , or a model can directly be given access to it during training , , . In the area of contextualized representations, many architectures employ subword segmentation methods , , , , whereas others use convolutional neural networks to directly access character-level information , , . Complementary to surface form, another useful source of information for understanding rare words are the contexts in which they occur , , . As recently shown by , , combining form and context leads to significantly better results than using just one of both input signals for a wide range of tasks. While all aforementioned methods are based on simple bag-of-words models,  recently proposed an architecture based on the context2vec language model . However, in contrast to our work, they (i) do not incorporate surface-form information and (ii) do not directly access the hidden states of the language model, but instead simply use its output distribution. There are several datasets explicitly focusing on rare words, e.g.

Generation of Rare Word Datasets ::: Dataset Splitting	We partition $\mathcal {D}$ into a train set $\mathcal {D}_\text{train}$ and a set of test candidates, $\mathcal {D}_\text{cand}$, with the latter containing all instances $(\mathbf {x},y) \in \mathcal {D}$ such that for at least one word $w$ in $\mathbf {x}$, $S(w) \ne \emptyset $. Additionally, we require that the training set consists of at least one third of the entire data.

Model ::: Training	Like previous work, we use mimicking  as a training objective. That is, given a frequent word $w$ with known embedding $e_w$ and a set of corresponding contexts $\mathcal {C}$, Bertram is trained to minimize $\Vert e_w - v_{(w, \mathcal {C})}\Vert ^2$. As training Bertram end-to-end requires much computation (processing a single training instance $(w,\mathcal {C})$ is as costly as processing an entire batch of $|\mathcal {C}|$ examples in the original BERT architecture), we resort to the following three-stage training process: We train only the form part, i.e. our loss for a single example $(w, \mathcal {C})$ is $\Vert e_w - v^\text{form}_{(w, \mathcal {C})} \Vert ^2$. We train only the context part, minimizing $\Vert e_w - A \cdot v^\text{context}_{(w, \mathcal {C})} \Vert ^2$ where the context embedding is obtained using the shallow variant of Bertram.

Introduction	As traditional word embedding algorithms  are known to struggle with rare words, several techniques for improving their representations have been proposed over the last few years. These approaches exploit either the contexts in which rare words occur , , , , their surface-form , , , or both , . However, all of these approaches are designed for and evaluated on uncontextualized word embeddings. With the recent shift towards contextualized representations obtained from pretrained deep language models , , , , the question naturally arises whether these approaches are facing the same problem. As all of them already handle rare words implicitly – using methods such as byte-pair encoding  and WordPiece embeddings , or even character-level CNNs  –, it is unclear whether these models even require special treatment of rare words. However, the listed methods only make use of surface-form information, whereas  found that for covering a wide range of rare words, it is crucial to consider both surface-form and contexts. Consistently,  recently showed that for BERT , a popular pretrained language model based on a Transformer architecture , performance on a rare word probing task can significantly be improve by relearning representations of rare words using Attentive Mimicking .

Generation of Rare Word Datasets ::: Test Set Generation	Let $p(y \mid \mathbf {x})$ be the probability that the finetuned model $M$ assigns to class $y$ given input $\mathbf {x}$, and let be the model's prediction for input $\mathbf {x}$ where $\mathcal {Y}$ denotes the set of all labels. For generating our test set, we only consider candidates that are classified correctly by the baseline model, i.e. candidates $(\mathbf {x}, y) \in \mathcal {D}_\text{cand}$ with $M(\mathbf {x}) = y$. For each such entry, let $\mathbf {x} = w_1, \ldots , w_n$ and let $\mathbf {x}_{w_i = t}$ be the sequence obtained from $\mathbf {x}$ by replacing $w_i$ with $t$. We compute i.e., we select the word $w_i$ whose masking pushes the model's prediction the furthest away from the correct label.

If the model's internal representation of each $\hat{w}_{i_j}$ is equal to its representation of $w_{i_j}$, the entry is classified correctly by $M$. That is, if the model is able to understand the rare words $\hat{w}_{i_j}$ and to identify them as synonyms of ${w_{i_j}}$, it predicts the correct label for each instance. It is important to notice that the so-obtained test set is very closely coupled to the baseline model $M$, because we selected the words to replace based on the model's predictions. Importantly, however, the model is never queried with any rare synonym during test set generation, so its representations of rare words are not taken into account for creating the test set. Thus, while the test set is not suitable for comparing $M$ with an entirely different model $M^{\prime }$, it allows us to compare various strategies for representing rare words in the embedding space of $M$. A similar constraint can be found in the Definitional Nonce dataset , which is tied to a given embedding space based on Word2Vec .

Furthermore, we exclude all of BERT's parameters from our optimization. We combine the pretrained form-only and context-only model and train all additional parameters. Pretraining the form and context parts individually allows us to train the full model for much fewer steps with comparable results. Importantly, for the first two stages of our training procedure, we do not have to backpropagate through the entire BERT model to obtain all required gradients, drastically increasing the training speed.

To make sure that the model does not simply learn to always focus on the first instance during training, we randomly mask each of the two repetitions with probability 25%. We do not finetune the model's embedding layer. In preliminary experiments, we found this not to hurt performance.

Just like for WNLaMPro, our default way of injecting Bertram embeddings into the baseline model is to replace the sequence of uncontextualized WordPiece tokens for a given rare word with its Bertram-based embedding. That is, given a sequence of uncontextualized token embeddings $\mathbf {e} = e_1, \ldots , e_n$ where $e_{i}, \ldots , e_{i+j}$ with $1 \le i \le i+j \le n$ is the sequence of WordPiece embeddings for a single rare word $w$, we replace $\mathbf {e}$ with By default, the set of contexts $\mathcal {C}$ required for this replacement is obtained by collecting all sentences from the WWC and BooksCorpus in which $w$ occurs. As our model architecture allows us to easily include new contexts without requiring any additional training, we also try a variant where we add in-domain contexts by giving the model access to the texts found in the test set.

As can be seen, supplementing BERT with any of the proposed relearning methods results in noticeable improvements for the rare subset, with add clearly outperforming replace. Moreover, the add and add-gated variants of Bertram perform surprisingly well for more frequent words, improving the score for WNLaMPro-medium by 50% compared to BERT$_\text{base}$ and 31% compared to Attentive Mimicking. This makes sense considering that compared to Attentive Mimicking, the key enhancement of Bertram lies in improving context representations and interconnection of form and context; naturally, the more contexts are given, the more this comes into play. Noticeably, despite being both based on and integrated into a BERT$_\text{base}$ model, our architecture even outperforms a standalone BERT$_\text{large}$ model by a large margin.

If removing this word already changes the model's prediction – that is, $M(\mathbf {x}_{w_i = \texttt {[MASK]}}) \ne y$ –, we select a random rare synonym $\hat{w}_i \in S(w_i)$ and add $(\mathbf {x}_{w_i = \hat{w}_i}, y)$ to the test set. Otherwise, we repeat the above procedure; if the label still has not changed after masking up to 5 words, we discard the corresponding entry. All so-obtained test set entries $(\mathbf {x}_{w_{i_1} = \hat{w}_{i_1}, \ldots , w_{i_k} = \hat{w}_{i_k} }, y)$ have the following properties: If each $w_{i_j}$ is replaced by a [MASK] token, the entry is classified incorrectly by $M$. In other words, understanding the words $w_{i_j}$ is essential for $M$ to determine the correct label.

Model ::: Form-Context Model	We review the architecture of the form-context model (FCM) , which forms the basis for our model. Given a set of $d$-dimensional high-quality embeddings for frequent words, FCM can be used to induce embeddings for infrequent words that are appropriate for the given embedding space. This is done as follows: Given a word $w$ and a context $C$ in which it occurs, a surface-form embedding $v_{(w,{C})}^\text{form} \in \mathbb {R}^d$ is obtained similar to  by averaging over embeddings of all $n$-grams in $w$; these $n$-gram embeddings are learned during training. Similarly, a context embedding $v_{(w,{C})}^\text{context} \in \mathbb {R}^d$ is obtained by averaging over the embeddings of all words in $C$.

The so-obtained form and context embeddings are then combined using a gate with parameters $w \in \mathbb {R}^{2d}, b \in \mathbb {R}$ and $\sigma $ denoting the sigmoid function, allowing the model to decide for each pair $(x,y)$ of form and context embeddings how much attention should be paid to $x$ and $y$, respectively. The final representation of $w$ is then simply a weighted sum of form and context embeddings: where $\alpha = g(v_{(w,C)}^\text{form}, v_{(w,C)}^\text{context})$ and $A$ is a $d\times d$ matrix that is learned during training. While the context-part of FCM is able to capture the broad topic of numerous rare words, in many cases it is not able to obtain a more concrete and detailed understanding thereof . This is hardly surprising given the model's simplicity; it does, for example, make no use at all of the relative positions of context words. Furthermore, the simple gating mechanism results in only a shallow combination of form and context.

That is, the model is not able to combine form and context until the very last step: While it can choose how much to attend to form and context, respectively, the corresponding embeddings do not share any information and thus cannot influence each other in any way.

Model ::: Bertram	To overcome both limitations described above, we introduce Bertram, an approach that combines a pretrained BERT language model  with Attentive Mimicking . To this end, let $d_h$ be the hidden dimension size and $l_\text{max}$ be the number of layers for the BERT model being used. We denote with $e_{t}$ the (uncontextualized) embedding assigned to a token $t$ by BERT and, given a sequence of such uncontextualized embeddings $\mathbf {e} = e_1, \ldots , e_n$, we denote by $\textbf {h}_j^l(\textbf {e})$ the contextualized representation of the $j$-th token at layer $l$ when the model is given $\mathbf {e}$ as input.

Given a word $w$ and a context $C = w_1, \ldots , w_n$ in which it occurs, let $\mathbf {t} = t_1, \ldots , t_{m}$ with $m \ge n$ be the sequence obtained from $C$ by (i) replacing $w$ with a [MASK] token and (ii) tokenizing the so-obtained sequence to match the BERT vocabulary; furthermore, let $i$ denote the index for which $t_i = \texttt {[MASK]}$. Perhaps the most simple approach for obtaining a context embedding from $C$ using BERT is to define where $\mathbf {e} = e_{t_1}, \ldots , e_{t_m}$. The so-obtained context embedding can then be combined with its form counterpart as described in Eq. DISPLAY_FORM8. While this achieves our first goal of using a more sophisticated context model that can potentially gain a deeper understanding of a word than just its broad topic, the so-obtained architecture still only combines form and context in a shallow fashion.

We thus refer to it as the shallow variant of our model and investigate two alternative approaches (replace and add) that work as follows: Replace: Before computing the context embedding, we replace the uncontextualized embedding of the [MASK] token with the word's surface-form embedding: As during BERT pretraining, words chosen for prediction are replaced with [MASK] tokens only 80% of the time and kept unchanged 10% of the time, we hypothesize that even without further training, BERT is able to make use of form embeddings ingested this way. Add: Before computing the context embedding, we prepad the input with the surface-form embedding of $w$, followed by a colon: We also experimented with various other prefixes, but ended up choosing this particular strategy because we empirically found that after masking a token $t$, adding the sequence “$t :$” at the beginning helps BERT the most in recovering this very token at the masked position.

tnode/.style=rectangle, inner sep=0.1cm, minimum height=4ex, text centered,text height=1.5ex, text depth=0.25ex, opnode/.style=draw, rectangle, rounded corners, minimum height=4ex, minimum width=4ex, text centered, arrow/.style=draw,->,>=stealth As for both variants, surface-form information is directly and deeply integrated into the computation of the context embedding, we do not require any further gating mechanism and may directly set $v_{(w,C)} = A \cdot v^\text{context}_{(w,C)}$. However, we note that for the add variant, the contextualized representation of the [MASK] token is not the only natural candidate to be used for computing the final embedding: We might just as well look at the contextualized representation of the surface-form based embedding added at the very first position. Therefore, we also try a shallow combination of both embeddings. Note, however, that unlike FCM, we combine the contextualized representations – that is, the form part was already influenced by the context part and vice versa before combining them using a gate.

For this combination, we define with $A^{\prime } \in \mathbb {R}^{d \times d_h}$ being an additional learnable parameter. We then combine the two contextualized embeddings similar to Eq. DISPLAY_FORM8 as where $\alpha = g(h^\text{form}_{(w,C)}, h^\text{context}_{(w,C)})$. We refer to this final alternative as the add-gated approach. The model architecture for this variant can be seen in Figure FIGREF14 (left). As in many cases, not just one, but a handful of contexts is known for a rare word, we follow the approach of  to deal with multiple contexts: We add an Attentive Mimicking head on top of our model, as can be seen in Figure FIGREF14 (right). That is, given a set of contexts $\mathcal {C} = \lbrace C_1, \ldots , C_m\rbrace $ and the corresponding embeddings $v_{(w,C_1)}, \ldots , v_{(w,C_m)}$, we apply a self-attention mechanism to all embeddings, allowing the model to distinguish informative contexts from uninformative ones.

The final embedding $v_{(w, \mathcal {C})}$ is then a linear combination of the embeddings obtained from each context, where the weight of each embedding is determined based on the self-attention layer. For further details on this mechanism, we refer to .

However, their proposed model is limited in two important respects: For processing contexts, it uses a simple bag-of-words model, throwing away much of the available information. It combines form and context only in a shallow fashion, thus preventing both input signals from sharing information in any sophisticated manner. Importantly, this limitation applies not only to their model, but to all previous work on obtaining representations for rare words by leveraging form and context. While using bag-of-words models is a reasonable choice for uncontextualized embeddings, which are often themselves based on such models , , it stands to reason that they are suboptimal for contextualized embeddings based on position-aware deep neural architectures. To overcome these limitations, we introduce Bertram (BERT for Attentive Mimicking), a novel architecture for understanding rare words that combines a pretrained BERT language model with Attentive Mimicking . Unlike previous approaches making use of language models , our approach integrates BERT in an end-to-end fashion and directly makes use of its hidden states. By giving Bertram access to both surface form and context information already at its very lowest layer, we allow for a deep connection and exchange of information between both input signals.

Question: What are three downstream task datasets?
Answer: MNLI , AG's News , DBPedia