Introduction
As traditional word embedding algorithms BIBREF1 are known to struggle with rare words, several techniques for improving their representations have been proposed over the last few years. These approaches exploit either the contexts in which rare words occur BIBREF2, BIBREF3, BIBREF4, BIBREF5, their surface-form BIBREF6, BIBREF7, BIBREF8, or both BIBREF9, BIBREF10. However, all of these approaches are designed for and evaluated on uncontextualized word embeddings.
With the recent shift towards contextualized representations obtained from pretrained deep language models BIBREF11, BIBREF12, BIBREF13, BIBREF14, the question naturally arises whether these approaches are facing the same problem. As all of them already handle rare words implicitly – using methods such as byte-pair encoding BIBREF15 and WordPiece embeddings BIBREF16, or even character-level CNNs BIBREF17 –, it is unclear whether these models even require special treatment of rare words. However, the listed methods only make use of surface-form information, whereas BIBREF9 found that for covering a wide range of rare words, it is crucial to consider both surface-form and contexts.
Consistently, BIBREF0 recently showed that for BERT BIBREF13, a popular pretrained language model based on a Transformer architecture BIBREF18, performance on a rare word probing task can significantly be improve by relearning representations of rare words using Attentive Mimicking BIBREF19. However, their proposed model is limited in two important respects:
For processing contexts, it uses a simple bag-of-words model, throwing away much of the available information.
It combines form and context only in a shallow fashion, thus preventing both input signals from sharing information in any sophisticated manner.
Importantly, this limitation applies not only to their model, but to all previous work on obtaining representations for rare words by leveraging form and context. While using bag-of-words models is a reasonable choice for uncontextualized embeddings, which are often themselves based on such models BIBREF1, BIBREF7, it stands to reason that they are suboptimal for contextualized embeddings based on position-aware deep neural architectures.
To overcome these limitations, we introduce Bertram (BERT for Attentive Mimicking), a novel architecture for understanding rare words that combines a pretrained BERT language model with Attentive Mimicking BIBREF19. Unlike previous approaches making use of language models BIBREF5, our approach integrates BERT in an end-to-end fashion and directly makes use of its hidden states. By giving Bertram access to both surface form and context information already at its very lowest layer, we allow for a deep connection and exchange of information between both input signals.
For various reasons, assessing the effectiveness of methods like Bertram in a contextualized setting poses a huge difficulty: While most previous work on rare words was evaluated on datasets explicitly focusing on such words BIBREF6, BIBREF3, BIBREF4, BIBREF5, BIBREF10, all of these datasets are tailored towards context-independent embeddings and thus not suitable for evaluating our proposed model. Furthermore, understanding rare words is of negligible importance for most commonly used downstream task datasets. To evaluate our proposed model, we therefore introduce a novel procedure that allows us to automatically turn arbitrary text classification datasets into ones where rare words are guaranteed to be important. This is achieved by replacing classification-relevant frequent words with rare synonyms obtained using semantic resources such as WordNet BIBREF20.
Using this procedure, we extract rare word datasets from three commonly used text (or text pair) classification datasets: MNLI BIBREF21, AG's News BIBREF22 and DBPedia BIBREF23. On both the WNLaMPro dataset of BIBREF0 and all three so-obtained datasets, our proposed Bertram model outperforms previous work by a large margin.
In summary, our contributions are as follows:
We show that a pretrained BERT instance can be integrated into Attentive Mimicking, resulting in much better context representations and a deeper connection of form and context.
We design a procedure that allows us to automatically transform text classification datasets into datasets for which rare words are guaranteed to be important.
We show that Bertram achieves a new state-of-the-art on the WNLaMPro probing task BIBREF0 and beats all baselines on rare word instances of AG's News, MNLI and DBPedia, resulting in an absolute improvement of up to 24% over a BERT baseline.
Related Work
Incorporating surface-form information (e.g., morphemes, characters or character $n$-grams) is a commonly used technique for improving word representations. For context-independent word embeddings, this information can either be injected into a given embedding space BIBREF6, BIBREF8, or a model can directly be given access to it during training BIBREF7, BIBREF24, BIBREF25. In the area of contextualized representations, many architectures employ subword segmentation methods BIBREF12, BIBREF13, BIBREF26, BIBREF14, whereas others use convolutional neural networks to directly access character-level information BIBREF27, BIBREF11, BIBREF17.
Complementary to surface form, another useful source of information for understanding rare words are the contexts in which they occur BIBREF2, BIBREF3, BIBREF4. As recently shown by BIBREF19, BIBREF9, combining form and context leads to significantly better results than using just one of both input signals for a wide range of tasks. While all aforementioned methods are based on simple bag-of-words models, BIBREF5 recently proposed an architecture based on the context2vec language model BIBREF28. However, in contrast to our work, they (i) do not incorporate surface-form information and (ii) do not directly access the hidden states of the language model, but instead simply use its output distribution.
There are several datasets explicitly focusing on rare words, e.g. the Stanford Rare Word dataset of BIBREF6, the Definitional Nonce dataset of BIBREF3 and the Contextual Rare Word dataset BIBREF4. However, all of these datasets are only suitable for evaluating context-independent word representations.
Our proposed method of generating rare word datasets is loosely related to adversarial example generation methods such as HotFlip BIBREF29, which manipulate the input to change a model's prediction. We use a similar mechanism to determine which words in a given sentence are most important and replace these words with rare synonyms.
Model ::: Form-Context Model
We review the architecture of the form-context model (FCM) BIBREF9, which forms the basis for our model. Given a set of $d$-dimensional high-quality embeddings for frequent words, FCM can be used to induce embeddings for infrequent words that are appropriate for the given embedding space. This is done as follows: Given a word $w$ and a context $C$ in which it occurs, a surface-form embedding $v_{(w,{C})}^\text{form} \in \mathbb {R}^d$ is obtained similar to BIBREF7 by averaging over embeddings of all $n$-grams in $w$; these $n$-gram embeddings are learned during training. Similarly, a context embedding $v_{(w,{C})}^\text{context} \in \mathbb {R}^d$ is obtained by averaging over the embeddings of all words in $C$. The so-obtained form and context embeddings are then combined using a gate
with parameters $w \in \mathbb {R}^{2d}, b \in \mathbb {R}$ and $\sigma $ denoting the sigmoid function, allowing the model to decide for each pair $(x,y)$ of form and context embeddings how much attention should be paid to $x$ and $y$, respectively.
The final representation of $w$ is then simply a weighted sum of form and context embeddings:
where $\alpha = g(v_{(w,C)}^\text{form}, v_{(w,C)}^\text{context})$ and $A$ is a $d\times d$ matrix that is learned during training.
While the context-part of FCM is able to capture the broad topic of numerous rare words, in many cases it is not able to obtain a more concrete and detailed understanding thereof BIBREF9. This is hardly surprising given the model's simplicity; it does, for example, make no use at all of the relative positions of context words. Furthermore, the simple gating mechanism results in only a shallow combination of form and context. That is, the model is not able to combine form and context until the very last step: While it can choose how much to attend to form and context, respectively, the corresponding embeddings do not share any information and thus cannot influence each other in any way.
Model ::: Bertram
To overcome both limitations described above, we introduce Bertram, an approach that combines a pretrained BERT language model BIBREF13 with Attentive Mimicking BIBREF19. To this end, let $d_h$ be the hidden dimension size and $l_\text{max}$ be the number of layers for the BERT model being used. We denote with $e_{t}$ the (uncontextualized) embedding assigned to a token $t$ by BERT and, given a sequence of such uncontextualized embeddings $\mathbf {e} = e_1, \ldots , e_n$, we denote by $\textbf {h}_j^l(\textbf {e})$ the contextualized representation of the $j$-th token at layer $l$ when the model is given $\mathbf {e}$ as input.
Given a word $w$ and a context $C = w_1, \ldots , w_n$ in which it occurs, let $\mathbf {t} = t_1, \ldots , t_{m}$ with $m \ge n$ be the sequence obtained from $C$ by (i) replacing $w$ with a [MASK] token and (ii) tokenizing the so-obtained sequence to match the BERT vocabulary; furthermore, let $i$ denote the index for which $t_i = \texttt {[MASK]}$. Perhaps the most simple approach for obtaining a context embedding from $C$ using BERT is to define
where $\mathbf {e} = e_{t_1}, \ldots , e_{t_m}$. The so-obtained context embedding can then be combined with its form counterpart as described in Eq. DISPLAY_FORM8. While this achieves our first goal of using a more sophisticated context model that can potentially gain a deeper understanding of a word than just its broad topic, the so-obtained architecture still only combines form and context in a shallow fashion. We thus refer to it as the shallow variant of our model and investigate two alternative approaches (replace and add) that work as follows:
Replace: Before computing the context embedding, we replace the uncontextualized embedding of the [MASK] token with the word's surface-form embedding:
As during BERT pretraining, words chosen for prediction are replaced with [MASK] tokens only 80% of the time and kept unchanged 10% of the time, we hypothesize that even without further training, BERT is able to make use of form embeddings ingested this way.
Add: Before computing the context embedding, we prepad the input with the surface-form embedding of $w$, followed by a colon:
We also experimented with various other prefixes, but ended up choosing this particular strategy because we empirically found that after masking a token $t$, adding the sequence “$t :$” at the beginning helps BERT the most in recovering this very token at the masked position.
tnode/.style=rectangle, inner sep=0.1cm, minimum height=4ex, text centered,text height=1.5ex, text depth=0.25ex, opnode/.style=draw, rectangle, rounded corners, minimum height=4ex, minimum width=4ex, text centered, arrow/.style=draw,->,>=stealth
As for both variants, surface-form information is directly and deeply integrated into the computation of the context embedding, we do not require any further gating mechanism and may directly set $v_{(w,C)} = A \cdot v^\text{context}_{(w,C)}$.
However, we note that for the add variant, the contextualized representation of the [MASK] token is not the only natural candidate to be used for computing the final embedding: We might just as well look at the contextualized representation of the surface-form based embedding added at the very first position. Therefore, we also try a shallow combination of both embeddings. Note, however, that unlike FCM, we combine the contextualized representations – that is, the form part was already influenced by the context part and vice versa before combining them using a gate. For this combination, we define
with $A^{\prime } \in \mathbb {R}^{d \times d_h}$ being an additional learnable parameter. We then combine the two contextualized embeddings similar to Eq. DISPLAY_FORM8 as
where $\alpha = g(h^\text{form}_{(w,C)}, h^\text{context}_{(w,C)})$. We refer to this final alternative as the add-gated approach. The model architecture for this variant can be seen in Figure FIGREF14 (left).
As in many cases, not just one, but a handful of contexts is known for a rare word, we follow the approach of BIBREF19 to deal with multiple contexts: We add an Attentive Mimicking head on top of our model, as can be seen in Figure FIGREF14 (right). That is, given a set of contexts $\mathcal {C} = \lbrace C_1, \ldots , C_m\rbrace $ and the corresponding embeddings $v_{(w,C_1)}, \ldots , v_{(w,C_m)}$, we apply a self-attention mechanism to all embeddings, allowing the model to distinguish informative contexts from uninformative ones. The final embedding $v_{(w, \mathcal {C})}$ is then a linear combination of the embeddings obtained from each context, where the weight of each embedding is determined based on the self-attention layer. For further details on this mechanism, we refer to BIBREF19.
Model ::: Training
Like previous work, we use mimicking BIBREF8 as a training objective. That is, given a frequent word $w$ with known embedding $e_w$ and a set of corresponding contexts $\mathcal {C}$, Bertram is trained to minimize $\Vert e_w - v_{(w, \mathcal {C})}\Vert ^2$.
As training Bertram end-to-end requires much computation (processing a single training instance $(w,\mathcal {C})$ is as costly as processing an entire batch of $|\mathcal {C}|$ examples in the original BERT architecture), we resort to the following three-stage training process:
We train only the form part, i.e. our loss for a single example $(w, \mathcal {C})$ is $\Vert e_w - v^\text{form}_{(w, \mathcal {C})} \Vert ^2$.
We train only the context part, minimizing $\Vert e_w - A \cdot v^\text{context}_{(w, \mathcal {C})} \Vert ^2$ where the context embedding is obtained using the shallow variant of Bertram. Furthermore, we exclude all of BERT's parameters from our optimization.
We combine the pretrained form-only and context-only model and train all additional parameters.
Pretraining the form and context parts individually allows us to train the full model for much fewer steps with comparable results. Importantly, for the first two stages of our training procedure, we do not have to backpropagate through the entire BERT model to obtain all required gradients, drastically increasing the training speed.
Generation of Rare Word Datasets
To measure the quality of rare word representations in a contextualized setting, we would ideally need text classification datasets with the following two properties:
A model that has no understanding of rare words at all should perform close to 0%.
A model that perfectly understands rare words should be able to classify every instance correctly.
Unfortunately, this requirement is not even remotely fulfilled by most commonly used datasets, simply because rare words occur in only a few entries and when they do, they are often of negligible importance.
To solve this problem, we devise a procedure to automatically transform existing text classification datasets such that rare words become important. For this procedure, we require a pretrained language model $M$ as a baseline, an arbitrary text classification dataset $\mathcal {D}$ containing labelled instances $(\mathbf {x}, y)$ and a substitution dictionary $S$, mapping each word $w$ to a set of rare synonyms $S(w)$. Given these ingredients, our procedure consists of three steps: (i) splitting the dataset into a train set and a set of test candidates, (ii) training the baseline model on the train set and (iii) modifying a subset of the test candidates to generate the final test set.
Generation of Rare Word Datasets ::: Dataset Splitting
We partition $\mathcal {D}$ into a train set $\mathcal {D}_\text{train}$ and a set of test candidates, $\mathcal {D}_\text{cand}$, with the latter containing all instances $(\mathbf {x},y) \in \mathcal {D}$ such that for at least one word $w$ in $\mathbf {x}$, $S(w) \ne \emptyset $. Additionally, we require that the training set consists of at least one third of the entire data.
Generation of Rare Word Datasets ::: Baseline Training
We finetune $M$ on $\mathcal {D}_\text{train}$. Let $(\mathbf {x}, y) \in \mathcal {D}_\text{train}$ where $\mathbf {x} = w_1, \ldots , w_n$ is a sequence of words. We deviate from the standard finetuning procedure of BIBREF13 in three respects:
We randomly replace 5% of all words in $\mathbf {x}$ with a [MASK] token. This allows the model to cope with missing or unknown words, a prerequisite for our final test set generation.
As an alternative to overwriting the language model's uncontextualized embeddings for rare words, we also want to allow models to simply add an alternative representation during test time, in which case we simply separate both representations by a slash. To accustom the language model to this duplication of words, we replace each word $w_i$ with “$w_i$ / $w_i$” with a probability of 10%. To make sure that the model does not simply learn to always focus on the first instance during training, we randomly mask each of the two repetitions with probability 25%.
We do not finetune the model's embedding layer. In preliminary experiments, we found this not to hurt performance.
Generation of Rare Word Datasets ::: Test Set Generation
Let $p(y \mid \mathbf {x})$ be the probability that the finetuned model $M$ assigns to class $y$ given input $\mathbf {x}$, and let
be the model's prediction for input $\mathbf {x}$ where $\mathcal {Y}$ denotes the set of all labels. For generating our test set, we only consider candidates that are classified correctly by the baseline model, i.e. candidates $(\mathbf {x}, y) \in \mathcal {D}_\text{cand}$ with $M(\mathbf {x}) = y$. For each such entry, let $\mathbf {x} = w_1, \ldots , w_n$ and let $\mathbf {x}_{w_i = t}$ be the sequence obtained from $\mathbf {x}$ by replacing $w_i$ with $t$. We compute
i.e., we select the word $w_i$ whose masking pushes the model's prediction the furthest away from the correct label. If removing this word already changes the model's prediction – that is, $M(\mathbf {x}_{w_i = \texttt {[MASK]}}) \ne y$ –, we select a random rare synonym $\hat{w}_i \in S(w_i)$ and add $(\mathbf {x}_{w_i = \hat{w}_i}, y)$ to the test set. Otherwise, we repeat the above procedure; if the label still has not changed after masking up to 5 words, we discard the corresponding entry. All so-obtained test set entries $(\mathbf {x}_{w_{i_1} = \hat{w}_{i_1}, \ldots , w_{i_k} = \hat{w}_{i_k} }, y)$ have the following properties:
If each $w_{i_j}$ is replaced by a [MASK] token, the entry is classified incorrectly by $M$. In other words, understanding the words $w_{i_j}$ is essential for $M$ to determine the correct label.
If the model's internal representation of each $\hat{w}_{i_j}$ is equal to its representation of $w_{i_j}$, the entry is classified correctly by $M$. That is, if the model is able to understand the rare words $\hat{w}_{i_j}$ and to identify them as synonyms of ${w_{i_j}}$, it predicts the correct label for each instance.
It is important to notice that the so-obtained test set is very closely coupled to the baseline model $M$, because we selected the words to replace based on the model's predictions. Importantly, however, the model is never queried with any rare synonym during test set generation, so its representations of rare words are not taken into account for creating the test set. Thus, while the test set is not suitable for comparing $M$ with an entirely different model $M^{\prime }$, it allows us to compare various strategies for representing rare words in the embedding space of $M$. A similar constraint can be found in the Definitional Nonce dataset BIBREF3, which is tied to a given embedding space based on Word2Vec BIBREF1.
Evaluation ::: Setup
For our evaluation of Bertram, we largely follow the experimental setup of BIBREF0. Our implementation of Bertram is based on PyTorch BIBREF30 and the Transformers library of BIBREF31. Throughout all of our experiments, we use BERT$_\text{base}$ as the underlying language model for Bertram. To obtain embeddings for frequent multi-token words during training, we use one-token approximation BIBREF0. Somewhat surprisingly, we found in preliminary experiments that excluding BERT's parameters from the finetuning procedure outlined in Section SECREF17 improves performance while speeding up training; we thus exclude them in the third step of our training procedure.
While BERT was trained on BooksCorpus BIBREF32 and a large Wikipedia dump, we follow previous work and train Bertram on only the much smaller Westbury Wikipedia Corpus (WWC) BIBREF33; this of course gives BERT a clear advantage over our proposed method. In order to at least partially compensate for this, in our downstream task experiments we gather the set of contexts $\mathcal {C}$ for a given rare word from both the WWC and BooksCorpus during inference.
Evaluation ::: WNLaMPro
We evalute Bertram on the WNLaMPro dataset of BIBREF0. This dataset consists of cloze-style phrases like
and the task is to correctly fill the slot (____) with one of several acceptable target words (e.g., “fruit”, “bush” and “berry”), which requires knowledge of the phrase's keyword (“lingonberry” in the above example). As the goal of this dataset is to probe a language model's ability to understand rare words without any task-specific finetuning, BIBREF0 do not provide a training set. Furthermore, the dataset is partitioned into three subsets; this partition is based on the frequency of the keyword, with keywords occurring less than 10 times in the WWC forming the rare subset, those occurring between 10 and 100 times forming the medium subset, and all remaining words forming the frequent subset. As our focus is on improving representations for rare words, we evaluate our model only on the former two sets.
Results on WNLaMPro rare and medium are shown in Table TABREF34, where the mean reciprocal rank (MRR) is reported for BERT, Attentive Mimicking and Bertram. As can be seen, supplementing BERT with any of the proposed relearning methods results in noticeable improvements for the rare subset, with add clearly outperforming replace. Moreover, the add and add-gated variants of Bertram perform surprisingly well for more frequent words, improving the score for WNLaMPro-medium by 50% compared to BERT$_\text{base}$ and 31% compared to Attentive Mimicking. This makes sense considering that compared to Attentive Mimicking, the key enhancement of Bertram lies in improving context representations and interconnection of form and context; naturally, the more contexts are given, the more this comes into play. Noticeably, despite being both based on and integrated into a BERT$_\text{base}$ model, our architecture even outperforms a standalone BERT$_\text{large}$ model by a large margin.
Evaluation ::: Downstream Task Datasets
To measure the effect of adding Bertram to BERT on downstream tasks, we apply the procedure described in Section SECREF4 to a commonly used textual entailment dataset as well as two text classification datasets: MNLI BIBREF21, AG's News BIBREF22 and DBPedia BIBREF23. For all three datasets, we use BERT$_\text{base}$ as a baseline model and create the substitution dictionary $S$ using the synonym relation of WordNet BIBREF20 and the pattern library BIBREF34 to make sure that all synonyms have consistent parts of speech. As an additional source of word substitutions, we make use of the misspellings dataset of BIBREF25, which is based on query logs of a search engine. To prevent misspellings from dominating the resulting dataset, we only assign misspelling-based substitutes to randomly selected 10% of the words contained in each sentence. Motivated by the results on WNLaMPro-medium, we consider every word that occurs less than 100 times in the WWC and our BooksCorpus replica combined as being rare. Some examples of entries in the resulting datasets can be seen in Table TABREF35.
Just like for WNLaMPro, our default way of injecting Bertram embeddings into the baseline model is to replace the sequence of uncontextualized WordPiece tokens for a given rare word with its Bertram-based embedding. That is, given a sequence of uncontextualized token embeddings $\mathbf {e} = e_1, \ldots , e_n$ where $e_{i}, \ldots , e_{i+j}$ with $1 \le i \le i+j \le n$ is the sequence of WordPiece embeddings for a single rare word $w$, we replace $\mathbf {e}$ with
By default, the set of contexts $\mathcal {C}$ required for this replacement is obtained by collecting all sentences from the WWC and BooksCorpus in which $w$ occurs. As our model architecture allows us to easily include new contexts without requiring any additional training, we also try a variant where we add in-domain contexts by giving the model access to the texts found in the test set.
In addition to the procedure described above, we also try a variant where instead of replacing the original WordPiece embeddings for a given rare word, we merely add the Bertram-based embedding, separating both representations using a single slash:
As it performs best on the rare and medium subsets of WNLaMPro combined, we use only the add-gated variant of Bertram for all datasets. Results can be seen in Table TABREF37, where for each task, we report the accuracy on the entire dataset as well as scores obtained considering only instances where at least one word was replaced by a misspelling or a WordNet synonym, respectively. Consistent with results on WNLaMPro, combining BERT with Bertram outperforms both a standalone BERT model and one combined with Attentive Mimicking across all tasks. While keeping the original BERT embeddings in addition to Bertram's representation brings no benefit, adding in-domain data clearly helps for two out of three datasets. This makes sense as for rare words, every single additional context can be crucial for gaining a deeper understanding.
To further understand for which words using Bertram is helpful, in Figure FIGREF39 we look at the accuracy of BERT both with and without Bertram on all three tasks as a function of word frequency. That is, we compute the accuracy scores for both models when considering only entries $(\mathbf {x}_{w_{i_1} = \hat{w}_{i_1}, \ldots , w_{i_k} = \hat{w}_{i_k} }, y)$ where each substituted word $\hat{w}_{i_j}$ occurs less than $c_\text{max}$ times in WWC and BooksCorpus, for various values of $c_\text{max}$. As one would expect, $c_\text{max}$ is positively correlated with the accuracies of both models, showing that the rarer a word is, the harder it is to understand. Perhaps more interestingly, for all three datasets the gap between Bertram and BERT remains more or less constant regardless of $c_\text{max}$. This indicates that using Bertram might also be useful for even more frequent words than the ones considered.
Conclusion
We have introduced Bertram, a novel architecture for relearning high-quality representations of rare words. This is achieved by employing a powerful pretrained language model and deeply connecting surface-form and context information. By replacing important words with rare synonyms, we have created various downstream task datasets focusing on rare words; on all of these datasets, Bertram improves over a BERT model without special handling of rare words, demonstrating the usefulness of our proposed method.
As our analysis has shown that even for the most frequent words considered, using Bertram is still beneficial, future work might further investigate the limits of our proposed method. Furthermore, it would be interesting to explore more complex ways of incorporating surface-form information – e.g., by using a character-level CNN similar to the one of BIBREF27 – to balance out the potency of Bertram's form and context parts.

Question: What models other than standalone BERT is new model compared to?
Answer: Only Bert base and Bert large are compared to proposed approach.

Question: How much is representaton improved for rare/medum frequency words compared to standalone BERT and previous work?
Answer: improving the score for WNLaMPro-medium by 50% compared to BERT$_\text{base}$ and 31% compared to Attentive Mimicking

Question: What are three downstream task datasets?
Answer: MNLI BIBREF21, AG's News BIBREF22, DBPedia BIBREF23

MNLI, AG's News, DBPedia

Question: What is dataset for word probing task?
Answer: WNLaMPro dataset