Can we identify word senses from deep contextualized word embeddings without supervision?
By: Leslie Huang
Tags: NLP, Deep Learning, Clustering, Unsupervised Learning

Contextualized representations

Deep contextualized representations have improved performance on an enormous range of NLP tasks, including word sense disambiguation.¹ ELMo was specifically designed to capture polysemy,² and BERT uses bidirectional training to learn context.³ Because BERT and ELMo learn contextualized representations, we expect they encode semantic information not captured by fastText or word2vec.⁴ Indeed, recent research has explored how word sense is encoded in BERT.⁵

In the section below, we describe our data and our methods for extracting embedding representations from BERT, ELMo, and fastText.

Data

We use a recent dump of simple Wikipedia,⁶ which we process using open source software.⁷ The dataset consists of the text from 143,411 Wikipedia articles. For our experiments, we use the 1,000 most frequently occurring tokens (excluding numbers and punctuation) in the entire corpus that appear in both WordNet (via PyDictionary) and Wiktionary.

We extract sentence- (the entire sentence in which the target word appears) and window-based (+/- a window of {5,10} words around the target word) contexts from the corpus.

Next, we turn these contexts into vector representations.

ELMo

We compute the representations of word-in-context samples using a pretrained ELMo model with the default weights and options. We compute the context representation two ways:

“Subtracted context’’ representation: We average the 3 layers of the ELMo representation for each token. We average the representations of all tokens except the target word (we refer to this as context-except-target); subsequently we subtract the context-except-target from the target word’s representation; we call the result the “subtracted context’’ representation. We do this so the target word’s representation does not include the context. (We thank Dirk Hovy for this idea.) Averaged representation: After averaging the 3 layers for each token, we simply average across token representations — including the target word.

BERT

We compute the vector representations using pretrained

BERT model (base BERT, uncased, with 768 hidden units and 12 heads)⁸ and extract the precomputed vector representations using open source software.⁹

“Subtracted context’’ representation: Same as above. Averaged representation: Same as above. Concat CLS: We concatenate the “subtracted context’’ representation

and the “[CLS]’’ classification token representation for that context.

fastText

We use pretrained fastText, which was trained on English Wikipedia.¹⁰ Following the fastText implementation of computing sentence vectors,

we divide each word vector by its L2 norm before averaging the word

vectors together to compute the sentence vector.

“Subtracted context’’: Same as above; we also apply tf-idf weighting¹¹ to each token representation. Unlike the contextualized representations in the previous sections, there is one representation of the target word regardless of context. However, the target-minus-context incorporates the context. Averaged representation: Same as above.

Examples

Do embedding representations actually capture word meaning? We first explore this question graphically using a small set of hand-labeled words from each of the corpora. Though limited, these results show a clear spatial clustering of similar usages of a given word.

Words were chosen specifically because they were thought to have multiple distinct senses. The number of senses was selected after examining the context sentences. We extract the averaged representations of the contexts from each of the three embedding models and subsequently decompose the vectors to 2-D using PCA, UMAP, and t-SNE.

The colors of the points on the figures represent the hand-labeled cluster assignment. Different combinations of the embedding models and decomposition methods generate similar results: a spatial grouping of contexts by color, i.e. according to word sense.

(a) BERT. Orange is the animal, blue (square) is the sports equipment.

(b) ELMo. Orange is the animal, blue (square) is the sports equipment.

(c) fastText. Orange is the animal, blue (square) is the sports equipment.

Frequentist experiment with Gaussian mixture models

Image credits: Bert: https://d827xgdhgqbnd.cloudfront.net/wp-content/uploads/2019/04/09110726/Bert-Head.png / Elmo: https://vignette.wikia.nocookie.net/muppet/images/5/5a/Elmo-elmo-elmo.jpg/revision/latest/scale-to-width-down/280?cb=20110917000614 / Graphic by author

For each of the top 1,000 words in the corpus, we extract 1,500 contexts⁵ and decompose them using PCA, t-SNE, and UMAP. Next, we fit Gaussian mixture models for each k in {2:30} using 5-fold cross-validation. For each word, the predicted k is the k from the model with the highest probability on the held-out data, averaged over the 5 folds. (In a smaller experiment not reported here, we also use AIC and BIC as criterion for selecting k, without largely the same results.)

(In another experiment, not reported here, we use the raw embedding representations — without any decomposition — as the input to our Gaussian mixture models. In that experiment, k is predicted to be 2 for nearly all words.)

Evaluation

Our “ground truth’’ is the number of meanings in WordNet and English Wiktionary.¹²

We show the frequentist results below with the “subtracted contexts’’ representation. The results are largely the same for the other methods we used to compute the embedding representations, with correlations mainly between -0.1 and 0.1.

Bayesian experiment with Gaussian mixture models

Our pipeline is the same as above, except that we fit a Bayesian Gaussian mixture model with a Dirichlet process prior. We fit 50 models with random starts and maximum k=30. For each word, we take the mean effective number of components over these 50 iterations.

Evaluation

The results from the Bayesian experiment are similar to the frequentist experiment, except that there were several decomposition settings (PCA with 200 dimensions, for example) that produced constant values; this results in NaN correlation with the ground truth.

NaN values in this table result from the model predicting the same value of k under some hyperparameter combinations.

Conclusion

Our initial experiments in visualizing word senses were promising: the visual separation of word senses into clusters suggests that contextualized representations contains the necessary information. In those examples, however, k is fixed. We find that automatically determining the correct k for each unique word is a difficult task.

It is worth noting Wiktionary and WordNet have a low correlation with each other: 0.377. While this is larger than the correlations with our estimated k, it raises the question of which of these datasets should serve as the “ground truth,’’ since no model can be highly correlated with both of them.

This point has been brought up in general by Adam Kilgarriff, who argues that the appropriate word senses may depend on the task or domain at hand.¹³ Thus, we should be careful about trying to generate a one-size-fits-all set of word senses, especially as the upper bound on performance may be low. However, non-geometric approaches — which we did not examine in our approach — could be a starting point for additional research.