EWISE: A New Approach to Word Sense Disambiguation
By: Kaushik Mahorker
Tags: NLP, Neural Networks, Machine Learning, Research

Natural language processing is a difficult task and the pure nature of complexity of the english language adds to this. Therefore, in this field we solve many subtasks to get closer to an interpretation that is truly natural. One such subtask is word sense disambiguation or WSD for short. This article explores this problem and delineates a new approach to solving it based on the recent outstanding paper in ACL 2019 by Kumar, Jat et al: Zero-shot Word Sense Disambiguation using Sense Definition Embeddings.

What is word sense disambiguation?

Words have different meanings and different intentions based on the context they are used in, even if they are the same words. These “difficult” words are more generally referred to as homonyms. Take the following example:

He was not feeling well after the marathon. The farmer grabbed the pail from the well.

Both sentences use the word “well,” but they clearly have different senses in each setting.

Word sense disambiguation (WSD) denotes the task of associating a word to its proper definition based on context clues.

RELATED WORK — What approaches exist to solve this problem?

WSD is a well explored problem and many approaches exist to tackle it.

The first are supervised and semi-supervised approaches that treat the definitions as discrete labels. One such supervised approach is context2vec that also uses Bidirectional LSTMs to learn generic context embedding functions. This and other supervised models fail to do well for words that are not included in the training dataset, so unseen and uncommon words cannot be deciphered.

The other type of approach is unsupervised knowledge based. One such unsupervised approach is Enhanced Lesk WSD, which uses WordNet to determine the number of shared words between the context of a word and each definition via a word similarity function. However, maximum overlap heuristics don’t perform as well as supervised approaches.

RELATED WORK — What relevant related approaches can we leverage to improve upon these existing solutions?

WordNet is a large dictionary database for the english language that not only contains various definitions for words, POS tags, and synonyms, but also semantic relations between different words in a format that machines can easily leverage and use.

To overcome the issue of not being able to get embeddings of rare and unseen words, we need to find a way to predict embeddings of these rare words with those of common words. The paper Learning to Compute Word Embeddings on the Fly helps us do exactly this with an LSTM network and access to dictionary data.

Using the above approaches we can create definition embeddings, but in WSD we still need to effectively connect the input sentences to the embedding outputs. We can leverage the work in Using the Output Embedding to Improve Language Models and use the update rules of word2vec forms to tie weights from input and output models together effectively.

EWISE: Extended WSD Incorporating Sense Embeddings

EWISE is the most current approach to tackling WSD and overcomes all aforementioned shortcomings. This approach learns a continuous space of sense embeddings as a target rather than discrete and thus enables generalized zero-shot learning. In other words, EWISE is able to identify definition mappings for both seen and unseen words.

Architecture

EWISE Architecture

The above architecture diagram has three main stages: attentive context encoding, definition encoding, and knowledge graph embedding.

Definition Encoding in EWISE takes a definition and leverages existing sentence representation models such as InferSent and USE to create a compressed representation of the definition. However, these baseline encodings are not sufficient to generalize to new words.

WordNet’s knowledge graph is used to train the encoder’s parameters. This graph contains nodes of senses and relations between these senses. The definitions are trained on sets of senses with up to 18 types of relations that connect them.

One of two approaches are used to calculate the scores between edges, TransE and ConvE, are delineated by the equations below:

Modified TransE equation to calculate dissimilarity between two sense embeddings in knowledge graph. q(*): sense embedding passed through definition encoder, h: head node knowledge graph, t: tail node of knowledge graph, e_l: link score to be learned

Modified ConvE equation to score the link between two sense embeddings in the knowledge graph. q(*): sense embedding passed through definition encoder, h: head node, t: tail node, e_l: link score to be learned, e_t: tail score to be learned, w: 2D convolution filters, W: linear transformation (i.e. fully connected layer), f: rectified linear unit (ReLU), vec: vectorized version, x_bar: 2D reshaping

Now that different senses are connected and their edges scored, these can be extended to create a larger and essentially continuous space of sense embeddings.

Notice how the above two stages have yet to involve the actual input sentences. This is what makes this approach so powerful. The sense embeddings are trained completely independently from the sentence inputs, so even arbitrary inputs will be able to be classified.

Attentive context encoding is where the input comes into play. This encoder generates context dependent representation for the input by leveraging a bidirectional LSTM. The words are represented as a concatenation of the forward and backward versions of the sentence. For example:

Input sentence: he wore a tie Backward: tie a wore he BiLSTM representation: [he, tie] ; [wore,a] ; [a,wore] ; [tie, he]

This representation is followed by a self-attention layer, detailed more in Vaswani et al (2017), and a fully connected layer to project into the space of sense embeddings.

During training, this representation is multiplied by each sense embedding of in the inventory and the one with the highest score is chosen as the label. Because this is a supervised setting, we can use average cross entropy loss to tune the model to pick correct mapping.

Softmax layer following fully connected layer to calculate sense embedding probabilities. v_i: sense embedding of a given input word; b: bias; S: sense inventory

Cross entropy loss function minimized over all annotated words in a batch. z_ij: one hot representation of the target (i.e. correct) sense in inventory S

Results

The model is evaluated on how it compares to other state of the art methods as well as its power to generalize to rare and unseen words.

Comparison of overall and part of speech wise F1 scores of EWISE across different test datasets with other state of the art models

EWISE beats all other models across all categories and across all part of speech tags except for adverbs where its F1 score falls just short of GAS.

Comparison of F1 scores of different models based on frequency of annotated words in the train set

The left most category with frequency=0 represents the zero-shot learning capability of EWISE. In other words, for words it has never seen before it can get an F1 score of 91, significantly higher than its counterparts. It also performs the best in all frequency buckets as well for rare words. This proves the model’s capacity to classify the word sense of unseen words.

Conclusion

In summary, EWISE is a new approach to the WSD problem that leverages independent training of word sense embeddings and inputs to extend its usability to unseen and rare words. EWISE consistently outperforms all other state of the art WSD models and proves that the model has zero-shot learning capabilities.