Introduction
Word Sense Disambiguation (WSD) is an important problem in Natural Language Processing (NLP), both in its own right and as a stepping stone to other advanced tasks in the NLP pipeline, applications such as machine translation BIBREF0 and question answering BIBREF1 . WSD specifically deals with identifying the correct sense of a word, among a set of given candidate senses for that word, when it is presented in a brief narrative (surrounding text) which is generally referred to as context. Consider the ambiguous word `cold'. In the sentence “He started to give me a cold shoulder after that experiment”, the possible senses for cold can be cold temperature (S1), a cold sensation (S2), common cold (S3), or a negative emotional reaction (S4). Therefore, the ambiguous word cold is specified along with the sense set {S1, S2, S3, S4} and our goal is to identify the correct sense S4 (as the closest meaning) for this specific occurrence of cold after considering - the semantic and the syntactic information of - its context.
In this effort, we develop our supervised WSD model that leverages a Bidirectional Long Short-Term Memory (BLSTM) network. This network works with neural sense vectors (i.e. sense embeddings), which are learned during model training, and employs neural word vectors (i.e. word embeddings), which are learned through an unsupervised deep learning approach called GloVe (Global Vectors for word representation) BIBREF2 for the context words. By evaluating our one-model-fits-all WSD network over the public gold standard dataset of SensEval-3 BIBREF3 , we demonstrate that the accuracy of our model in terms of F-measure is comparable with the state-of-the-art WSD algorithms'.
We outline the organization of the rest of the paper as follows. In Section 2, we briefly explore earlier efforts in WSD and discuss recent approaches that incorporate deep neural networks and word embeddings. Our main model that employs BLSTM with the sense and word embeddings is detailed in Section 3. We then present our experiments and results in Section 4 supported by a discussion on how to avoid some drawbacks of the current model in order to achieve higher accuracies and demand less number of training data which is desirable. Finally, in Section 5, we conclude with some future research directions for the construction of sense embeddings as well as applications of such model in other domains such as biomedicine.
Background and Related Work
Generally, there are three categories of WSD algorithms: supervised, knowledge-based, and unsupervised. Supervised algorithms consist of automatically inducing classification models or rules from labeled examples BIBREF4 . Knowledge-based WSD approaches are dependent on manually created lexical resources such as WordNet BIBREF5 and the Unified Medical Language System (UMLS) BIBREF6 . Unsupervised algorithms may employ topic modeling-based methods to disambiguate when the senses are known ahead of time BIBREF7 . For a thorough survey of WSD algorithms refer to Navigli BIBREF8 .
Neural Embeddings for WSD
In the past few years, there has been an increasing interest in training neural word embeddings from large unlabeled corpora using neural networks BIBREF9 BIBREF10 . Word embeddings are typically represented as a dense real-valued low dimensional matrix INLINEFORM0 (i.e. a lookup table) of size INLINEFORM1 , where INLINEFORM2 is the predefined embedding dimension and INLINEFORM3 is the vocabulary size. Each column of the matrix is an embedding vector associated with a word in the vocabulary and each row of the matrix represents a latent feature. These vectors can subsequently be used to initialize the input layer of a neural network or some other NLP model. GloVe BIBREF2 is one of the existing unsupervised learning algorithms for obtaining these vector representations of the words in which training is performed on aggregated global word-word co-occurrence statistics from a corpus.
Besides word embeddings, recently, computation of sense embeddings has gained the attention of numerous studies as well. For example, Chen et al. BIBREF11 adapted neural word embeddings to compute different sense embeddings (of the same word) and showed competitive performance on the SemEval-2007 data BIBREF12 .
Bidirectional LSTM
Long Short-Term Memory (LSTM), introduced by Hochreiter and Schmidhuber (1997) BIBREF13 , is a gated recurrent neural network (RNN) architecture that has been designed to address the vanishing and exploding gradient problems of conventional RNNs. Unlike feedforward neural networks, RNNs have cyclic connections making them powerful for modeling sequences. A Bidirectional LSTM is made up of two reversed unidirectional LSTMs BIBREF14 . For WSD this means we are able to encode information of both preceding and succeeding words within context of an ambiguous word, which is necessary to correctly classify its sense.
One Single BLSTM network for WSD
Given a document and the position of a target word, our model computes a probability distribution over possible senses related to that word. The architecture of our model, depicted in Fig. FIGREF4 , consist of 6 layers which are a sigmoid layer (at the top), a fully-connected layer, a concatenation layer, a BLSTM layer, a cosine layer, and a sense and word embeddings layer (on the bottom).
In contrast to other supervised neural WSD networks in which generally a softmax layer - with a cross entropy or hinge loss - is parameterized by the context words and selects the corresponding weight matrix and bias vector for each ambiguous word's senses BIBREF15 BIBREF16 , our network shares parameters over all words' senses. While remaining computationally efficient, this structure aims to encode statistical information across different words enabling the network to select the true sense (or even a proper word) in a blank space within a context.
Due to the replacement of their softmax layers with a sigmoid layer in our network, we need to impose a modification in the input of the model. For this purpose, not only the contextual features are going to make the input of the network, but also, the sense for which we are interested to find out whether that given context makes sense or not (no pun intended) would be provided to the network. Next, the context words would be transferred to a sequence of word embeddings while the sense would be represented as a sense embedding (the shaded embeddings in Fig. FIGREF4 ). For a set of candidate senses (i.e. INLINEFORM0 ) for an ambiguous term, after computing cosine similarities of each sense embedding with the word embeddings of the context words, we expect the sequence result of similarities between the true sense and the surrounding context communicate a pattern-like information that can be encoded through our BLSTM network; for the incorrect senses this premise does not hold. Several WSD studies already incorporated the idea of sense-context cosine similarities in their models BIBREF17 BIBREF18 .
Model Definition
For one instance (or one document), the input of the network consists of a sense and a list of context words (left and right) which paired together form a list of context components. For the context D which encompasses the ambiguous term INLINEFORM0 , that takes the set of predefined candidate senses INLINEFORM1 , the input for the sense INLINEFORM2 for which we are interested in to find out whether the context is a proper match will be determined by Eq. ( EQREF6 ). Then, this input is copied (next) to INLINEFORM3 positions of the context to form the first pair of the context components. DISPLAYFORM0
Here, INLINEFORM0 is the one-hot representation of the sense corresponding to INLINEFORM1 . A one-hot representation is a vector with dimension INLINEFORM2 consisting of INLINEFORM3 zeros and a single one which index indicates the sense. The INLINEFORM4 size is equal to the number of all senses in the language (or the domain of interest). Eq. ( EQREF6 ) will have the effect of picking the column (i.e. sense embeddings) from INLINEFORM5 corresponding to that sense. The INLINEFORM6 (stored in the sense embeddings lookup table) is initialized randomly since no sense embedding is computed a priori.
Regarding the context words inputs that form the second pairs of context components, at position m in the same context D the input is determined by: DISPLAYFORM0
Here, INLINEFORM0 is the one-hot representation of the word corresponding to INLINEFORM1 . Similar to a sense one-hot representation ( INLINEFORM2 ), this one-hot representation is a vector with dimension INLINEFORM3 consisting of INLINEFORM4 zeros and a single one which index indicates the word in the context. The INLINEFORM5 size is equal to the number of words in the language (or the domain of interest). Eq. ( EQREF7 ) will choose the column (i.e. word embeddings) from INLINEFORM6 corresponding to that word. The INLINEFORM7 (stored in the word embeddings lookup table) can be initialized using pre-trained word embeddings; in this work, GloVe vectors are used.
On the other hand, the output of the network that is examining sense INLINEFORM0 is DISPLAYFORM0
where INLINEFORM0 and INLINEFORM1 are the weights and the bias of the classification layer (sigmoid), and INLINEFORM2 is the result of the merge layer (concatenation).
When we train the network, for an instance with the correct sense and the given context as inputs, INLINEFORM0 is set to be 1.0, and for incorrect senses they are set to be 0.0. During testing, however, among all the senses, the output of the network for a sense that gives the highest value of INLINEFORM1 will be considered as the true sense of the ambiguous term, in other words, the correct sense would be: DISPLAYFORM0
By applying softmax to the result of estimated classification values, INLINEFORM0 , we can show them as probabilities; this facilitates interpretation of the results.
Further, the hidden layer INLINEFORM0 is computed as DISPLAYFORM0
where INLINEFORM0 means rectified linear unit; INLINEFORM1 is the concatenated outputs of the right and left traversing LSTMs of the BLSTM when the last context components are met. INLINEFORM2 and INLINEFORM3 are the weights and bias for the hidden layer.
Validation for Selection of Hyper-parameters
SensEval-3 data BIBREF3 on which the network is evaluated, consist of separate training and test samples. In order to find hyper-parameters of the network 5% of the training samples were used for the validation in advance. Once the hyper-parameters are selected, the whole network is trained on all training samples prior to testing. As to the loss function employed for the network, even though is it common to use (binary) cross entropy loss function when the last unit is a sigmoidal classification, we observed that mean square error led to better results for the final argmax classification (Eq. ( EQREF9 )) that we used. Regarding parameter optimization, RMSprop BIBREF19 is employed. Also, all weights including embeddings are updated during training.
Dropout and Dropword
Dropout BIBREF20 is a regularization technique for neural network models where randomly selected neurons are ignored during training. This means that their contribution to the activation of downstream neurons is temporally removed on the forward pass, and any weight updates are not applied to the neuron on the backward pass. The effect is that the network becomes less sensitive to the specific weights of neurons, resulting in better generalization, and a network that is less likely to overfit the training data. In our network, dropout is applied to the embeddings as well as the outputs of the merge and fully-connected layers.
Following the dropout logic, dropword BIBREF21 is the word level generalizations of it, but in word dropout the word is set to zero while in dropword it is replaced with a specific tag. The tag is subsequently treated just like one word in the vocabulary. The motivation for doing dropword and word dropout is to decrease the dependency on individual words in the training context. Since by replacing word dropout with dropword we observed no change in the results, only word dropout was applied to the sequence of context words during training.
Experiments
In SensEval-3 data (lexical sample task), the sense inventory used for nouns and adjectives is WordNet 1.7.1 BIBREF5 whereas verbs are annotated with senses from Wordsmyth. Table TABREF15 presents the number of words under each part of speech, and the average number of senses for each class.
As stated, training and test data are supplied as the instances of this task; and the task consist of disambiguating one indicated word within a context.
Experimental Settings
The hyper-parameters that were determined during the validation is presented in Table TABREF17 . The preprocessing of the data was conducted by lower-casing all the words in the documents and removing numbers. This results in a vocabulary size of INLINEFORM0 = 29044. Words not present in the training set are considered unknown during testing. Also, in order to have fixed-size contexts around the ambiguous words, the padding and truncating are applied to them whenever needed.
Results
Between-all-models comparisons - When SensEval-3 task was launched 47 submissions (supervised and unsupervised algorithms) were received addressing this task. Afterward, some other papers tried to work on this data and reported their results in separate articles as well. We compare the result of our model with the top-performing and low-performing algorithms (supervised). We show our single model sits among the 5 top-performing algorithms, considering that in other algorithms for each ambiguous word one separate classifier is trained (i.e. in the same number of ambiguous words in a language there have to be classifiers; which means 57 classifiers for this specific task). Table TABREF19 shows the results of the top-performing and low-performing supervised algorithms.
The first two algorithms represent the state-of-the-art models of supervised WSD when evaluated on SensEval-3. Multi-classifier BLSTM BIBREF15 consists of deep neural networks which make use of pre-trained word embeddings. While the lower layers of these networks are shared, upper layers of each network are responsible to individually classify the ambiguous that word the network is associated with. IMS+adapted CW BIBREF16 is another WSD model that considers deep neural networks and also uses pre-trained word embeddings as inputs. In contrast to Multi-classifier BLSTM, this model relies on features such as POS tags, collocations, and surrounding words to achieve their result. For these two models, softmax constitutes the output layers of all networks. htsa3 BIBREF22 was the winner of the SensEval-3 lexical sample. It is a Naive Bayes system applied mainly to raw words, lemmas, and POS tags with correction of the a-priori frequencies. IRST-Kernels BIBREF23 utilizes kernel methods for pattern abstraction, paradigmatic and syntagmatic information and unsupervised term proximity on British National Corpus (BNC), in SVM classifiers. Likewise, nusels BIBREF24 makes use of SVM classifiers with a combination of knowledge sources (part-of-speech of neighboring words, words in context, local collocations, syntactic relations. The second part of the table lists the low-performing supervised algorithms BIBREF3 . Considering their ranking scores we see that there are unsupervised methods that outperform these supervised algorithms.
Within-our-model comparisons - Besides several internal experiments to examine the importance of some hyper-parameters to our network, we investigated if the sequential follow of cosine similarities computed between a true sense and its preceding and succeeding context words carries a pattern-like information that can be encoded with BLSTM. Table TABREF20 presents the results of these experiments.
The first row shows the best result of the network that we described above (and depicted in Fig. FIGREF4 ). Each of the other rows shows one change that we applied to the network to see the behavior of the network in terms of F-measure. In the middle part, we are specifically concerned about the importance of the presence of a BLSTM layer in our network. So, we introduced some fundamental changes in the input or in the structure of the network. Generally, it is expected that the cosine similarities of closer words (in the context) to the true sense be larger than the incorrect senses' BIBREF17 ; however, if a series of cosine similarities can be encoded through an LSTM (or BLSTM) network should be experimented. We observe if reverse the sequential follow of information into our Bidirectional LSTM, we shuffle the order of the context words, or even replace our Bidirectional LSTMs with two different fully-connected networks of the same size 50 (the size of the LSTMs outputs), the achieved results were notably less than 72.5%.
In the third section of the table, we report our changes to the hyper-parameters. Specifically, we see the importance of using GloVe as pre-trained word embeddings, how word dropout improves generalization, and how context size plays an important role in the final classification result (showing one of our experiments).
Discussion
From the results of Table TABREF19 , we notice our single WSD network, despite eliminating the problem of having a large number of WSD classifiers, still falls short when is compared with the state-of-the-art WSD algorithms. Based on our intuition and supported by some of our preliminary experiments, this deficiency stems from an important factor in our BLSTM network. Since no sense embedding is made publicly available for use, the sense embeddings are initialized randomly; yet, word embeddings are initialized by pre-trained GloVe vectors in order to benefit from the semantic and syntactic properties of the context words conveyed by these embeddings. That is to say, the separate spaces that the sense embeddings and the (context) word embeddings come from enforces some delay for the alignment of these spaces which in turn demands more training data. Furthermore, this early misalignment does not allow the BLSTM fully take advantage of larger context sizes which can be helpful. Our first attempt to deal with such problem was to pre-train the sense embeddings by some techniques - such as taking the average of the GloVe embeddings of the (informative) definition content words of senses, or taking the average of the GloVe embeddings of the (informative) context words in their training samples - did not give us a better result than our random initialization. Our preliminary experiments though in which we replaced all GloVe embeddings in the network with sense embeddings (using a method proposed by Chen et al. BIBREF11 ), showed considerable improvements in the results of some ambiguous words. That means both senses and context words (while they can be ambiguous by themselves) come from one vector space. In other words, the context would also be represented by the possible senses that its words can take. This idea not only can help to improve the results of the current model, it can also avoid the need for a large amount of training data since senses can be seen in both places, center and context, to be trained.
Conclusion
In contrast to common one-classifier-per-each-word supervised WSD algorithms, we developed our single network of BLSTM that is able to effectively exploit word orders and achieve comparable results with the best-performing supervised algorithms. This single WSD BLSTM network is language and domain independent and can be applied to resource-poor languages (or domains) as well. As an ongoing project, we also provided a direction which can lead us to the improvement of the results of the current network using pre-trained sense embeddings.
For future work, besides following the discussed direction in order to resolve the inadequacy of the network regarding having two non-overlapping vector spaces of the embeddings, we plan to examine the network on technical domains such as biomedicine as well. In this case, our model will be evaluated on MSH WSD dataset prepared by National Library of Medicine (NLM). Also, construction of sense embeddings using (extended) definitions of senses BIBREF25 BIBREF26 can be tested. Moreover, considering that for many senses we have at least one (lexically) unambiguous word representing that sense, we also aim to experiment with unsupervised (pre-)training of our network which benefits form quarry management by which more training data will be automatically collected from the web.

Question: How long is their dataset?
Answer: Unanswerable

Unanswerable

Unanswerable

Question: Do they use pretrained word embeddings?
Answer: No

Yes

Yes

Question: How many layers does their model have?
Answer: 6

6

6 layers

Question: What metrics do they use?
Answer: F-measure

F-measure

F-measure