Venue: EMNLP 2017

# Neural Sequence Learning Models for

Word Sense Disambiguation

Alessandro Raganato, Claudio Delli Bovi and Roberto Navigli

Department of Computer Science

Sapienza University of Rome

{raganato,dellibovi,navigli}@di.uniroma1.it

###### Abstract

Word Sense Disambiguation models exist in many flavors. Even though supervised ones tend to perform best in terms of accuracy, they often lose ground to more flexible knowledge-based solutions, which do not require training by a word expert for every disambiguation target. To bridge this gap we adopt a different perspective and rely on sequence learning to frame the disambiguation problem: we propose and study in depth a series of end-to-end neural architectures directly tailored to the task, from bidirectional Long Short-Term Memory to encoder-decoder models. Our extensive evaluation over standard benchmarks and in multiple languages shows that sequence learning enables more versatile all-words models that consistently lead to state-of-the-art results, even against word experts with engineered features.

## 1 Introduction

As one of the long-standing challenges in Natural Language Processing (NLP), Word Sense Disambiguation [20] has received considerable attention over recent years. Indeed, by dealing with lexical ambiguity an effective WSD model brings numerous benefits to a variety of downstream tasks and applications, from Information Retrieval and Extraction [35, 47] to Machine Translation [6, 8]. Recently, WSD has also been leveraged to build continuous vector representations for word senses [9, 10, 11].

Inasmuch as WSD is described as the task of associating words in context with the most suitable entries in a pre-defined sense inventory, the majority of WSD approaches to date can be grouped into two main categories: supervised (or semi-supervised) and knowledge-based. Supervised models have been shown to consistently outperform knowledge-based ones in all standard benchmarks [10], at the expense, however, of harder training and limited flexibility. First of all, obtaining reliable sense-annotated corpora is highly expensive and especially difficult when non-expert annotators are involved [11], and as a consequence approaches based on unlabeled data and semi-supervised learning are emerging [12, 13, 14, 15].

Apart from the shortage of training data, a crucial limitation of current supervised approaches is that a dedicated classifier (_word expert_) needs to be trained for every target lemma, making them less flexible and hampering their use within end-to-end applications. In contrast, knowledge-based systems do not require sense-annotated data and often draw upon the structural properties of lexicosemantic resources [1, 12, 13]. Such systems construct a model based only on the underlying resource, which is then able to handle multiple target words at the same time and disambiguate them jointly, whereas word experts are forced to treat each disambiguation target in isolation.

In this paper our focus is on supervised WSD, but we depart from previous approaches and adopt a different perspective on the task: instead of framing a separate classification problem for each given word, we aim at modeling the joint disambiguation of the target text as a whole in terms of a sequence labeling problem. From this standpoint, WSD amounts to translating a sequence of words into a sequence of potentially sense-tagged tokens.



With this in mind, we design, analyze and compare experimentally various neural architectures of different complexities, ranging from a single bidirectional Long Short-Term Memory (Graves and Schmidhuber, 2005, LSTM) to a sequence-to-sequence approach Sutskever et al. (2014). Each architecture reflects a particular way of modeling the disambiguation problem, but they all share some key features that set them apart from previous supervised approaches to WSD: they are trained end-to-end from sense-annotated text to sense labels, and learn a single all-words model from the training data, without fine tuning or explicit engineering of local features.

The contributions of this paper are twofold. First, we show that neural sequence learning represents a novel and effective alternative to the traditional way of modeling supervised WSD, enabling a single all-words model to compete with a pool of word experts and achieve state-of-the-art results, while also being easier to train, arguably more versatile to use within downstream applications, and directly adaptable to different languages without requiring additional sense-annotated data (as we show in Section 6.2); second, we carry out an extensive experimental evaluation where we compare various neural architectures designed for the task (and somehow left underinvestigated in previous literature), exploring different configurations and training procedures, and analyzing their strengths and weaknesses on all the standard benchmarks for all-words WSD.

## 2 Related Work

The literature on WSD is broad and comprehensive (Agirre and Edmonds, 2007; Navigli, 2009): new models are continuously being developed Yuan et al. (2016); Tripodi and Pelillo (2017); Butnaru et al. (2017) and tested over a wide variety of standard benchmarks Edmonds and Cotton (2001); Snyder and Palmer (2004); Pradhan et al. (2007); Navigli et al. (2007, 2013); Moro and Navigli (2015). Moreover, the field has been explored in depth from different angles by means of extensive empirical studies and evaluation frameworks Pilehvar and Navigli (2014); Iacobacci et al. (2016); McCarthy et al. (2016); Raganato et al. (2017).

As regards supervised WSD, traditional approaches are generally based on extracting local features from the words surrounding the target, and then training a classifier Zhong and Ng (2010); Shen et al. (2013) for each target lemma. In their latest developments, these models include more complex features based on word embeddings Taghipour and Ng (2015); Rothe and Schutze (2015); Iacobacci et al. (2016).

The recent upsurge of neural networks has also contributed to fueling WSD research: Yuan et al. (2016) rely on a powerful neural language model to obtain a latent representation for the whole sentence containing a target word \(w\); their instance-based system then compares that representation with those of example sentences annotated with the candidate meanings of \(w\). Similarly, Context2Vec Melamud et al. (2016) makes use of a bidirectional LSTM architecture trained on an unlabeled corpus and learns a context vector for each sense annotation in the training data. Finally, Kageback and Salomonsson (2016) present a supervised classifier based on bidirectional LSTM for the lexical sample task Kilgarriff (2001); Mihalcea et al. (2004). All these contributions have shown that supervised neural models can achieve state-of-the-art performances without taking advantage of external resources or language-specific features. However, they all consider each target word as a separate classification problem and, to the best of our knowledge, very few attempts have been made to disambiguate a text jointly using sequence learning Ciaramita and Altun (2006).

Sequence learning, especially using LSTM Hochreiter and Schmidhuber (1997); Graves and Schmidhuber (2005); Graves (2013), has become a well-established standard in numerous NLP tasks Zhou and Xu (2015); Ma and Hovy (2016); Wang and Chang (2016). In particular, sequence-to-sequence models Sutskever et al. (2014) have grown increasingly popular and are used extensively in, e.g., Machine Translation Cho et al. (2014); Bahdanau et al. (2015), Sentence Representation Kiros et al. (2015), Syntactic Parsing Vinyals et al. (2015), Conversation Modeling Vinyals and Le (2015), Morphological Inflection Faruqui et al. (2016) and Text Summarization Gu et al. (2016). In line with this trend, we focus on the (so far unexplored) context of supervised WSD, and investigate state-of-the-art all-words approaches that are based on neural sequence learning and capable of disambiguating all target content words within an input text, a key feature in several knowledge-based approaches 



## 3 Sequence Learning for Word Sense Disambiguation

In this section we define WSD in terms of a sequence learning problem. While in its classical formulation Navigli2009word WSD is viewed as a classification problem for a given word \(w\) in context, with word senses of \(w\) being the class labels, here we consider a variable-length sequence of input symbols \(\vec{x}=\langle x_{1},...,x_{T}\rangle\) and we aim at predicting a sequence of output symbols \(\vec{y}=\langle y_{1},...,y_{T^{\prime}}\rangle\).1 Input symbols are word tokens drawn from a given vocabulary \(V\).2 Output symbols are either drawn from a pre-defined sense inventory \(S\) (if the corresponding input symbols are open-class content words, i.e., nouns, verbs, adjectives or adverbs), or from the same input vocabulary \(V\) (e.g., if the corresponding input symbols are function words, like prepositions or determiners). Hence, we can define a WSD model in terms of a function that maps sequences of symbols \(x_{i}\in V\) into sequences of symbols \(y_{j}\in O=S\cup V\).

Footnote 1: In general \(\vec{x}\) and \(\vec{y}\) might have different lengths, e.g., if \(\vec{x}\) contains a multi-word expression (_European Union_) which is mapped to a unique sense identifier (European Union\({}_{n}^{1}\)).

Here all-words WSD is no longer broken down into a series of distinct and separate classification tasks (one per target word) but rather treated directly at the sequence level, with a single model handling all disambiguation decisions. In what follows, we describe three different models for accomplishing this: a traditional LSTM-based model (Section 3.1), a variant that incorporates an attention mechanism (Section 3.2), and an encoder-decoder architecture (Section 3.3).

### Bidirectional LSTM Tagger

The most straightforward way of modeling WSD as formulated in Section 3 is that of considering a sequence labeling architecture that tags each symbol \(x_{i}\in V\) in the input sequence with a label \(y_{j}\in O\). Even though the formulation is rather general, previous contributions Melamud et al. (2016); Kageback and Salomonsson (2016) have already shown the effectiveness of recurrent neural networks for WSD. We follow the same line and employ a bidirectional LSTM architecture: in fact, important clues for disambiguating a target word could be located anywhere in the context (not necessarily before the target) and for a model to be effective it is crucial that it exploits information from the whole input sequence at every time step.

**Architecture.** A sketch of our bidirectional LSTM tagger is shown in Figure 1. It consists of:

* An embedding layer that converts each word \(x_{i}\in\vec{x}\) into a real-valued \(d\)-dimensional vector \(\mathbf{x}_{i}\) via the embedding matrix \(\mathbf{W}\in\mathbb{R}^{d\times|V|}\);
* One or more stacked layers of bidirectional LSTM Graves and Schmidhuber (2005). The hidden state vectors \(\mathbf{h}_{i}\) and output vectors \(\mathbf{o}_{i}\) at the \(i^{th}\) time step are then obtained as the concatenations of the forward and backward pass vectors \(\overrightarrow{\mathbf{h}}_{i},\overrightarrow{\mathbf{o}}_{i}\) and \(\overleftarrow{\mathbf{h}}_{i},\overleftarrow{\mathbf{o}}_{i}\);
* A fully-connected layer with softmax activation that turns the output vector \(\mathbf{o}_{i}\) at the \(i^{th}\) time step into a probability distribution over the output vocabulary \(O\).

**Training.** The tagger is trained on a dataset of \(N\) labeled sequences \(\{(\vec{x}_{k},\vec{y}_{k})\}_{k=1}^{N}\) directly obtained from the sentences of a sense-annotated corpus, where each \(\vec{x}_{k}\) is a sequence of word tokens, and each \(\vec{y}_{k}\) is a sequence containing both word tokens and sense labels. Ideally \(\vec{y}_{k}\) is a copy of \(\vec{x}_{k}\) where each content word is sense-tagged. This is, however, not the case in many real-world datasets, where only a subset of the content words is annotated; hence the architecture is designed to deal with both fully and partially annotated sentences. Apart from sentence splitting and tokenization, no preprocessing is required on the training data.

Figure 1: Bidirectional LSTM sequence labeling architecture for WSD (2 hidden layers). We use the notation of Navigli2009word for word senses: \(w_{p}^{i}\) is the \(i\)-th sense of \(w\) with part of speech \(p\).





### Attentive Bidirectional LSTM Tagger

The bidirectional LSTM tagger of Section 3.1 exploits information from the whole input sequence \(\vec{x}\), which is encoded in the hidden state \(\mathbf{h}_{i}\). However, certain elements of \(\vec{x}\) might be more discriminative than others in predicting the output label at a given time step (e.g., the syntactic subject and object when predicting the sense label of a verb).

We model this hunch by introducing an attention mechanism, already proven to be effective in other NLP tasks Bahdanau et al. (2015); Vinyals et al. (2015), into the sequence labeling architecture of Section 3.1. The resulting _attentive_ bidirectional LSTM tagger augments the original architecture with an attention layer, where a context vector \(\mathbf{c}\) is computed from all the hidden states \(\mathbf{h}_{1},...,\mathbf{h}_{T}\) of the bidirectional LSTM. The attentive tagger first reads the entire input sequence \(\vec{x}\) to construct \(\mathbf{c}\), and then exploits \(\mathbf{c}\) to predict the output label \(y_{j}\) at each time step, by concatenating it with the output vector \(\mathbf{o}_{j}\) of the bidirectional LSTM (Figure 2).

We follow previous work Vinyals et al. (2015); Zhou et al. (2016) and compute \(\mathbf{c}\) as the weighted sum of the hidden state vectors \(\mathbf{h}_{1},...,\mathbf{h}_{T}\). Formally, let \(H\in\mathbb{R}^{n\times T}\) be the matrix of hidden state vectors \([\:\mathbf{h}_{1},...,\mathbf{h}_{T}\:]\), where \(n\) is the hidden state dimension and \(T\) is the input sequence length (cf. Section 3). \(\mathbf{c}\) is obtained as follows:

\[\textbf{u}=\omega^{T}\tanh(H)\] \[\textbf{a}=softmax(\textbf{u})\] \[\textbf{c}=H\textbf{a}^{T} \tag{1}\]

where \(\omega\in\mathbb{R}^{n}\) is a parameter vector, and \(\textbf{a}\in\mathbb{R}^{T}\) is the vector of normalized attention weights.

### Sequence-to-Sequence Model

The attentive tagger of Section 3.2 performs a two-pass procedure by first reading the input sequence \(\vec{x}\) to construct the context vector \(\mathbf{c}\), and then predicting an output label \(y_{j}\) for each element in \(\vec{x}\). In this respect, the attentive architecture can effectively be viewed as an encoder for \(\vec{x}\). A further generalization of this model would then be a complete encoder-decoder architecture Sutskever et al. (2014) where WSD is treated as a sequence-to-sequence mapping (_sequence-to-sequence WSD_), i.e., as the "translation" of word sequences into sequences of potentially sense-tagged tokens.

In the sequence-to-sequence framework, a variable-length sequence of input symbols \(\vec{x}\) is represented as a sequence of vectors \(\vec{\mathbf{x}}=\langle\mathbf{x}_{1},...,\mathbf{x}_{T}\rangle\) by converting each symbol \(x_{i}\in\vec{x}\) into a real-valued vector \(\mathbf{x}_{i}\) via an embedding layer, and then fed to an encoder, which generates a fixed-dimensional vector representation of the sequence. Traditionally, the encoder function is a Recurrent Neural Network (RNN) such that:

\[\textbf{h}_{t}=f(\textbf{h}_{t-1},\textbf{x}_{t})\] \[\textbf{c}=q(\{\textbf{h}_{1},...,\textbf{h}_{T}\}) \tag{2}\]

where \(\textbf{h}_{t}\in\mathbb{R}^{n}\) is the \(n\)-dimensional hidden state vector at time \(t\), \(\textbf{c}\in\mathbb{R}^{n}\) is a vector generated from the whole sequence of input states, and \(f\) and \(q\) are non-linear functions.3 A decoder is then trained to predict the next output symbol \(y_{t}\) given the encoded input vector \(\mathbf{c}\) and all the previously predicted output symbols \(\langle y_{1},...,y_{t-1}\rangle\). More formally, the decoder defines a probability over the output sequence \(\vec{y}=\langle y_{1},...,y_{T^{\prime}}\rangle\) by decomposing the joint probability into ordered conditionals:

Footnote 3: For instance, Sutskever et al. (2014) used an LSTM as \(f\), and \(q(\{\textbf{h}_{1},...,\textbf{h}_{T}\})=\textbf{h}_{T}\).

\[p(\vec{y}\:|\:\vec{x})=\prod_{t=1}^{T^{\prime}}p(y_{t}\:|\:\textbf{c},\langle y _{1},...,y_{t-1}\rangle) \tag{3}\]

Typically a decoder RNN defines the hidden state at time \(t\) as \(\textbf{s}_{t}=g(\textbf{s}_{t-1},\{\textbf{c},\:y_{t-1}\})\) and then feeds \(\textbf{s}_{t}\) to a softmax layer in order to obtain a conditional probability over output symbols.

Figure 2: Attentive bidirectional LSTM sequence labeling architecture for WSD (2 hidden layers).



In the context of WSD framed as a sequence learning problem, a sequence-to-sequence model takes as input a training set of labeled sequences (cf. Section3.1) and learns to replicate an input sequence while replacing each content word with its most suitable word sense from . In other words, sequence-to-sequence WSD can be viewed as the combination of two sub-tasks:

* A _memorization_ task, where the model learns to replicate the input sequence token by token at decoding time;
* The actual _disambiguation_ task where the model learns to replace content words across the input sequence with their most suitable senses from the sense inventory .

In the latter stage, multi-word expressions (such as nominal entity mentions or phrasal verbs) are replaced by their sense identifiers, hence yielding an output sequence that might have a different length than .

**Architecture.** The encoder-decoder architecture generalizes over both the models in Sections3.1 and 3.2. In particular, we include one or more bidirectional LSTM layers at the core of both the encoder and the decoder modules. The encoder utilizes an embedding layer (cf. Section3.1) to convert input symbols into embedded representations, feeds it to the bidirectional LSTM layer, and then constructs the context vector , either by simply letting (i.e., the hidden state of the bidirectional LSTM layer after reading the whole input sequence), or by computing the weighted sum described in Section3.2 (if an attention mechanism is employed). In either case, the context vector is passed over to the decoder, which generates the output symbols sequentially based on and the current hidden state , using one or more bidirectional LSTM layers as in the encoder module. Instead of feeding to the decoder only at the first time step [45, 70], we condition each output symbol  on , allowing the decoder to peek into the input at every step, as in Cho et al. cho2014learning. Finally, a fully-connected layer with softmax activation converts the current output vector of the last LSTM layer into a probability distribution over the output vocabulary . The complete encoder-decoder architecture (including the attention mechanism) is shown in Figure3.

## 4 Multitask Learning with Multiple Auxiliary Losses

Several recent contributions [69, 8, 10, 11] have shown the effectiveness of _multitask learning_[12], MTL) in a sequence learning scenario. In MTL the idea is that of improving generalization performance by leveraging training signals contained in related tasks, in order to exploit their commonalities and differences. MTL is typically carried out by training a single architecture using multiple loss functions and a shared representation, with the underlying intention of improving a main task by incorporating joint learning of one or more related auxiliary tasks. From a practical point of view, MTL works by including one task-specific output layer per additional task, usually at the outermost level of the architecture, while keeping the remaining hidden layers common across all tasks.

In line with previous approaches, and guided by the intuition that WSD is strongly linked to other NLP tasks at various levels, we also design and study experimentally a multitask augmentation of

Figure 3: Encoder-decoder architecture for sequence-to-sequence WSD, with 2 bidirectional LSTM layers and an attention layer.

 

the models described in Section 3. In particular, we consider two auxiliary tasks:

* **Part-of-speech (POS) tagging**, a standard auxiliary task extensively studied in previous work Sogaard and Goldberg (2016); Plank et al. (2016). Predicting the part-of-speech tag for a given token can also be informative for word senses, and help in dealing with cross-POS lexical ambiguities (e.g., _book a flight_ vs. _reading a good book_);
* **Coarse-grained semantic labels (LEX)** based on the WordNet Miller et al. (1990) lexicographer files,4 i.e., 45 coarse-grained semantic categories manually associated with all the synsets in WordNet on the basis of both syntactic and logical groupings (e.g., _noun.location_, or _verb.motion_). These very coarse semantic labels, recently employed in a multitask setting by Alonso and Plank (2017), group together related senses and help the model to generalize, especially over senses less covered at training time. Footnote 4: [https://wordnet.princeton.edu/man/lexnames.5WN.html](https://wordnet.princeton.edu/man/lexnames.5WN.html) We follow previous work Plank et al. (2016); Alonso and Plank (2017) and define an auxiliary loss function for each additional task. The overall loss is then computed by summing the main loss (i.e., the one associated with word sense labels) and all the auxiliary losses taken into account. As regards the architecture, we consider both the models described in Sections 3.2 and 3.3 modify them by adding two softmax layers in addition to the one in the original architecture. Figure 4 illustrates this for the attentive tagger of Section 3.2, considering both POS and LEX as auxiliary tasks. At the \(j^{th}\) time step the model predicts a sense label \(y_{j}\) together with a part-of-speech tag \(\text{POS}_{j}\) and a coarse semantic label \(\text{LEX}_{j}\).5 Footnote 5: We use a dummy LEX label (other) for punctuation and function words.

## 5 Experimental Setup

In this section we detail the setup of our experimental evaluation. We first describe the training corpus and all the standard benchmarks for all-words WSD; we then report technical details on the architecture and on the training process for all the models described throughout Section 3 and their multitask augmentations (Section 4).

Evaluation Benchmarks.We evaluated our models on the English all-words WSD task, considering both the fine-grained and coarse-grained benchmarks (Section 6.1). As regards fine-grained WSD, we relied on the evaluation framework of Raganato et al. (2017), which includes five standardized test sets from the Senseval/SemEval series: Senseval-2 Edmonds and Cotton (2001, **SE2**), Senseval-3 Snyder and Palmer (2004, **SE3**), SemEval-2007 Pradhan et al. (2007, **SE07**), SemEval-2013 Navigli et al. (2013, **SE13**) and SemEval-2015 Moro and Navigli (2015, **SE15**). Due to the lack of a reasonably large development set for our setup, we considered the smallest among these test sets, i.e., **SE07**, as development set and excluded it from the evaluation of Section 6.1. As for coarse-grained WSD, we used the SemEval-2007 task 7 test set Navigli et al. (2007), which is not included in the standardized framework, and mapped the original sense inventory from WordNet 2.1 to WordNet 3.0.6 Finally, we carried out an experiment on multilingual WSD using the Italian, German, French and Spanish data of **SE13**. For these benchmarks we relied on BabelNet Navigli and Ponzetto (2012)7 as unified sense inventory.

Footnote 6: We utilized the original sense-key mappings available at [http://wordnetcode.princeton.edu/3.0](http://wordnetcode.princeton.edu/3.0) for nouns and verbs, and the automatic mappings by Daude et al. (2003) for the remaining parts of speech (not available in the original mappings).

Figure 4: Multitask augmentation (with both POS and LEX as auxiliary tasks) for the attentive bidirectional LSTM tagger of Section 3.2.

 

At testing time, given a target word \(w\), our models used the probability distribution over \(O\), computed by the softmax layer at the corresponding time step, to rank the candidate senses of \(w\); we then simply selected the top ranking candidate as output of the model.

**Architecture Details.** To set a level playing field with comparison systems on English all-words WSD, we followed Raganato et al. (2017) and, for all our models, we used a layer of word embeddings pre-trained8 on the English ukWaC corpus Baroni et al. (2009) as initialization, and kept them fixed during the training process. For all architectures we then employed 2 layers of bidirectional LSTM with 2048 hidden units (1024 units per direction).

Footnote 8: We followed Iacobacci et al. (2016) and used the Word2Vec Mikolov et al. (2013) skip-gram model with 400 dimensions, 10 negative samples and a window size of 10.

As regards multilingual all-words WSD (Section 6.2), we experimented, instead, with two different configurations of the embedding layer: the pre-trained bilingual embeddings by Mrksic et al. (2017) for all the language pairs of interest (EN-IT, EN-FR, EN-DE, and EN-ES), and the pre-trained multilingual 512-dimensional embeddings for 12 languages by Ammar et al. (2016).

**Training.** We used SemCor 3.0 Miller et al. (1993) as training corpus for all our experiments. Widely known and utilized in the WSD literature, SemCor is one of the largest corpora annotated manually with word senses from the sense inventory of WordNet Miller et al. (1990) for all open-class parts of speech. We used the standardized version of SemCor as provided in the evaluation framework9 which also includes coarse-grained POS tags from the universal tagset. All models were trained for a fixed number of epochs \(E=40\) using Adadelta Zeiler (2012) with learning rate 1.0 and batch size 32. After each epoch we evaluated our models on the development set, and then compared the best iterations (\(E^{*}\)) on the development set with the reported state of the art in each benchmark.

Footnote 9: [http://lcl.uniroma1.it/wsdeval](http://lcl.uniroma1.it/wsdeval)

## 6 Experimental Results

Throughout this section we identify the models based on the LSTM tagger (Sections 3.1-3.2) by the label **BLSTM**, and the sequence-to-sequence models (Section 3.3) by the label **Seq2Seq**.

### English All-words WSD

Table 1 shows the performance of our models on the standardized benchmarks for all-words fine-grained WSD. We report the F1-score on each in

\begin{table}
\begin{tabular}{|p{113.8pt}||c|c|c|c|c|c|c|c|c|} \cline{2-10} \multicolumn{1}{c|}{} & \multicolumn{3}{c|}{**Dev**} & \multicolumn{3}{c||}{**Test Datasets**} & \multicolumn{3}{c|}{**Concatenation of All Test Datasets**} \\ \hline  & **SE07** & **SE2** & **SE3** & **SE13** & **SE15** & **Nouns** & Verbs & Adj. & Adv. & **All** \\ \hline BLSTM & 61.8 & 71.4 & 68.8 & 65.6 & 69.2 & 70.2 & 56.3 & 75.2 & **84.4** & 68.9 \\ BLSTM + attt. & 62.4 & 71.4 & **70.2** & 66.4 & 70.8 & 71.0 & **58.4** & 75.2 & 83.5 & 69.7 \\ BLSTM + attt. + Lex & 63.7 & **72.0** & 69.4 & 66.4 & **72.4** & **71.6** & 57.1 & **75.6** & 83.2 & **69.9** \\ BLSTM + attt. + Lex + POS & **64.8** & **72.0** & 69.1 & **66.9** & 71.5 & 71.5 & 57.5 & 75.0 & 83.8 & **69.9** \\ \hline Seq2Seq & 60.9 & 68.5 & 67.9 & 65.3 & 67.0 & 68.7 & 54.5 & 74.0 & 81.2 & 67.3 \\ Seq2Seq + attt. & 62.9 & 69.9 & 69.6 & 65.6 & 67.7 & 69.5 & 57.2 & 74.5 & 81.8 & 68.4 \\ Seq2Seq + attt. + Lex & 64.6 & 70.6 & 67.8 & 66.5 & 68.7 & 70.4 & 55.7 & 73.3 & 82.9 & 68.5 \\ Seq2Seq + attt. + Lex + POS & 63.1 & 70.1 & 68.5 & 66.5 & 69.2 & 70.1 & 55.2 & 75.1 & 84.4 & 68.6 \\ \hline \hline IMS & 61.3 & 70.9 & 69.3 & 65.3 & 69.5 & 70.5 & 55.8 & 75.6 & 82.9 & 68.9 \\ IMS\(+\)emb & **62.6** & **72.2** & **70.4** & 65.9 & 71.5 & **71.9** & 56.6 & **75.9** & **84.7** & **70.1** \\ Context2Vec & 61.3 & 71.8 & 69.1 & 65.6 & **71.9** & 71.2 & **57.4** & 75.2 & 82.7 & 69.6 \\ \hline Leske\({}_{ext}+\)emb & \(\star\)56.7 & 63.0 & 63.7 & 66.2 & 64.6 & 70.0 & 51.1 & 51.7 & 80.6 & 64.2 \\ UKB\({}_{loss}\) w2w & 42.9 & 63.5 & 55.4 & \(\star\)62.9 & 63.3 & 64.9 & 41.4 & 69.5 & 69.7 & 61.1 \\ Babelfy & 51.6 & \(\star\)67.0 & 63.5 & **66.4** & 70.3 & 68.9 & 50.7 & 73.2 & 79.8 & 66.4 \\ \hline MFS & 54.5 & 65.6 & \(\star\)66.0 & 63.8 & \(\star\)67.1 & 67.7 & 49.8 & 73.1 & 80.5 & 65.5 \\ \hline \end{tabular}
\end{table}
Table 1: F-scores (%) for English all-words fine-grained WSD on the test sets in the framework of Raganato et al. (2017) (including the development set **SE07**). The first system with a statistically significant difference from our best models is marked with \(\star\) (unpaired \(t\)-test, \(p<0.05\)).

 

dividual test set, as well as the F1-score obtained on the concatenation of all four test sets, divided by part-of-speech tag.

We compared against the best supervised and knowledge-based systems evaluated on the same framework. As supervised systems, we considered **Context2Vec**Melamud et al. (2016) and It Makes Sense Zhong and Ng (2010), **IMS**), both the original implementation and the best configuration reported by Iacobacci et al. (2016, **IMS\(+\)emb**), which also integrates word embeddings using exponential decay.10 All these supervised systems were trained on the standardized version of SemCor. As knowledge-based systems we considered the embeddings-enhanced version of Lesk by Basile et al. (2014, **Lesk\({}_{ext}+\)emb**), UKB Agirre et al. (2014) (**UKB\({}_{gloss}\) w2w**) , and **Babelfly**Moro et al. (2014). All these systems relied on the Most Frequent Sense (**MFS**) baseline as back-off strategy.11 Overall, both **BLSTM** and **Seq2Seq** achieved results that are either state-of-the-art or statistically equivalent (unpaired \(t\)-test, \(p<0.05\)) to the best supervised system in each benchmark, performing on par with word experts tuned over explicitly engineered features Iacobacci et al. (2016). Interestingly enough, **BLSTM** models tended consistently to outperform their **Seq2Seq** counterparts, suggesting that an encoder-decoder architecture, despite being more powerful, might be suboptimal for WSD. Furthermore, introducing LEX (cf. Section 4) as auxiliary task was generally helpful; on the other hand, POS did not seem to help, corroborating previous findings Alonso and Plank (2017); Bingel and Sogaard (2017).

Footnote 10: We are not including Yuan et al. (2016), as their models are not available and not replicable on the standardized test sets, being based on proprietary data.

The overall performance by part of speech was consistent with the above analysis, showing that our models outperformed all knowledge-based systems, while obtaining results that are superior or equivalent to the best supervised models. It is worth noting that RNN-based architectures outperformed classical supervised approaches Zhong and Ng (2010); Iacobacci et al. (2016) when dealing with verbs, which are shown to be highly ambiguous Raganato et al. (2017).

The performance on coarse-grained WSD followed the same trend (Table 2). Both **BLSTM** and **Seq2Seq** outperformed UKB Agirre et al. (2014) and IMS trained on SemCor Taghipour and Ng (2015a), as well as recent supervised approaches based on distributional semantics and neural architectures Chen et al. (2014); Yuan et al. (2016).

### Multilingual All-words WSD

All the neural architectures described in this paper can be readily adapted to work with different languages without adding sense-annotated data in the target language. In fact, as long as the first layer (cf. Figures 1-3) is equipped with _bilingual_ or _multilingual_ embeddings where word vectors in the training and target language are defined in the same space, the training process can be left unchanged, even if based only on English data. The underlying assumption is that words that are translations of each other (e.g., _house_ in English and _casa_ in Italian) are mapped to word embeddings that are as close as possible in the vector space.

In order to assess this, we considered one of our best models (**BLSTM\(+\)att.\(+\)LEX**) and replaced the monolingual embeddings with bilingual and multilingual embeddings (as specified in Section 5), leaving the rest of the architecture unchanged. We then trained these architectures on the same English training data, and ran the resulting models on the multilingual benchmarks of SemEval-2013 for Italian, French, German and Spanish. While doing this, we exploited BabelNet's inter-resource mappings to convert WordNet sense labels (used at training time) into BabelNet synsets compliant with the sense inventory of the task.

F-score figures (Table 3) show that bilingual and multilingual models, despite being trained only on English data, consistently outperformed the MFS

\begin{table}
\begin{tabular}{|l|r|l|l|} \hline \multicolumn{4}{|c|}{**SemEval-2007 task 7**} \\ \hline BLSTM \(+\) att. \(+\) Lex & 83.0 & MS & 81.9 \\ BLSTM \(+\) att. \(+\) Lex + pos & **83.1** & Chen et al. (2014) & 82.6 \\ Seq2Seq \(+\) att. \(+\) Lex & 82.3 & Yuan et al. (2016) & **82.8** \\ Seq2Seq \(+\) att. \(+\) Lex + pos & 81.6 & UKB w2w & 80.1 \\ \hline \end{tabular}
\end{table}
Table 2: F-scores (%) for coarse-grained WSD.

\begin{table}
\begin{tabular}{|l|c|c|c|c|} \cline{2-5} \multicolumn{1}{c|}{} & \multicolumn{4}{|c|}{**SemEval-2013 task 12**} \\ \hline  & **IT** & **FR** & **DE** & **ES** \\ \hline BLSTM (bilingual) & 61.6 & 55.2 & **69.2** & 65.0 \\ BLSTM (multilingual) & 62.0 & 55.5 & **69.2** & 66.4 \\ \hline UMCC-DLSI & **65.8** & **60.5** & 62.1 & **71.0** \\ DAEBAK! & 61.3 & 53.8 & 59.1 & 60.0 \\ \hline MFS & 57.5 & 45.3 & 67.4 & 64.5 \\ \hline \end{tabular}
\end{table}
Table 3: F-scores (%) for multilingual WSD.

 

baseline and achieved results that are competitive with the best participating systems in the task. We also note that the overall F-score performance did not change substantially (and slightly improved) when moving from bilingual to multilingual models, despite the increase in the number of target languages treated simultaneously.

### Discussion and Error Analysis

All the neural models evaluated in Section 6.1 utilized the MFS back-off strategy for instances unseen at training time, which amounted to 9.4% overall for fine-grained WSD and 10.5% for coarse-grained WSD. Back-off strategy aside, 85% of the times the top candidate sense for a target instance lay within the 10 most probable entries in the probability distribution over \(O\) computed by the softmax layer.12 In fact, our sequence models learned, on the one hand, to associate a target word with its candidate senses (something word experts are not required to learn, as they only deal with a single word type at a time); on the other, they tended to generate softmax distributions reflecting the semantics of the surronding context. For example, in the sentence:

Footnote 12: We refer here to the same model considered in Section 6.2 (i.e., **BLSTM+att_+LEX**).

* The two _justices_ have been attending federalist society events for years,

our model correctly disambiguated _justices_ with the WordNet sense justice\({}^{3}_{n}\) (public official) rather than justice\({}^{1}_{n}\) (the quality of being just), and the corresponding softmax distribution was heavily biased towards words and senses related to persons or groups (_commissioners_, _defendants_, _jury_, _cabinet_, _directors_). On the other hand, in the sentence:

* Xavi Hernandez, the player of Barcelona, has 106 _matches_,

the same model disambiguated _matches_ with the wrong WordNet sense match\({}^{1}_{n}\) (tool for starting a fire). This suggests that the signal carried by discriminative words like _player_ vanishes rather quickly. In order to enforce global coherence further, recent contributions have proposed more sophisticated models where recurrent architectures are combined with Conditional Random Fields Huang et al. (2015); Ma and Hovy (2016). Finally, a number of errors were connected to shorter sentences with limited context for disambiguation: in fact, we noted that the average precision of our model, without MFS back-off, increased by 6.2% (from 74.6% to 80.8%) on sentences with more than 20 word tokens.

## 7 Conclusion

In this paper we adopted a new perspective on supervised WSD, so far typically viewed as a classification problem at the word level, and framed it using neural sequence learning. To this aim we defined, analyzed and compared experimentally different end-to-end models of varying complexities, including augmentations based on an attention mechanism and multitask learning.

Unlike previous supervised approaches, where a dedicated model needs to be trained for every content word and each disambiguation target is treated in isolation, sequence learning approaches learn a single model in one pass from the training data, and then disambiguate jointly all target words within an input text. The resulting models consistently achieved state-of-the-art (or statistically equivalent) figures in all benchmarks for all-words WSD, both fine-grained and coarse-grained, effectively demonstrating that we can overcome the so far undisputed and long-standing word-expert assumption of supervised WSD, while retaining the accuracy of supervised word experts.

Furthermore, these models are sufficiently flexible to allow them, for the first time in WSD, to be readily adapted to languages different from the one used at training time, and still achieve competitive results (as shown in Section 6.2). This crucial feature could potentially pave the way for cross-lingual supervised WSD, and overcome the shortage of sense-annotated data in multiple languages that, to date, has prevented the development of supervised models for languages other than English.

As future work, we plan to extend our evaluation to larger sense-annotated corpora Raganato et al. (2016) as well as to different sense inventories and different languages. We also plan to exploit the flexibility of our models by integrating them into downstream applications, such as Machine Translation and Information Extraction.

## Acknowledgments

The authors gratefully acknowledge the support of the ERC Consolidator Grant MOUSSE No. 726487 



## References

* Agirre and Edmonds (2007) Eneko Agirre and Philip Edmonds. 2007. _Word sense disambiguation: Algorithms and applications_, volume 33. Springer Science & Business Media.
* Agirre et al. (2014) Eneko Agirre, Oier Lopez de Lacalle, and Aitor Soroa. 2014. Random Walks for Knowledge-Based Word Sense Disambiguation. _Comput. Linguist._, 40(1):57-84.
* Alonso and Plank (2017) Hector Martinez Alonso and Barbara Plank. 2017. When is Multitask Learning Effective? Semantic Sequence Prediction under Varying Data Conditions. In _Proc. of ACL_, pages 44-53.
* Ammar et al. (2016) Waleed Ammar, George Mulcaire, Yulia Tsvetkov, Guillaume Lample, Chris Dyer, and Noah A. Smith. 2016. Massively Multilingual Word Embeddings. _CoRR_, abs/1602.01925.
* Baskaya and Jurgens (2016) Osman Baskaya and David Jurgens. 2016. Semi-supervised Learning with Induced Word Senses for State of the Art Word Sense Disambiguation. _J. Artif. Int. Res._, 55(1):1025-1058.
* Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In _ICLR Workshop_.
* Baroni et al. (2009) Marco Baroni, Silvia Bernardini, Adriano Ferraresi, and Eros Zanchetta. 2009. The WaCky wide web: a collection of very large linguistically processed web-crawled corpora. _Language Resources and Evaluation_, 43(3):209-226.
* Basile et al. (2014) Pierpaolo Basile, Annalina Caputo, and Giovanni Semeraro. 2014. An Enhanced Lesk Word Sense Disambiguation Algorithm through a Distributional Semantic Model. In _Proc. of COLING_, pages 1591-1600.
* Bingel and Sogaard (2017) Joachim Bingel and Anders Sogaard. 2017. Identifying Beneficial Task Relations for Multi-task Learning in Deep Neural Networks. In _Proc. of EACL_, pages 164-169.
* Bjerva et al. (2016) Johannes Bjerva, Barbara Plank, and Johan Bos. 2016. Semantic Tagging with Deep Residual Networks. In _Proc. of COLING_, pages 3531-3541.
* Butnaru et al. (2017) Andrei Butnaru, Radu Tudor Ionescu, and Florentina Infrstea. 2017. ShotgunWSD: an Unsupervised Algorithm for Global Word Sense Disambiguation Inspired by DNA Sequencing. In _Proc. of EACL_, pages 916-926.
* Carpuat and Wu (2007) Marine Carpuat and Dekai Wu. 2007. Improving Statistical Machine Translation using Word Sense Disambiguation. In _Proc. of EMNLP_, pages 61-72.
* Caruana (1997) Rich Caruana. 1997. Multitask learning. _Machine Learning_, 28(1):41-75.
* Chen et al. (2014) Xinxiong Chen, Zhiyuan Liu, and Maosong Sun. 2014. A Unified Model for Word Sense Representation and Disambiguation. In _Proc. of EMNLP_, pages 1025-1035.
* Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. In _Proc. of EMNLP_, pages 1724-1734.
* Ciaramita and Altun (2006) Massimiliano Ciaramita and Yasemin Altun. 2006. Broad-coverage Sense Disambiguation and Information Extraction with a Supersense Sequence Tagger. In _Proc. of EMNLP_, pages 594-602.
* Daude et al. (2003) Jordi Daude, Lluis Padro, and German Rigau. 2003. Validation and Tuning of WordNet Mapping Techniques. In _Proc. of RANLP_, pages 117-123.
* Lopez de Lacalle and Agirre (2015) Oier Lopez de Lacalle and Eneko Agirre. 2015. A Methodology for Word Sense Disambiguation at 90% based on large-scale Crowdourcing. In _Proc. of SEM_, pages 61-70.
* Bovi et al. (2015) Claudio Delli Bovi, Luca Telesca, and Roberto Navigli. 2015. Large-Scale Information Extraction from Textual Definitions through Deep Syntactic and Semantic Analysis. _TACL_, 3:529-543.
* Edmonds and Cotton (2001) Philip Edmonds and Scott Cotton. 2001. Senseval-2: Overview. In _Proc. of SENSEAL_, pages 1-5.
* Faruqui et al. (2016) Manaal Faruqui, Yulia Tsvetkov, Graham Neubig, and Chris Dyer. 2016. Morphological Infection Generation Using Character Sequence to Sequence Learning. In _Proc. of NAACL-HLT_, pages 634-643.
* Flekova and Gurevych (2016) Lucie Flekova and Iryna Gurevych. 2016. Supersense embeddings: A unified model for supersense interpretation, prediction, and utilization. In _Proc. of ACL_, pages 2029-2041.
* Graves (2013) Alex Graves. 2013. Generating Sequences With Recurrent Neural Networks. _CoRR_, abs/1308.0850.
* Graves and Schmidhuber (2005) Alex Graves and Jurgen Schmidhuber. 2005. Framewise phoneme classification with bidirectional LSTM and other neural network architectures. _Neural Networks_, 18(5-6):602-610.
* Gu et al. (2016) Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. 2016. Incorporating Copying Mechanism in Sequence-to-Sequence Learning. In _Proc. of ACL_, pages 1631-1640.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long Short-Term Memory. _Neural Comput._, 9(8):1735-1780.
* Huang et al. (2015) Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF Models for Sequence Tagging. _CoRR_, abs/1508.01991.
* Iacobacci et al. (2015) Ignacio Iacobacci, Mohammad Taher Pilehurar, and Roberto Navigli. 2015. SensEmbed: Learning Sense Embeddings for Word and Relational Similarity. In _Proc. of ACL_, pages 95-105

Ignacio Iacobacci, Mohammad Taher Pilehvar, and Roberto Navigli. 2016. Embeddings for Word Sense Disambiguation: An Evaluation Study. In _Proc. of ACL_, pages 897-907.
* Kageback and Salomonsson (2016) Mikael Kageback and Hans Salomonsson. 2016. Word Sense Disambiguation using a Bidirectional LSTM. In _Proceedings of CogALex_, pages 51-56.
* Kilgarriff (2001) Adam Kilgarriff. 2001. English Lexical Sample Task Description. In _Proc. of SENSEVAL_, pages 17-20.
* Kiros et al. (2015) Ryan Kiros, Yukun Zhu, Ruslan R. Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In _Proc. of NIPS_, pages 3294-3302.
* Luong et al. (2016) Thang Luong, Quoc V. Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2016. Multi-task sequence to sequence learning. In _ICLR Workshop_.
* Ma and Hovy (2016) Xuezhe Ma and Eduard Hovy. 2016. End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF. In _Proc. of ACL_, pages 1064-1074.
* McCarthy et al. (2016) Diana McCarthy, Marianna Apidianaki, and Katrin Erk. 2016. Word Sense Clustering and Clusterability. _Comput. Linguist._, 42(2):245-275.
* Melamud et al. (2016) Oren Melamud, Jacob Goldberger, and Ido Dagan. 2016. context2vec: Learning Generic Context Embedding with Bidirectional LSTM. In _Proc. of CoNLL_, pages 51-61.
* Mihalcea et al. (2004) Rada Mihalcea, Timothy Chklovski, and Adam Kilgarriff. 2004. The Senseval-3 English Lexical Sample Task. In _Proc. of SENSEVAL_, pages 25-28.
* Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In _ICLR Workshop_.
* Miller et al. (1990) George A. Miller, R.T. Beckwith, Christiane D. Fellbaum, D. Gross, and K. Miller. 1990. WordNet: an online lexical database. _International Journal of Lexicography_, 3(4):235-244.
* Miller et al. (1993) George A. Miller, Claudia Leacock, Randee Tengi, and Ross T. Bunker. 1993. A semantic concordance. In _Proc. of HLT_, pages 303-308.
* Moro and Navigli (2015) Andrea Moro and Roberto Navigli. 2015. SemEval-2015 Task 13: Multilingual All-Words Sense Dissambiguation and Entity Linking. In _Proc. of SemEval_, pages 288-297.
* Moro et al. (2014) Andrea Moro, Alessandro Raganato, and Roberto Navigli. 2014. Entity Linking meets Word Sense Disambiguation: a Unified Approach. _TACL_, 2:231-244.
* Mrksic et al. (2017) Nikola Mrksic, Ivan Vulic, Diarmuid O Seaghdha, Roi Reichart, Ira Leviant, Milica Gasic, Anna Kornonen, and Steve Young. 2017. Semantic Specialisation of Distributional Word Vector Spaces using Monolingual and Cross-Linequal Constraints. _TACL_, 5.
* Navigli (2009) Roberto Navigli. 2009. Word sense disambiguation: A survey. _ACM Computing Surveys_, 41(2):10.
* Navigli et al. (2013) Roberto Navigli, David Jurgens, and Daniele Vannella. 2013. SemEval-2013 Task 12: Multilingual Word Sense Disambiguation. In _Proc. of SemEval_, volume 2, pages 222-231.
* Navigli et al. (2007) Roberto Navigli, Kenneth C. Litkowski, and Orin Hargraves. 2007. SemEval-2007 Task 07: Coarse-grained English All-words Task. In _Proc. of SemEval_, pages 30-35.
* Navigli and Ponzetto (2012) Roberto Navigli and Simone Paolo Ponzetto. 2012. BabelNet: The Automatic Construction, Evaluation and Application of a Wide-Coverage Multilingual Semantic Network. _Artificial Intelligence_, 193:217-250.
* Neale et al. (2016) Steven Neale, Luis Gomes, Eneko Agirre, Oier Lopez de Lacalle, and Antnio Branco. 2016. Word Senseware Machine Translation: Including Senses as Contextual Features for Improved Translation Models. In _Proc. of LREC_, pages 2777-2783.
* Pasini and Navigli (2017) Tommaso Pasini and Roberto Navigli. 2017. Train-O-Matic: Large-Scale Supervised Word Sense Disambiguation in Multiple Languages without Manual Training Data. In _Proc. of EMNLP_.
* Pilehvar and Navigli (2014) Mohammad Taher Pilehvar and Roberto Navigli. 2014. A Large-scale Pseudoword-based Evaluation Framework for State-of-the-art Word Sense Disambiguation. _Comput. Linguistic._, 40(4):837-881.
* Plank et al. (2016) Barbara Plank, Anders Sogaard, and Yoav Goldberg. 2016. Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Loss. In _Proc. of ACL_, pages 412-418.
* Pradhan et al. (2007) Sameer S. Pradhan, Edward Loper, Dmitriy Digach, and Martha Palmer. 2007. SemEval-2007 Task 17: English Lexical Sample, SRL and All Words. In _Proc. of SemEval-2007_, pages 87-92.
* Raganato et al. (2016) Alessandro Raganato, Claudio Delli Bovi, and Roberto Navigli. 2016. Automatic Construction and Evaluation of a Large Semantically Enriched Wikipedia. In _Proc. of IJCAI_, pages 2894-2900.
* Raganato et al. (2017) Alessandro Raganato, Jose Camacho-Collados, and Roberto Navigli. 2017. Word Sense Disambiguation: A Unified Evaluation Framework and Empirical Comparison. In _Proc. of EACL_, pages 99-110.
* Rothe and Schutze (2015) Sascha Rothe and Hinrich Schutze. 2015. AutoExtend: Extending Word Embeddings to Embeddings for Synsets and Lexemes. In _Proc. of ACL_, pages 1793-1803.
* Shen et al. (2013) Hui Shen, Razvan Bunescu, and Rada Mihalcea. 2013. Coarse to Fine Grained Sense Disambiguation in Wikipedia. In _Proc. of SEM_, pages 22-31.
* Snyder and Palmer (2004) Benjamin Snyder and Martha Palmer. 2004. The english all-words task. In _Proc. of Senseval-3_, pages 41-43 

Anders Sogaard and Yoav Goldberg. 2016. Deep multi-task learning with low level tasks supervised at lower layers. In _Proc. of ACL_, pages 231-235.
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to Sequence Learning with Neural Networks. In _Proc. of NIPS_, pages 3104-3112.
* Taghipour and Ng (2015a) Kaveh Taghipour and Hwee Tou Ng. 2015a. One Million Sense-Tagged Instances for Word Sense Disambiguation and Induction. In _Proc. of CoNLL_, pages 338-344.
* Taghipour and Ng (2015b) Kaveh Taghipour and Hwee Tou Ng. 2015b. Semi-Supervised Word Sense Disambiguation Using Word Embeddings in General and Specific Domains. In _Proc. of NAACL-HLT_, pages 314-323.
* Tripodi and Pelillo (2017) Rocco Tripodi and Marcello Pelillo. 2017. A Game-Theoretic Approach to Word Sense Disambiguation. _Comput. Linguist._, 43(1):31-70.
* Vinyals et al. (2015) Oriol Vinyals, L ukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. 2015. Grammar as a foreign language. In _Proc. of NIPS_, pages 2773-2781.
* Vinyals and Le (2015) Oriol Vinyals and Quoc V. Le. 2015. A Neural Conversational Model. In _Proc. of ICML, JMLR: W&CP_, volume 37.
* Wang and Chang (2016) Wenhui Wang and Baobao Chang. 2016. Graph-based Dependency Parsing with Bidirectional LSTM. In _Proc. of ACL_, pages 2306-2315.
* Weissenborn et al. (2015) Dirk Weissenborn, Leonhard Hennig, Feiyu Xu, and Hans Uszkoreit. 2015. Multi-Objective Optimization for the Joint Disambiguation of Nouns and Named Entities. In _Proc. of ACL_, pages 596-605.
* Xiong and Zhang (2014) Deyi Xiong and Min Zhang. 2014. A Sense-Based Translation Model for Statistical Machine Translation. In _Proc. of ACL_, pages 1459-1469.
* Yuan et al. (2016) Dayu Yuan, Ryan Doherty, Julian Richardson, Colin Evans, and Eric Altendorf. 2016. Semi-supervised Word Sense Disambiguation with Neural Models. In _Proc. of COLING_, pages 1374-1385.
* Zeiler (2012) Matthew D. Zeiler. 2012. ADADELTA: An Adaptive Learning Rate Method. _CoRR_, abs/1212.5701.
* Zhong and Ng (2010) Zhi Zhong and Hwee Tou Ng. 2010. It Makes Sense: A Wide-Coverage Word Sense Disambiguation System for Free Text. In _Proc. of ACL: System Demonstrations_, pages 78-83.
* Zhong and Ng (2012) Zhi Zhong and Hwee Tou Ng. 2012. Word Sense Disambiguation Improves Information Retrieval. In _Proc. of ACL_, pages 273-282.
* Zhou and Xu (2015) Jie Zhou and Wei Xu. 2015. End-to-End Learning of Semantic Role Labeling using Recurrent Neural Networks. In _Proc. of ACL_, pages 1127-1137.
* Zhou et al. (2016) Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, and Bo Xu. 2016. Attention-Based Bidirectional Long Short-Term Memory Networks for Relation Classification. In _Proc. of ACL_, pages 207-212.