text: {
    "name" : "494.pdf",
    "metadata" : {
      "source" : "CRF",
      "title" : "Morph-fitting: Fine-Tuning Word Vector Spaces with Simple Language-Specific Rules",
      "authors" : [ ],
      "emails" : [ ],
      "sections" : [ {
        "heading" : null,
        "text" : "1 000\n011\n012\n013\n014\n015\n016\n017\n018\n019\n020\n021\n022\n023\n024\n025\n026\n027\n028\n029\n030\n031\n032\n033\n034\n035\n036\n037\n038\n039\n040\n041\n042\n043\n044\n045\n046\n047\n048\n049\n061\n062\n063\n064\n065\n066\n067\n068\n069\n070\n071\n072\n073\n074\n075\n076\n077\n078\n079\n080\n081\n082\n083\n084\n085\n086\n087\n088\n089\n090\n091\n092\n093\n094\n095\n096\n097\n098\n099"
      }, {
        "heading" : "1 Introduction",
        "text" : "Word representation learning has become a research area of central importance in natural language processing (NLP), with its usefulness demonstrated across many application areas such as parsing (Chen and Manning, 2014), machine translation (Zou et al., 2013), and many others (Turian et al., 2010; Collobert et al., 2011). Most promi-\nnent word representation techniques are grounded in the distributional hypothesis, relying on word co-occurrence information in large textual corpora (Curran, 2004; Turney and Pantel, 2010; Mikolov et al., 2013; Mnih and Kavukcuoglu, 2013; Levy and Goldberg, 2014; Schwartz et al., 2015, i.a.).\nMorphologically rich languages, in which “substantial grammatical information. . . is expressed at word level” (Tsarfaty et al., 2010), pose specific challenges for NLP. This is not always considered when techniques are evaluated on languages such as English or Chinese, which do not have rich morphology. In the case of distributional vector space models, morphological complexity brings two challenges to the fore:\n1. Estimating Rare Words: A single lemma can have many different surface realisations. Naively treating each realisation as a separate word leads to sparsity problems and a failure to exploit their shared semantics. On the other hand, lemmatising the entire corpus can obfuscate the differences that exist between different word forms even though they share some aspects of meaning.\n2. Embedded Semantics: Morphology can encode semantic relations such as antonymy (e.g. literate and illiterate, expensive and inexpensive) or synonymy (north, northern, northerly).\nIn this work, we tackle the two challenges jointly by introducing a resource-light vector space finetuning procedure termed morph-fitting. The proposed method does not require curated knowledge bases or gold lexicons. Instead, it makes use of the observation that morphology implicitly encodes semantic signals pertaining to synonymy (e.g., German word inflections katalanisch, katalanischem, katalanischer denote the same semantic concept in different grammatical roles), and antonymy (e.g., mature vs. immature), capitalising on the proliferation of word forms in morphologically\n2\n101\n102\n103\n104\n105\n106\n107\n108\n109\n110\n111\n112\n113\n114\n115\n116\n117\n118\n119\n120\n121\n122\n123\n124\n125\n126\n127\n128\n129\n130\n131\n132\n133\n134\n135\n136\n137\n138\n139\n140\n141\n142\n143\n144\n145\n146\n147\n148\n149\n150\n151\n152\n153\n154\n155\n156\n157\n165\n166\n167\n168\n169\n170\n171\n172\n173\n174\n175\n176\n177\n178\n179\n180\n181\n182\n183\n184\n185\n186\n187\n188\n189\n190\n191\n192\n193\n194\n195\n196\n197\n198\n199\nen_expensive de_teure it_costoso en_slow de_langsam it_lento en_book de_buch it_libro costly teuren dispendioso fast allmählich lentissimo books sachbuch romanzo\ncostlier kostspielige remunerativo slower rasch lenta memoir buches racconto cheaper aufwändige redditizio slower gemächlich inesorabile novel romandebüt volumetto prohibitively kostenintensive rischioso slowed schnell rapidissimo storybooks büchlein saggio pricey aufwendige costosa slowing explosionsartig graduale blurb pamphlet ecclesiaste\nexpensiveness teures costosa slow langsamer lenti booked bücher libri costly teuren costose slowing langsames lente rebook büch libra\ncostlier teurem costosi slowed langsame lenta booking büche librare ruinously teurer dispendioso slowness langsamem veloce rebooked büches libre unaffordable teurerer dispendiose slows langsamen rapido books büchen librano\nTable 1: The nearest neighbours of three example words (expensive, slow and book) in English, German and Italian before (top) and after (bottom) morph-fitting.\nrich languages. Formalised as an instance of the post-processing semantic specialisation paradigm (Faruqui et al., 2015; Mrkšić et al., 2016), morphfitting is steered by a set of linguistic constraints derived from simple language-specific rules which describe (a subset of) morphological processes in a language. The constraints emphasise similarity on one side (e.g., by extracting morphological synonyms), and antonymy on the other (by extracting morphological antonyms), see Fig. 1 and Tab. 2.\nThe key idea of the fine-tuning process is to pull synonymous examples described by the constraints closer together in a transformed vector space, while at the same time pushing antonymous examples away from each other. The explicit post-hoc injection of morphological constraints enables: a) estimating more accurate vectors for low-frequency words if they are described by the constraints containing their relation with high-frequency words,1 thus tackling the data sparsity problem; and b) specialising the distributional space to distinguish between similarity and association, thus supporting language understanding applications such as dialogue state tracking (DST).\nAs a post-processor, morph-fitting allows the integration of morphological rules with any distributional vector space in any language: it treats an input distributional word vector space as a black box and fine-tunes it so that the transformed space reflects the knowledge coded in the input morphological constraints (e.g., Italian words rispettoso and irrispetosa should be far apart in the transformed vector space, see Fig. 1). Tab. 1 illustrates the effects of morph-fitting by qualitative examples in three languages: the vast majority of nearest neighbours are “morphological” synonyms.\nWe demonstrate the efficacy of morph-fitting in four languages (English, German, Italian, Rus-\n1For instance, the vector for the word katalanischem which occurs only 9 times in the German Wikipedia will be pulled closer to the more reliable vectors for katalanisch and katalanischer, with frequencies of 2097 and 1383 respectively.\nsian), yielding large and consistent improvements on benchmarking word similarity evaluation sets such as SimLex-999 (Hill et al., 2015), its multilingual extension (Leviant and Reichart, 2015), and SimVerb-3500 (Gerz et al., 2016). The improvements are reported for all four languages, and with a variety of input distributional spaces, verifying the robustness of the approach.\nWe then show that incorporating morph-fitted vectors into a state-of-the-art neural-network DST model results in improved tracking performance, especially for morphologically rich languages. We report an improvement of 4% on Italian, and 6% on German when using morph-fitted vectors instead of the distributional ones, setting a new state-of-theart DST performance for the two datasets.2"
      }, {
        "heading" : "2 Morph-fitting: Methodology",
        "text" : "Preliminaries In this work, we focus on four languages with varying levels of morphological complexity: English (EN), German (DE), Italian (IT), and Russian (RU). These correspond to languages in the Multilingual SimLex-999 dataset. Vocabularies Wen, Wde, Wit, Wru are compiled by retaining all word forms from the four Wikipedias with\n2There are no readily available DST datasets for Russian.\n3\n201\n202\n203\n204\n205\n206\n207\n208\n209\n210\n211\n212\n213\n214\n215\n216\n217\n218\n219\n220\n221\n222\n223\n224\n225\n226\n227\n228\n229\n230\n231\n232\n233\n234\n235\n236\n237\n238\n239\n240\n241\n242\n243\n244\n245\n246\n247\n248\n249\n250\n251\n252\n253\n254\n255\n256\n257\n263\n264\n265\n266\n267\n268\n269\n270\n271\n272\n273\n274\n275\n276\n277\n278\n279\n280\n281\n282\n283\n284\n285\n286\n287\n288\n289\n290\n291\n292\n293\n294\n295\n296\n297\n298\n299\nword frequency over 10, see Tab. 3. We then query these (large) vocabularies using a set of simple language-specific if-then-else rules to extract sets of linguistic constraints, see Tab. 2.3 These constraints (Sect. 2.2) are used as input for the vector space post-processing ATTRACT-REPEL algorithm (outlined in Sect. 2.1)."
      }, {
        "heading" : "2.1 The ATTRACT-REPEL Model",
        "text" : "The ATTRACT-REPEL model is an extension of PARAGRAM, proposed by Wieting et al. (2015). It provides a generic framework for incorporating similarity (e.g. successful and accomplished) and antonymy constraints (e.g. nimble and clumsy) into pre-trained word vectors. Given the initial vector space and collections of ATTRACT and REPEL constraints A and R, the model gradually modifies the space to bring the designated word vectors closer together or further apart. The method’s cost function consists of three terms. The first term pulls the ATTRACT examples (xl, xr) ∈ A closer together. If BA denotes the current mini-batch of ATTRACT examples, this term can be expressed as:\nA(BA) = ∑\n(xl,xr)∈BA\n(ReLU (δatt + xltl − xlxr)\n+ ReLU (δatt + xrtr − xlxr))\nwhere δatt is the similarity margin which determines how much closer synonymous vectors should be to each other than to each of their respective negative examples. ReLU(x) = max(0, x) is the standard rectified linear unit (Nair and Hinton, 2010). The ‘negative’ example ti for each word xi in any ATTRACT pair is the word vector closest to xi among the examples in the current minibatch (distinct from its target synonym and xi itself). This means that this term forces synonymous words from the in-batch ATTRACT constraints to be closer to one another than to any other word in the current mini-batch.\nThe second term pushes antonyms away from each other. If (xl, xr) ∈ BR is the current minibatch of REPEL constraints, this term is:\nR(BR) = ∑\n(xl,xr)∈BR\n(ReLU (δrpl + xlxr − xltr)\n+ ReLU (δrpl + xlxr − xrtr)) 3A native speaker is able to easily come up with these sets of morphological rules (or at least with a reasonable subset of rules) without any linguistic training. What is more, the rules for DE, IT, and RU were created by non-native, non-fluent speakers with a limited knowledge of the three languages, exemplifying the simplicity and portability of the approach.\nEnglish German Italian\n(discuss, discussed) (schottisch, schottischem) (golfo, golfi) (laugh, laughing) (damalige, damaligen) (minato, minata) (pacifist, pacifists) (kombiniere, kombinierte) (mettere, metto) (evacuate, evacuated) (schweigt, schweigst) (crescono, cresci) (evaluate, evaluates) (hacken, gehackt) (crediti, credite)\n(dressed, undressed) (stabil, unstabil) (abitata, inabitato) (similar, dissimilar) (geformtes, ungeformt) (realtà, irrealtà) (formality, informality) (relevant, irrelevant) (attuato, inattuato)\nIn this case, each word’s ‘negative’ example is the (in-batch) word vector furthest away from it (and distinct from the word’s target antonym). The intuition is that we want antonymous words from the input REPEL constraints to be further away from each other than from any other word in the current mini-batch; δrpl is now the repel margin.\nThe final term of the cost function serves to retain the abundance of semantic information encoded in the starting distributional space. If xiniti is the initial distributional vector and V (B) is the set of all vectors present in the given mini-batch, this term (per mini-batch) is expressed as:\nR(BA,BR) = ∑\nxi∈V (BA∪BR)\nλreg ∥∥∥xiniti − xi∥∥∥ 2\nwhere λreg is the L2 regularisation constant.4 This term effectively pulls word vectors towards their initial (distributional) values, ensuring that relations encoded in initial vectors persist as long as they do not contradict the newly injected ones."
      }, {
        "heading" : "2.2 Language-Specific Rules and Constraints",
        "text" : "Semantic Specialisation with Constraints The fine-tuning ATTRACT-REPEL procedure is entirely driven by the input ATTRACT and REPEL sets of constraints. These can be extracted from a variety of semantic databases such as WordNet (Fellbaum, 1998), the Paraphrase Database (Ganitkevitch et al., 2013; Pavlick et al., 2015), or BabelNet (Navigli and Ponzetto, 2012; Ehrmann et al., 2014) as done in prior work (Faruqui et al., 2015; Wieting et al.,\n4We use hyperparameter values δatt = 0.6, δrpl = 0.0, λreg = 10\n−9 from prior work without fine-tuning. We train all models for 10 epochs with AdaGrad (Duchi et al., 2011).\n4\n301\n302\n303\n304\n305\n306\n307\n308\n309\n310\n311\n312\n313\n314\n315\n316\n317\n318\n319\n320\n321\n322\n323\n324\n325\n326\n327\n328\n329\n330\n331\n332\n333\n334\n335\n336\n337\n338\n339\n340\n341\n342\n343\n344\n345\n346\n347\n348\n349\n350\n351\n352\n353\n354\n355\n356\n357\n358\n359\n360\n361\n362\n363\n364\n365\n366\n367\n368\n369\n370\n371\n372\n373\n374\n375\n376\n377\n378\n379\n380\n381\n382\n383\n384\n385\n386\n387\n388\n389\n390\n391\n392\n393\n394\n395\n396\n397\n398\n399\n2015; Mrkšić et al., 2016, i.a.). In this work, we investigate another option: extracting constraints without curated knowledge bases in a spectrum of languages by exploiting inherent language-specific properties related to linguistic morphology. This relaxation ensures a wider portability of ATTRACTREPEL to languages and domains without readily available or adequate resources.\nExtracting ATTRACT Pairs For the ATTRACT constraints, we focus on inflectional rather than on derivational morphology rules as the former preserve the full meaning of a word, modifying it only to reflect grammatical roles (e.g., verb tense, case markers; (read, reads)).5 This choice is guided by our intent to fine-tune the original vector space to improve the embedded semantic relations.\nWe define two rules for English, widely recognised as morphologically simple (Avramidis and Koehn, 2008; Cotterell et al., 2016). These are: (R1) if w1, w2 ∈Wen, where w2 = w1 + ing/ed/s, then add (w1, w2) and (w2, w1) to the set of ATTRACT constraints A. This rule yields pairs such as (look, looks), (look, looking), (look, looked).\nIf w[: −1] is a function which strips the last character from word w, the second rule is: (R2) if w1 ends with the letter e and w1 ∈Wen and w2 ∈ Wen, where w2 = w1[: −1] + ing/ed/s, then add (w1, w2) and (w2, w1) toA. This creates pairs such as (create, creates), (create, creating) and (create, created). Naturally, introducing more sophisticated rules is possible in order to cover for other special cases and morphological irregularities (e.g., sweep / swept), but in all our EN experiments, A is based on the two simple EN rules R1 and R2.\nThe other three languages, with more complicated morphology, yield a larger number of rules. In Italian, we rely on the sets of rules spanning: (1) regular formation of plural (libro / libri); (2) regular verb conjugation (aspettare / aspettiamo); (3) regular formation of past participle (aspettare / aspettato); and (4) rules regarding grammatical gender (bianco / bianca). Besides these, another set of rules is used for German and Russian: (5) regular declension (e.g., asiatisch / asiatischem).\n5The core difference between inflectional and derivational morphology may be summarised in a few lines as follows: the former refers to a set of processes through which the word form expresses meaningful syntactic information, e.g., verb tense, without any change to the semantics of the word. On the other hand, the latter refers to the formation of new words with semantic shifts in meaning (Schone and Jurafsky, 2001; Haspelmath and Sims, 2013; Lazaridou et al., 2013; Zeller et al., 2013; Cotterell and Schütze, 2017).\nExtracting REPEL Pairs As another source of implicit semantic signals, W also contains words which represent derivational antonyms: e.g., two words that denote concepts with opposite meanings, generated through a derivational process. We use a standard set of EN “antonymy” prefixes: APen = {dis, il, un, in, im, ir, mis, non, anti} (Fromkin et al., 2013). If w1, w2 ∈ Wen, where w2 is generated by adding a prefix from APen to w1, then (w1, w2) and (w2, w1) are added to the set of REPEL constraints R. This rule generates pairs such as (advantage, disadvantage) and (regular, irregular). An additional rule replaces the suffix -ful with -less, extracting antonyms such as (careful, careless).\nFollowing the same principle, we use APde = {un, nicht, anti, ir, in, miss}, APit = {in, ir, im, anti}, and APru = {не, анти}. For instance, this generates an IT pair (rispettoso, irrispettoso) (see Fig. 1). For DE, we use another rule targeting suffix replacement: -voll is replaced by -los.\nWe further expand the set of REPEL constraints by transitively combining antonymy pairs from the previous step with inflectional ATTRACT pairs. This step yields additional constraints such as (rispettosa, irrispettosi) (see Fig. 1). The final A andR constraint counts are given in Tab. 3. The full sets of rules are available as supplemental material."
      }, {
        "heading" : "3 Experimental Setup",
        "text" : "Training Data and Setup For each of the four languages we train the skip-gram with negative sampling (SGNS) model (Mikolov et al., 2013) on the latest Wikipedia dump of each language. We induce 300-dimensional word vectors, with the frequency cut-off set to 10. The vocabulary sizes |W | for each language are provided in Tab. 3.6 We label these collections of vectors SGNS-LARGE.\nOther Starting Distributional Vectors We also analyse the impact of morph-fitting on other collections of well-known EN word vectors. These vectors have varying vocabulary coverage and are trained with different architectures. We test standard distributional models: Common-Crawl GloVe (Pennington et al., 2014), SGNS vectors (Mikolov et al., 2013) with various contexts (BOW = bag-ofwords; DEPS = dependency contexts), and training data (PW = Polyglot Wikipedia from Al-Rfou\n6Other SGNS parameters were set to standard values (Baroni et al., 2014; Vulić and Korhonen, 2016b): 15 epochs, 15 negative samples, global learning rate: .025, subsampling rate: 1e− 4. Similar trends in results persist with d = 100, 500.\n5\n401\n402\n403\n404\n405\n406\n407\n408\n409\n410\n411\n412\n413\n414\n415\n416\n417\n418\n419\n420\n421\n422\n423\n424\n425\n426\n427\n428\n429\n430\n431\n432\n433\n434\n435\n436\n437\n438\n439\n440\n441\n442\n443\n444\n445\n446\n447\n448\n449\n450\n451\n452\n453\n454\n455\n456\n457\n458\n459\n460\n461\n462\n463\n464\n465\n466\n467\n468\n469\n470\n471\n472\n473\n474\n475\n476\n477\n478\n479\n480\n481\n482\n483\n484\n485\n486\n487\n488\n489\n490\n491\n492\n493\n494\n495\n496\n497\n498\n499\net al. (2013); 8B = 8 billion token word2vec corpus), following (Levy and Goldberg, 2014) and (Schwartz et al., 2015). We also test the symmetricpattern based vectors of Schwartz et al. (2016) (SymPat-Emb), count-based PMI-weighted vectors reduced by SVD (Baroni et al., 2014) (Count-SVD), a model which replaces the context modelling function from CBOW with bidirectional LSTMs (Melamud et al., 2016) (Context2Vec), and two sets of EN vectors trained by injecting multilingual information: BiSkip (Luong et al., 2015) and MultiCCA (Faruqui and Dyer, 2014). We also experiment with a selection of standard distributional spaces in other languages from prior work (Dinu et al., 2015; Luong et al., 2015; Vulić and Korhonen, 2016a).\nMorph-fixed Vectors A baseline which utilises an equal amount of knowledge as morph-fitting, termed morph-fixing, fixes the vector of each word to the distributional vector of its most frequent inflectional synonym, tying the vectors of lowfrequency words to their more frequent inflections. For each word w1, we construct a set of M + 1 words Ww1 = {w1, w′1, . . . , w′M} consisting of the word w1 itself and all M words which cooccur with w1 in the ATTRACT constraints. We then choose the word w′max from the set Ww1 with the maximum frequency in the training data, and fix all other word vectors in Ww1 to its word vector. The morph-fixed vectors (MFIX) serve as our primary baseline, as they outperformed another straightforward baseline based on stemming across all of our intrinsic and extrinsic experiments.\nMorph-fitting Variants We analyse two variants of morph-fitting: (1) using ATTRACT constraints only (MFIT-A), and (2) using both ATTRACT and REPEL constraints (MFIT-AR).7"
      }, {
        "heading" : "4 Intrinsic Evaluation: Word Similarity",
        "text" : "Evaluation Setup and Datasets The first set of experiments intrinsically evaluates morph-fitted vector spaces on word similarity benchmarks, using Spearman’s rank correlation as the evaluation metric. First, we use the SimLex-999 dataset, as well as SimVerb-3500, a recent EN verb pair similarity dataset providing similarity ratings for 3,500 verb\n7We also tried using another post-processing model (Mrkšić et al., 2016) in lieu of ATTRACT-REPEL. However, this model was computationally intractable with SGNSLARGE vectors. Moreover, it was consistently outperformed by ATTRACT-REPEL on vector spaces with smaller vocabularies.\npairs.8 SimLex-999 was translated to DE, IT, and RU by Leviant and Reichart (2015), and they crowdsourced similarity scores from native speakers. We use this dataset for our multilingual evaluation.9\nMorph-SimLex We also introduce a synthetic dataset based on multilingual SimLex, termed Morph-SimLex. Since the original sets contain only word lemmas, they are unable to evaluate whether a representation model improves vectors for all synonymous word inflections. Therefore, we enrich the sets of pairs using the same set of ATTRACT rules from Sect. 2.2. In short, given a word pair (w1, w2) with a SimLex score sl1,2, we again construct sets Ww1 = {w1, w′1, . . . , w′M} and Ww2 = {w2, w′′1 , . . . , w′′N}, where Ww1 consists of w1 and all words which co-occur with w1 in the A constraints; the same holds for Ww2 . Morph-SimLex pairs are then generated by taking the Cartesian product between Ww1 and Ww2 , and assigning the same score sl1,2 to each such pair. The final dataset is constructed by repeating the procedure for each of the 999 SimLex pairs, yielding 13,213 EN pairs, 17,021 DE pairs, 18,281 IT pairs, and 10,289 RU pairs. We make this dataset available in hope it can aid further research on improving morphological relations in vector spaces.\nMorph-fitting EN Word Vectors As the first experiment, we morph-fit a wide spectrum of EN distributional vectors induced by various architectures (see Sect. 3). The results on SimLex and SimVerb are summarised in Tab. 4. The results with EN SGNS-LARGE vectors are shown in Fig. 2a. Morphfitted vectors bring consistent improvement across all experiments, regardless of the quality of the initial distributional space. This finding confirms that the method is robust: its effectiveness does not depend on the architecture used to construct the initial space. To illustrate the improvements, note that the best score on SimVerb for a model trained on running text is achieved by Context2vec (ρ = 0.388); injecting morphological constraints into this vector space results in a gain of 7.1 ρ points.\nExperiments on Other Languages We next extend our experiments to other languages, testing both morph-fitting variants. The results are sum-\n8Unlike other gold standard resources such as WordSim353 (Finkelstein et al., 2002) or MEN (Bruni et al., 2014), SimLex and SimVerb provided explicit guidelines to discern between semantic similarity and association, so that related but non-similar words (e.g. cup and coffee) have a low rating.\n9Since Leviant and Reichart (2015) re-scored the original EN SimLex, we use their EN SimLex version for consistency.\n6\n501\n502\n503\n504\n505\n506\n507\n508\n509\n510\n511\n512\n513\n514\n515\n516\n517\n518\n519\n520\n521\n522\n523\n524\n525\n526\n527\n528\n529\n530\n531\n532\n533\n534\n535\n536\n537\n538\n539\n540\n541\n542\n543\n544\n545\n546\n547\n548\n549\n550\n551\n552\n553\n554\n555\n556\n557\n558\n559\n560\n561\n562\n563\n564\n565\n566\n567\n568\n569\n570\n571\n572\n573\n574\n575\n576\n577\n578\n579\n580\n581\n582\n583\n584\n585\n586\n587\n588\n589\n590\n591\n592\n593\n594\n595\n596\n597\n598\n599\nEvaluation Vectors SimLex-999 SimVerb-3500 1. SG-BOW2-PW (300) (Mikolov et al., 2013) .339→ .439 .277→ .381 2. GloVe-6B (300) (Pennington et al., 2014) .324→ .438 .286→ .405 3. Count-SVD (500) (Baroni et al., 2014) .267→ .360 .199→ .301 4. SG-DEPS-PW (300) (Levy and Goldberg, 2014) .376→ .434 .313→ .418 5. SG-DEPS-8B (500) (Bansal et al., 2014) .373→ .441 .356→ .473 6. MultiCCA-EN (512) (Faruqui and Dyer, 2014) .314→ .391 .296→ .354 7. BiSkip-EN (256) (Luong et al., 2015) .276→ .356 .260→ .333 8. SG-BOW2-8B (500) (Schwartz et al., 2015) .373→ .440 .348→ .441 9. SymPat-Emb (500) (Schwartz et al., 2016) .381→ .442 .284→ .373 10. Context2Vec (600) (Melamud et al., 2016) .371→ .440 .388→ .459\nTable 4: The impact of morph-fitting (MFIT-AR used) on a representative set of EN vector space models. All results show the Spearman’s ρ correlation before and after morph-fitting. The numbers in parentheses refer to the vector dimensionality.\nVectors Distrib. MFIT-A MFIT-AR EN: GloVe-6B (300) .324 .376 .438 EN: SG-BOW2-PW (300) .339 .385 .439 DE: SG-DEPS-PW (300) (Vulić and Korhonen, 2016a) .267 .318 .325 DE: BiSkip-DE (256) (Luong et al., 2015) .354 .414 .421 IT: SG-DEPS-PW (300) (Vulić and Korhonen, 2016a) .237 .351 .391 IT: CBOW5-Wacky (300) (Dinu et al., 2015) .363 .417 .446\nTable 5: Results on multilingual SimLex-999 (EN, DE, and IT) with two morph-fitting variants.\nmarised in Tab. 5, while Fig. 2a-2d show results for the morph-fitted SGNS-LARGE vectors. These scores confirm the effectiveness and robustness of morph-fitting across languages, suggesting that the idea of fitting to morphological constraints is indeed language-agnostic, given the set of languagespecific rule-based constraints. Fig. 2 also demonstrates that the morph-fitted vector spaces consistently outperform the morph-fixed ones.\nMorph-SimLex performance across all languages shows even stronger relative gains over distributional and morph-fixed vectors. The original SimLex dataset only contains word lemmas. Consequently, it fails to penalise word vector collections with bad estimates of less-frequent word forms. The comparison between MFIT-A and MFIT-AR indicates that both sets of constraints are important\nfor the fine-tuning process: while MFIT-A already yields consistent gains over the initial spaces, a further refinement can be achieved by also incorporating the antonymous REPEL constraints."
      }, {
        "heading" : "5 Downstream Task: Dialogue State Tracking (DST)",
        "text" : "Goal-oriented dialogue systems provide conversational interfaces for tasks such as booking flights or finding restaurants. In slot-based systems, application domains are specified using ontologies that define the search constraints which users can express. An ontology consists of a number of slots and their assorted slot values. In a restaurant search domain, sets of slot-values could include PRICE = [cheap, expensive] or FOOD = [Thai, Indian, ...]. The DST model is the first component of modern dialogue pipelines (Young, 2010). It serves to capture the intents expressed by the user at each dialogue turn and update the belief state. This is the system’s internal estimate of the user’s goals, used by the downstream dialogue manager to choose the system response. The following example shows the true dialogue state in a multi-turn dialogue:\nUser: What’s good in the southern part of town? inform(area=south) System: Vedanta is the top-rated Indian place. User: How about something cheaper? inform(area=south, price=cheap) System: Seven Days is very popular. Great hot pot. User: What’s the address? inform(area=south, price=cheap); request(address) System: Seven Days is at 66 Regent Street.\nThe Dialogue State Tracking Challenge (DSTC) shared task series formalised the evaluation and provided labelled DST datasets (Henderson et al., 2014a,b; Williams et al., 2016). While a plethora of DST models are available based on, e.g., handcrafted rules (Wang et al., 2014) or conditional random fields (Lee and Eskenazi, 2013), the recent DST methodology has seen a shift towards neuralnetwork architectures (Henderson et al., 2014c; Mrkšić et al., 2015; Liu and Perez, 2017, i.a.)\nModel: Neural Belief Tracker To detect intents in user utterances, most existing models rely on either (or both): 1) Spoken Language Understanding models which require large amounts of annotated training data; or 2) hand-crafted, domain-specific lexicons which try to capture lexical and morphological variation. The Neural Belief Tracker (NBT) is a novel DST model which overcomes both issues\n7\n601\n602\n603\n604\n605\n606\n607\n608\n609\n610\n611\n612\n613\n614\n615\n616\n617\n618\n619\n620\n621\n622\n623\n624\n625\n626\n627\n628\n629\n630\n631\n632\n633\n634\n635\n636\n637\n638\n639\n640\n641\n642\n643\n644\n645\n646\n647\n648\n649\n650\n651\n652\n653\n654\n655\n656\n657\n658\n659\n660\n661\n662\n663\n664\n665\n666\n667\n668\n669\n670\n671\n672\n673\n674\n675\n676\n677\n678\n679\n680\n681\n682\n683\n684\n685\n686\n687\n688\n689\n690\n691\n692\n693\n694\n695\n696\n697\n698\n699\n0.1\n0.15\n0.2\n0.25\n0.3\n0.35\n0.4\n0.45\nDistributional MFix MFit-A MFit-AR 0.6\n0.65\n0.7\n0.75\n0.8\nS im\nL ex\n(S p ea rm\nan ’s ρ )\nEN Vector Collections\nD S T P erform ace (Joint)\nSimLex Morph-SimLex DST\n(a) English\n0.1\n0.15\n0.2\n0.25\n0.3\n0.35\n0.4\n0.45\nDistributional MFix MFit-A MFit-AR 0.6\n0.65\n0.7\n0.75\n0.8\nS im\nL ex\n(S p ea rm\nan ’s ρ )\nDE Vector Collections\nD S T P erform ace (Joint) SimLex Morph-SimLex DST\n(b) German\n0.1\n0.15\n0.2\n0.25\n0.3\n0.35\n0.4\n0.45\nDistributional MFix MFit-A MFit-AR 0.6\n0.65\n0.7\n0.75\n0.8\nS im\nL ex\n(S p ea rm\nan ’s ρ )\nIT Vector Collections\nD S T P erform ace (Joint) SimLex Morph-SimLex DST\n(c) Italian\n0.1\n0.15\n0.2\n0.25\n0.3\n0.35\n0.4\n0.45\nDistributional MFix MFit-A MFit-AR S im\nL ex\n(S p ea rm\nan ’s ρ )\nRU Vector Collections\nSimLex Morph-SimLex\n(d) Russian\nFigure 2: An overview of the results (Spearman’s ρ correlation) for four languages on SimLex-999 (blue squares), Morph-SimLex-999 (red triangles), and the downstream DST performance (black diamonds) using SGNS-LARGE vectors (d = 300), see Tab. 3 and Sect. 3. The left y axis measures the intrinsic word similarity performance, while the right y axis provides the scale for the DST performance.\nby reasoning purely over pre-trained word vectors (Mrkšić et al., 2016). The NBT learns to compose these vectors into intermediate utterance and context representations. These are then used to decide which of the ontology-defined intents (goals) have been expressed by the user. The NBT model keeps word vectors fixed during training, so that unseen, yet related words can be mapped to the right intent at test time (e.g. northern to north).\nData: Multilingual WOZ 2.0 Dataset Our DST evaluation is based on the WOZ dataset, released by Wen et al. (2017). In this Wizard-of-Oz setup, two Amazon Mechanical Turk workers assumed the role of the user and the system asking/providing restaurant information. Users typed instead of speaking, removing the need to deal with noisy speech recognition. In DSTC datasets, users would quickly adapt to the system’s inability to deal with complex queries. Conversely, the WOZ setup allowed them to use sophisticated language. The WOZ 2.0 release expanded the dataset to 1,200 dialogues (Mrkšić et al., 2016). In this work, we use translations of this dataset to Italian and German, provided by the authors of the original dataset.\nEvaluation Setup The principal metric we use to\nmeasure DST performance is joint goal accuracy, which represents the proportion of test set dialogue turns where all user goals expressed up to that point of the dialogue were decoded correctly (Henderson et al., 2014a). The NBT models for EN, DE and IT are trained using four variants of the SGNSLARGE vectors: 1) the initial distributional vectors; 2) morph-fixed; 3) and 4) the two variants of morphfitted vectors (see Sect. 3). Results and Discussion The diamond-dashed lines (against the right axes) in Fig. 2 show the DST performance of NBT models making use of the four vector collections. IT and DE benefit from both kinds of morph-fitting: IT performance increases 74.1→ 78.1 (MFIT-A) and DE performance rises even more: 60.6 → 66.3 (MFIT-AR), setting a new state-of-the-art score for both languages. The morph-fixed vectors do not enhance DST performance, probably because fixing word vectors to their highest frequency inflectional form eliminates useful semantic content encoded in the original vectors. On the other hand, morph-fitting makes use of this information, supplementing it with semantic relations between different morphological forms. These conclusions are in line with the SimLex and Morph-SimLex gains, where morph-fitting outper-\n8\n701\n702\n703\n704\n705\n706\n707\n708\n709\n710\n711\n712\n713\n714\n715\n716\n717\n718\n719\n720\n721\n722\n723\n724\n725\n726\n727\n728\n729\n730\n731\n732\n733\n734\n735\n736\n737\n738\n739\n740\n741\n742\n743\n744\n745\n746\n747\n748\n749\n750\n751\n752\n753\n754\n755\n756\n757\n758\n759\n760\n761\n762\n763\n764\n765\n766\n767\n768\n769\n770\n771\n772\n773\n774\n775\n776\n777\n778\n779\n780\n781\n782\n783\n784\n785\n786\n787\n788\n789\n790\n791\n792\n793\n794\n795\n796\n797\n798\n799\nforms distributional and morph-fixed vectors. English performance shows little variation across the four word vector collections investigated here. This corroborates our intuition that, as a morphologically simpler language, English stands to gain less from fine-tuning the morphological variation for downstream applications. This result again points at the discrepancy between intrinsic and extrinsic evaluation: the considerable gains in SimLex performance do not necessarily induce similar gains in downstream performance."
      }, {
        "heading" : "6 Related Work",
        "text" : "Semantic Specialisation A standard approach to incorporating external information into vector spaces is to pull the representations of similar words closer together. Some models integrate such constraints into the training procedure, modifying the prior or the regularisation (Yu and Dredze, 2014; Xu et al., 2014; Bian et al., 2014; Kiela et al., 2015), or using a variant of the SGNS-style objective (Liu et al., 2015; Osborne et al., 2016). Another class of models, popularly termed retrofitting, injects lexical knowledge from available semantic databases (e.g., WordNet, PPDB) into pre-trained word vectors (Faruqui et al., 2015; Jauhar et al., 2015; Wieting et al., 2015; Nguyen et al., 2016; Mrkšić et al., 2016). Morph-fitting falls into the latter category. However, instead of resorting to curated knowledge bases, and experimenting solely with English, we show that the morphological richness of any language can be exploited as a source of inexpensive supervision for fine-tuning vector spaces, at the same time specialising them to better reflect true semantic similarity.\nWord Vectors and Morphology The use of morphological resources to improve the representations of morphemes and words is an active area of research. The majority of proposed architectures encode morphological information, provided either as gold standard morphological resources (SylakGlassman et al., 2015) such as CELEX (Baayen et al., 1995) or as an external analyser such as Morfessor (Creutz and Lagus, 2007), along with distributional information jointly at training time in the language modelling (LM) objective (Luong et al., 2013; Botha and Blunsom, 2014; Qiu et al., 2014; Cotterell and Schütze, 2015; Bhatia et al., 2016, i.a.). The key idea is to learn a morphological composition function (Lazaridou et al., 2013; Cotterell and Schütze, 2017) which synthesises the\nrepresentation of a word given the representations of its constituent morphemes. Contrary to our work, these models typically coalesce all lexical relations.\nAnother class of models, operating at the character level, shares a similar methodology: such models compose token-level representations from subcomponent embeddings (subwords, morphemes, or characters) (dos Santos and Zadrozny, 2014; Ling et al., 2015; Cao and Rei, 2016; Kim et al., 2016; Wieting et al., 2016; Verwimp et al., 2017, i.a.).\nIn contrast to prior work, our model decouples the use of morphological information, now provided in the form of inflectional and derivational rules transformed into linguistic constraints, from the actual training. This pipelined approach results in a simpler, more portable model. In spirit, our work is similar to Cotterell et al. (2016), who formulate the idea of post-training specialisation in a generative Bayesian framework. Their work uses gold morphological lexicons; we show that competitive performance can be achieved using a nonexhaustive set of simple rules. Our framework facilitates the inclusion of antonyms at no extra cost and naturally extends to constraints from other sources (e.g., WordNet) in future work. Another practical difference is that we focus on similarity and evaluate morph-fitting in a well-defined downstream task where the artefacts of the distributional hypothesis are known to prompt statistical system failures."
      }, {
        "heading" : "7 Conclusion and Future Work",
        "text" : "We have presented a novel morph-fitting method which injects morphological knowledge in the form of linguistic constraints into word vector spaces. The method makes use of implicit semantic signals encoded in inflectional and derivational rules which describe the morphological processes in a language. The results in intrinsic word similarity tasks show that morph-fitting improves vector spaces induced by distributional models across four languages. Finally, we have shown that the use of morph-fitted vectors boosts the performance of downstream language understanding models which rely on word representations as features, especially for morphologically rich languages such as German.\nFuture work will focus on other potential sources of morphological knowledge (Soricut and Och, 2015), porting the framework to other morphologically rich languages and downstream tasks, and on further refinements of the post-processing algorithm and constraints selection.\n9\n801\n802\n803\n804\n805\n806\n807\n808\n809\n810\n811\n812\n813\n814\n815\n816\n817\n818\n819\n820\n821\n822\n823\n824\n825\n826\n827\n828\n829\n830\n831\n832\n833\n834\n835\n836\n837\n838\n839\n840\n841\n842\n843\n844\n845\n846\n847\n848\n849\n850\n851\n852\n853\n854\n855\n856\n857\n858\n859\n860\n861\n862\n863\n864\n865\n866\n867\n868\n869\n870\n871\n872\n873\n874\n875\n876\n877\n878\n879\n880\n881\n882\n883\n884\n885\n886\n887\n888\n889\n890\n891\n892\n893\n894\n895\n896\n897\n898\n899"
      } ],
      "references" : [ {
        "title" : "Polyglot: Distributed word representations for multilingual NLP",
        "author" : [ "Rami Al-Rfou", "Bryan Perozzi", "Steven Skiena." ],
        "venue" : "Proceedings of CoNLL. pages 183–192. http://www.aclweb.org/anthology/W133520.",
        "citeRegEx" : "Al.Rfou et al\\.,? 2013",
        "shortCiteRegEx" : "Al.Rfou et al\\.",
        "year" : 2013
      }, {
        "title" : "Enriching morphologically poor languages for statistical machine translation",
        "author" : [ "Eleftherios Avramidis", "Philipp Koehn." ],
        "venue" : "Proceedings of ACL. pages 763–770. http://www.aclweb.org/anthology/P/P08/P08-1087.",
        "citeRegEx" : "Avramidis and Koehn.,? 2008",
        "shortCiteRegEx" : "Avramidis and Koehn.",
        "year" : 2008
      }, {
        "title" : "The CELEX lexical data base on CD-ROM",
        "author" : [ "Harald R. Baayen", "Richard Piepenbrock", "Hedderik van Rijn" ],
        "venue" : null,
        "citeRegEx" : "Baayen et al\\.,? \\Q1995\\E",
        "shortCiteRegEx" : "Baayen et al\\.",
        "year" : 1995
      }, {
        "title" : "Tailoring continuous word representations for dependency parsing",
        "author" : [ "Mohit Bansal", "Kevin Gimpel", "Karen Livescu." ],
        "venue" : "Proceedings of ACL. pages 809– 815. http://www.aclweb.org/anthology/P14-2131.",
        "citeRegEx" : "Bansal et al\\.,? 2014",
        "shortCiteRegEx" : "Bansal et al\\.",
        "year" : 2014
      }, {
        "title" : "Don’t count, predict! A systematic comparison of contextcounting vs",
        "author" : [ "Marco Baroni", "Georgiana Dinu", "Germán Kruszewski." ],
        "venue" : "context-predicting semantic vectors. In Proceedings of ACL. pages 238–247.",
        "citeRegEx" : "Baroni et al\\.,? 2014",
        "shortCiteRegEx" : "Baroni et al\\.",
        "year" : 2014
      }, {
        "title" : "Morphological priors for probabilistic neural word embeddings",
        "author" : [ "Parminder Bhatia", "Robert Guthrie", "Jacob Eisenstein." ],
        "venue" : "Proceedings of EMNLP. pages 490–500. https://aclweb.org/anthology/D16-1047.",
        "citeRegEx" : "Bhatia et al\\.,? 2016",
        "shortCiteRegEx" : "Bhatia et al\\.",
        "year" : 2016
      }, {
        "title" : "Knowledge-powered deep learning for word embedding",
        "author" : [ "Jiang Bian", "Bin Gao", "Tie-Yan Liu." ],
        "venue" : "Proceedings of ECML-PKDD. pages 132– 148. https://doi.org/10.1007/978-3-662-44848-9_9.",
        "citeRegEx" : "Bian et al\\.,? 2014",
        "shortCiteRegEx" : "Bian et al\\.",
        "year" : 2014
      }, {
        "title" : "Compositional morphology for word representations and language modelling",
        "author" : [ "Jan A. Botha", "Phil Blunsom." ],
        "venue" : "Proceedings of ICML. pages 1899–1907. http://jmlr.org/proceedings/papers/v32/botha14.html.",
        "citeRegEx" : "Botha and Blunsom.,? 2014",
        "shortCiteRegEx" : "Botha and Blunsom.",
        "year" : 2014
      }, {
        "title" : "Multimodal distributional semantics",
        "author" : [ "Elia Bruni", "Nam-Khanh Tran", "Marco Baroni." ],
        "venue" : "Journal of Artificial Intelligence Research 49:1–47. https://doi.org/10.1613/jair.4135.",
        "citeRegEx" : "Bruni et al\\.,? 2014",
        "shortCiteRegEx" : "Bruni et al\\.",
        "year" : 2014
      }, {
        "title" : "A joint model for word embedding and word morphology",
        "author" : [ "Kris Cao", "Marek Rei." ],
        "venue" : "Proceedings of the 1st Workshop on Representation Learning for NLP. pages 18–26. http://aclweb.org/anthology/W/W16/W16-1603.",
        "citeRegEx" : "Cao and Rei.,? 2016",
        "shortCiteRegEx" : "Cao and Rei.",
        "year" : 2016
      }, {
        "title" : "A fast and accurate dependency parser using neural networks",
        "author" : [ "Danqi Chen", "Christopher D. Manning." ],
        "venue" : "Proceedings of EMNLP. pages 740–750. http://www.aclweb.org/anthology/D14-1082.",
        "citeRegEx" : "Chen and Manning.,? 2014",
        "shortCiteRegEx" : "Chen and Manning.",
        "year" : 2014
      }, {
        "title" : "Natural language processing (almost) from scratch",
        "author" : [ "Pavel P. Kuksa." ],
        "venue" : "Journal of Machine Learning Research 12:2493–2537. http://dl.acm.org/citation.cfm?id=1953048.2078186.",
        "citeRegEx" : "Kuksa.,? 2011",
        "shortCiteRegEx" : "Kuksa.",
        "year" : 2011
      }, {
        "title" : "Morphological word-embeddings",
        "author" : [ "Ryan Cotterell", "Hinrich Schütze." ],
        "venue" : "Proceedings of NAACL-HLT . pages 1287–1292. http://www.aclweb.org/anthology/N15-1140.",
        "citeRegEx" : "Cotterell and Schütze.,? 2015",
        "shortCiteRegEx" : "Cotterell and Schütze.",
        "year" : 2015
      }, {
        "title" : "Joint semantic synthesis and morphological analysis of the derived word",
        "author" : [ "Ryan Cotterell", "Hinrich Schütze." ],
        "venue" : "Transactions of the ACL (to appear) https://arxiv.org/abs/1701.00946.",
        "citeRegEx" : "Cotterell and Schütze.,? 2017",
        "shortCiteRegEx" : "Cotterell and Schütze.",
        "year" : 2017
      }, {
        "title" : "Morphological smoothing and extrapolation of word embeddings",
        "author" : [ "Ryan Cotterell", "Hinrich Schütze", "Jason Eisner." ],
        "venue" : "Proceedings of ACL. pages 1651–1660. http://www.aclweb.org/anthology/P161156.",
        "citeRegEx" : "Cotterell et al\\.,? 2016",
        "shortCiteRegEx" : "Cotterell et al\\.",
        "year" : 2016
      }, {
        "title" : "Unsupervised models for morpheme segmentation and morphology learning",
        "author" : [ "Mathias Creutz", "Krista Lagus." ],
        "venue" : "TSLP 4(1):3:1–3:34. http://doi.acm.org/10.1145/1217098.1217101.",
        "citeRegEx" : "Creutz and Lagus.,? 2007",
        "shortCiteRegEx" : "Creutz and Lagus.",
        "year" : 2007
      }, {
        "title" : "From Distributional to Semantic Similarity",
        "author" : [ "James Curran." ],
        "venue" : "Ph.D. thesis, School of Informatics, University of Edinburgh. http://hdl.handle.net/1842/563.",
        "citeRegEx" : "Curran.,? 2004",
        "shortCiteRegEx" : "Curran.",
        "year" : 2004
      }, {
        "title" : "Improving zero-shot learning by mitigating the hubness problem",
        "author" : [ "Georgiana Dinu", "Angeliki Lazaridou", "Marco Baroni." ],
        "venue" : "Proceedings of ICLR (Workshop Papers). http://arxiv.org/abs/1412.6568.",
        "citeRegEx" : "Dinu et al\\.,? 2015",
        "shortCiteRegEx" : "Dinu et al\\.",
        "year" : 2015
      }, {
        "title" : "Learning character-level representations for part-of-speech tagging",
        "author" : [ "Cícero Nogueira dos Santos", "Bianca Zadrozny." ],
        "venue" : "Proceedings of ICML. pages 1818–1826. http://jmlr.org/proceedings/papers/v32/santos14.html.",
        "citeRegEx" : "Santos and Zadrozny.,? 2014",
        "shortCiteRegEx" : "Santos and Zadrozny.",
        "year" : 2014
      }, {
        "title" : "Adaptive subgradient methods for online learning and stochastic optimization",
        "author" : [ "John C. Duchi", "Elad Hazan", "Yoram Singer." ],
        "venue" : "Journal of Machine Learning Research 12:2121–2159. http://dl.acm.org/citation.cfm?id=2021068.",
        "citeRegEx" : "Duchi et al\\.,? 2011",
        "shortCiteRegEx" : "Duchi et al\\.",
        "year" : 2011
      }, {
        "title" : "Representing multilingual data as linked data: The case of BabelNet 2.0",
        "author" : [ "Maud Ehrmann", "Francesco Cecconi", "Daniele Vannella", "John Philip Mccrae", "Philipp Cimiano", "Roberto Navigli" ],
        "venue" : "In Proceedings of LREC",
        "citeRegEx" : "Ehrmann et al\\.,? \\Q2014\\E",
        "shortCiteRegEx" : "Ehrmann et al\\.",
        "year" : 2014
      }, {
        "title" : "Retrofitting word vectors to semantic lexicons",
        "author" : [ "Manaal Faruqui", "Jesse Dodge", "Sujay Kumar Jauhar", "Chris Dyer", "Eduard Hovy", "Noah A. Smith." ],
        "venue" : "Proceedings of NAACL-HLT . pages 1606– 1615. http://www.aclweb.org/anthology/N15-1184.",
        "citeRegEx" : "Faruqui et al\\.,? 2015",
        "shortCiteRegEx" : "Faruqui et al\\.",
        "year" : 2015
      }, {
        "title" : "Improving vector space word representations using multilingual correlation",
        "author" : [ "Manaal Faruqui", "Chris Dyer." ],
        "venue" : "Proceedings of EACL. pages 462– 471. http://www.aclweb.org/anthology/E14-1049.",
        "citeRegEx" : "Faruqui and Dyer.,? 2014",
        "shortCiteRegEx" : "Faruqui and Dyer.",
        "year" : 2014
      }, {
        "title" : "Placing search in context: The concept revisited",
        "author" : [ "Lev Finkelstein", "Evgeniy Gabrilovich", "Yossi Matias", "Ehud Rivlin", "Zach Solan", "Gadi Wolfman", "Eytan Ruppin." ],
        "venue" : "ACM Transactions on Information Systems 20(1):116–131.",
        "citeRegEx" : "Finkelstein et al\\.,? 2002",
        "shortCiteRegEx" : "Finkelstein et al\\.",
        "year" : 2002
      }, {
        "title" : "An Introduction to Language, 10th Edition",
        "author" : [ "Victoria Fromkin", "Robert Rodman", "Nina Hyams" ],
        "venue" : null,
        "citeRegEx" : "Fromkin et al\\.,? \\Q2013\\E",
        "shortCiteRegEx" : "Fromkin et al\\.",
        "year" : 2013
      }, {
        "title" : "PPDB: The Paraphrase Database",
        "author" : [ "Juri Ganitkevitch", "Benjamin Van Durme", "Chris Callison-Burch." ],
        "venue" : "Proceedings of NAACL-HLT . pages 758–764. http://www.aclweb.org/anthology/N131092.",
        "citeRegEx" : "Ganitkevitch et al\\.,? 2013",
        "shortCiteRegEx" : "Ganitkevitch et al\\.",
        "year" : 2013
      }, {
        "title" : "SimVerb3500: A large-scale evaluation set of verb similarity",
        "author" : [ "Daniela Gerz", "Ivan Vulić", "Felix Hill", "Roi Reichart", "Anna Korhonen." ],
        "venue" : "Proceedings of EMNLP. pages 2173–2182. https://aclweb.org/anthology/D16-1235.",
        "citeRegEx" : "Gerz et al\\.,? 2016",
        "shortCiteRegEx" : "Gerz et al\\.",
        "year" : 2016
      }, {
        "title" : "Understanding morphology",
        "author" : [ "Martin Haspelmath", "Andrea Sims" ],
        "venue" : null,
        "citeRegEx" : "Haspelmath and Sims.,? \\Q2013\\E",
        "shortCiteRegEx" : "Haspelmath and Sims.",
        "year" : 2013
      }, {
        "title" : "The Second Dialog State Tracking Challenge",
        "author" : [ "Matthew Henderson", "Blaise Thomson", "Jason D. Wiliams." ],
        "venue" : "Proceedings of SIGDIAL. pages 263– 272. http://aclweb.org/anthology/W/W14/W144337.pdf.",
        "citeRegEx" : "Henderson et al\\.,? 2014a",
        "shortCiteRegEx" : "Henderson et al\\.",
        "year" : 2014
      }, {
        "title" : "The Third Dialog State Tracking Challenge",
        "author" : [ "Matthew Henderson", "Blaise Thomson", "Jason D. Wiliams." ],
        "venue" : "Proceedings of IEEE SLT . pages 324– 329. https://doi.org/10.1109/SLT.2014.7078595.",
        "citeRegEx" : "Henderson et al\\.,? 2014b",
        "shortCiteRegEx" : "Henderson et al\\.",
        "year" : 2014
      }, {
        "title" : "Word-based dialog state tracking with recurrent neural networks",
        "author" : [ "Matthew Henderson", "Blaise Thomson", "Steve Young." ],
        "venue" : "Proceedings of SIGDIAL. pages 292–299. http://aclweb.org/anthology/W/W14/W14-",
        "citeRegEx" : "Henderson et al\\.,? 2014c",
        "shortCiteRegEx" : "Henderson et al\\.",
        "year" : 2014
      }, {
        "title" : "SimLex-999: Evaluating semantic models with (genuine) similarity estimation",
        "author" : [ "Felix Hill", "Roi Reichart", "Anna Korhonen." ],
        "venue" : "Computational Linguistics 41(4):665–695. https://doi.org/10.1162/COLI_a_00237.",
        "citeRegEx" : "Hill et al\\.,? 2015",
        "shortCiteRegEx" : "Hill et al\\.",
        "year" : 2015
      }, {
        "title" : "Ontologically grounded multi-sense representation learning for semantic vector space models",
        "author" : [ "Sujay Kumar Jauhar", "Chris Dyer", "Eduard H. Hovy." ],
        "venue" : "Proceedings of NAACL. pages 683–693. http://www.aclweb.org/anthology/N15-1070.",
        "citeRegEx" : "Jauhar et al\\.,? 2015",
        "shortCiteRegEx" : "Jauhar et al\\.",
        "year" : 2015
      }, {
        "title" : "Specializing word embeddings for similarity or relatedness",
        "author" : [ "Douwe Kiela", "Felix Hill", "Stephen Clark." ],
        "venue" : "Proceedings of EMNLP. pages 2044– 2048. http://aclweb.org/anthology/D15-1242.",
        "citeRegEx" : "Kiela et al\\.,? 2015",
        "shortCiteRegEx" : "Kiela et al\\.",
        "year" : 2015
      }, {
        "title" : "Character-aware neural language models",
        "author" : [ "Yoon Kim", "Yacine Jernite", "David Sontag", "Alexander M. Rush." ],
        "venue" : "Proceedings of AAAI. pages 2741– 2749.",
        "citeRegEx" : "Kim et al\\.,? 2016",
        "shortCiteRegEx" : "Kim et al\\.",
        "year" : 2016
      }, {
        "title" : "Compositionally derived representations of morphologically complex words in distributional semantics",
        "author" : [ "Angeliki Lazaridou", "Marco Marelli", "Roberto Zamparelli", "Marco Baroni." ],
        "venue" : "Proceedings of ACL. pages 1517–1526.",
        "citeRegEx" : "Lazaridou et al\\.,? 2013",
        "shortCiteRegEx" : "Lazaridou et al\\.",
        "year" : 2013
      }, {
        "title" : "Recipe for building robust spoken dialog state trackers: Dialog State Tracking Challenge system description",
        "author" : [ "Sungjin Lee", "Maxine Eskenazi." ],
        "venue" : "Proceedings of SIGDIAL. pages 414– 422. http://aclweb.org/anthology/W/W13/W13-",
        "citeRegEx" : "Lee and Eskenazi.,? 2013",
        "shortCiteRegEx" : "Lee and Eskenazi.",
        "year" : 2013
      }, {
        "title" : "Separated by an un-common language: Towards judgment language informed vector space modeling",
        "author" : [ "Ira Leviant", "Roi Reichart." ],
        "venue" : "CoRR abs/1508.00106. http://arxiv.org/abs/1508.00106.",
        "citeRegEx" : "Leviant and Reichart.,? 2015",
        "shortCiteRegEx" : "Leviant and Reichart.",
        "year" : 2015
      }, {
        "title" : "Dependency-based word embeddings",
        "author" : [ "Omer Levy", "Yoav Goldberg." ],
        "venue" : "Proceedings of ACL. pages 302–308. http://www.aclweb.org/anthology/P14-2050.",
        "citeRegEx" : "Levy and Goldberg.,? 2014",
        "shortCiteRegEx" : "Levy and Goldberg.",
        "year" : 2014
      }, {
        "title" : "Finding function in form: Compositional character models for open vocabulary word representation",
        "author" : [ "Wang Ling", "Chris Dyer", "Alan W. Black", "Isabel Trancoso", "Ramon Fermandez", "Silvio Amir", "Luis Marujo", "Tiago Luis." ],
        "venue" : "Proceedings of EMNLP.",
        "citeRegEx" : "Ling et al\\.,? 2015",
        "shortCiteRegEx" : "Ling et al\\.",
        "year" : 2015
      }, {
        "title" : "Gated end-to-end memory networks",
        "author" : [ "Fei Liu", "Julien Perez." ],
        "venue" : "Proceedings of EACL (to appear). http://arxiv.org/abs/1610.04211.",
        "citeRegEx" : "Liu and Perez.,? 2017",
        "shortCiteRegEx" : "Liu and Perez.",
        "year" : 2017
      }, {
        "title" : "Learning semantic word embeddings based on ordinal knowledge constraints",
        "author" : [ "Quan Liu", "Hui Jiang", "Si Wei", "Zhen-Hua Ling", "Yu Hu." ],
        "venue" : "Proceedings of ACL. pages 1501–1511. http://www.aclweb.org/anthology/P15-1145.",
        "citeRegEx" : "Liu et al\\.,? 2015",
        "shortCiteRegEx" : "Liu et al\\.",
        "year" : 2015
      }, {
        "title" : "Bilingual word representations with monolingual quality in mind",
        "author" : [ "Thang Luong", "Hieu Pham", "Christopher D. Manning." ],
        "venue" : "Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing. pages 151–159.",
        "citeRegEx" : "Luong et al\\.,? 2015",
        "shortCiteRegEx" : "Luong et al\\.",
        "year" : 2015
      }, {
        "title" : "Better word representations with recursive neural networks for morphology",
        "author" : [ "Thang Luong", "Richard Socher", "Christopher Manning." ],
        "venue" : "Proceedings of CoNLL. pages 104–113. http://www.aclweb.org/anthology/W13-3512.",
        "citeRegEx" : "Luong et al\\.,? 2013",
        "shortCiteRegEx" : "Luong et al\\.",
        "year" : 2013
      }, {
        "title" : "Context2vec: Learning generic context embedding with bidirectional LSTM",
        "author" : [ "Oren Melamud", "Jacob Goldberger", "Ido Dagan." ],
        "venue" : "Proceedings of CoNLL. pages 51–61. http://aclweb.org/anthology/K/K16/K16-1006.pdf.",
        "citeRegEx" : "Melamud et al\\.,? 2016",
        "shortCiteRegEx" : "Melamud et al\\.",
        "year" : 2016
      }, {
        "title" : "Distributed representations of words and phrases and their compositionality",
        "author" : [ "Tomas Mikolov", "Ilya Sutskever", "Kai Chen", "Gregory S. Corrado", "Jeffrey Dean." ],
        "venue" : "Proceedings of NIPS. pages 3111–3119.",
        "citeRegEx" : "Mikolov et al\\.,? 2013",
        "shortCiteRegEx" : "Mikolov et al\\.",
        "year" : 2013
      }, {
        "title" : "Learning word embeddings efficiently with noise-contrastive estimation",
        "author" : [ "Andriy Mnih", "Koray Kavukcuoglu." ],
        "venue" : "Proceedings of NIPS. pages 2265– 2273.",
        "citeRegEx" : "Mnih and Kavukcuoglu.,? 2013",
        "shortCiteRegEx" : "Mnih and Kavukcuoglu.",
        "year" : 2013
      }, {
        "title" : "Multidomain dialog state tracking using recurrent neural networks",
        "author" : [ "Nikola Mrkšić", "Diarmuid Ó Séaghdha", "Blaise Thomson", "Milica Gašić", "Pei-Hao Su", "David Vandyke", "Tsung-Hsien Wen", "Steve Young." ],
        "venue" : "Proceedings of ACL. pages 794–799.",
        "citeRegEx" : "Mrkšić et al\\.,? 2015",
        "shortCiteRegEx" : "Mrkšić et al\\.",
        "year" : 2015
      }, {
        "title" : "Neural Belief Tracker: Data-driven dialogue state tracking",
        "author" : [ "Nikola Mrkšić", "Diarmuid Ó Séaghdha", "Blaise Thomson", "Tsung-Hsien Wen", "Steve Young." ],
        "venue" : "arXiv preprint: 1606.03777. http://arxiv.org/abs/1606.03777.",
        "citeRegEx" : "Mrkšić et al\\.,? 2016",
        "shortCiteRegEx" : "Mrkšić et al\\.",
        "year" : 2016
      }, {
        "title" : "Counter-fitting word vectors to linguistic constraints",
        "author" : [ "Nikola Mrkšić", "Diarmuid Ó Séaghdha", "Blaise Thomson", "Milica Gašić", "Lina Maria Rojas-Barahona", "Pei-Hao Su", "David Vandyke", "Tsung-Hsien Wen", "Steve Young." ],
        "venue" : "Proceedings of NAACL-",
        "citeRegEx" : "Mrkšić et al\\.,? 2016",
        "shortCiteRegEx" : "Mrkšić et al\\.",
        "year" : 2016
      }, {
        "title" : "Rectified linear units improve restricted Boltzmann machines",
        "author" : [ "Vinod Nair", "Geoffrey E. Hinton." ],
        "venue" : "Proceedings of ICML. pages 807–814. http://www.icml2010.org/papers/432.pdf.",
        "citeRegEx" : "Nair and Hinton.,? 2010",
        "shortCiteRegEx" : "Nair and Hinton.",
        "year" : 2010
      }, {
        "title" : "BabelNet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network",
        "author" : [ "Roberto Navigli", "Simone Paolo Ponzetto." ],
        "venue" : "Artificial Intelligence 193:217–250. https://doi.org/10.1016/j.artint.2012.07.001.",
        "citeRegEx" : "Navigli and Ponzetto.,? 2012",
        "shortCiteRegEx" : "Navigli and Ponzetto.",
        "year" : 2012
      }, {
        "title" : "Integrating distributional lexical contrast into word embeddings for antonymsynonym distinction",
        "author" : [ "Kim Anh Nguyen", "Sabine Schulte im Walde", "Ngoc Thang Vu." ],
        "venue" : "Proceedings of ACL. pages 454–459. http://anthology.aclweb.org/P16-2074.",
        "citeRegEx" : "Nguyen et al\\.,? 2016",
        "shortCiteRegEx" : "Nguyen et al\\.",
        "year" : 2016
      }, {
        "title" : "Encoding prior knowledge with eigenword embeddings",
        "author" : [ "Dominique Osborne", "Shashi Narayan", "Shay Cohen." ],
        "venue" : "Transactions of the ACL 4:417–430.",
        "citeRegEx" : "Osborne et al\\.,? 2016",
        "shortCiteRegEx" : "Osborne et al\\.",
        "year" : 2016
      }, {
        "title" : "PPDB 2.0: Better paraphrase ranking, finegrained entailment relations, word embeddings, and style classification",
        "author" : [ "Ellie Pavlick", "Pushpendre Rastogi", "Juri Ganitkevitch", "Benjamin Van Durme", "Chris Callison-Burch" ],
        "venue" : "In Proceedings of ACL",
        "citeRegEx" : "Pavlick et al\\.,? \\Q2015\\E",
        "shortCiteRegEx" : "Pavlick et al\\.",
        "year" : 2015
      }, {
        "title" : "Glove: Global vectors for word representation",
        "author" : [ "Jeffrey Pennington", "Richard Socher", "Christopher Manning." ],
        "venue" : "Proceedings of EMNLP. pages 1532– 1543. http://www.aclweb.org/anthology/D14-1162.",
        "citeRegEx" : "Pennington et al\\.,? 2014",
        "shortCiteRegEx" : "Pennington et al\\.",
        "year" : 2014
      }, {
        "title" : "Co-learning of word representations and morpheme representations",
        "author" : [ "Siyu Qiu", "Qing Cui", "Jiang Bian", "Bin Gao", "Tie-Yan Liu." ],
        "venue" : "Proceedings of COLING. pages 141–150. http://www.aclweb.org/anthology/C14-1015.",
        "citeRegEx" : "Qiu et al\\.,? 2014",
        "shortCiteRegEx" : "Qiu et al\\.",
        "year" : 2014
      }, {
        "title" : "Knowledge-free induction of inflectional morphologies",
        "author" : [ "Patrick Schone", "Daniel Jurafsky." ],
        "venue" : "Proceedings of NAACL. http://aclweb.org/anthology/N/N01/N01-1024.",
        "citeRegEx" : "Schone and Jurafsky.,? 2001",
        "shortCiteRegEx" : "Schone and Jurafsky.",
        "year" : 2001
      }, {
        "title" : "Symmetric pattern based word embeddings for improved word similarity prediction",
        "author" : [ "Roy Schwartz", "Roi Reichart", "Ari Rappoport." ],
        "venue" : "Proceedings of CoNLL. pages 258–267. http://www.aclweb.org/anthology/K15-1026.",
        "citeRegEx" : "Schwartz et al\\.,? 2015",
        "shortCiteRegEx" : "Schwartz et al\\.",
        "year" : 2015
      }, {
        "title" : "Symmetric patterns and coordinations: Fast and enhanced representations of verbs and adjectives",
        "author" : [ "Roy Schwartz", "Roi Reichart", "Ari Rappoport." ],
        "venue" : "Proceedings of NAACL-HLT . pages 499–505. http://www.aclweb.org/anthology/N16-1060.",
        "citeRegEx" : "Schwartz et al\\.,? 2016",
        "shortCiteRegEx" : "Schwartz et al\\.",
        "year" : 2016
      }, {
        "title" : "Unsupervised morphology induction using word embeddings",
        "author" : [ "Radu Soricut", "Franz Och." ],
        "venue" : "Proceedings of NAACL-HLT . pages 1627–1637. http://www.aclweb.org/anthology/N15-1186.",
        "citeRegEx" : "Soricut and Och.,? 2015",
        "shortCiteRegEx" : "Soricut and Och.",
        "year" : 2015
      }, {
        "title" : "A languageindependent feature schema for inflectional morphology",
        "author" : [ "John Sylak-Glassman", "Christo Kirov", "David Yarowsky", "Roger Que." ],
        "venue" : "Proceedings of ACL. pages 674–680. http://www.aclweb.org/anthology/P15-2111.",
        "citeRegEx" : "Sylak.Glassman et al\\.,? 2015",
        "shortCiteRegEx" : "Sylak.Glassman et al\\.",
        "year" : 2015
      }, {
        "title" : "Statistical parsing of morphologically rich languages (SPMRL) What, how and whither",
        "author" : [ "Reut Tsarfaty", "Djamé Seddah", "Yoav Goldberg", "Sandra Kuebler", "Yannick Versley", "Marie Candito", "Jennifer Foster", "Ines Rehbein", "Lamia Tounsi." ],
        "venue" : "Proceed-",
        "citeRegEx" : "Tsarfaty et al\\.,? 2010",
        "shortCiteRegEx" : "Tsarfaty et al\\.",
        "year" : 2010
      }, {
        "title" : "Word representations: A simple and general method for semi-supervised learning",
        "author" : [ "Joseph P. Turian", "Lev-Arie Ratinov", "Yoshua Bengio." ],
        "venue" : "Proceedings of ACL. pages 384–394. http://www.aclweb.org/anthology/P10-1040.",
        "citeRegEx" : "Turian et al\\.,? 2010",
        "shortCiteRegEx" : "Turian et al\\.",
        "year" : 2010
      }, {
        "title" : "From frequency to meaning: vector space models of semantics",
        "author" : [ "Peter D. Turney", "Patrick Pantel." ],
        "venue" : "Journal of Artifical Intelligence Research 37(1):141–188. https://doi.org/10.1613/jair.2934.",
        "citeRegEx" : "Turney and Pantel.,? 2010",
        "shortCiteRegEx" : "Turney and Pantel.",
        "year" : 2010
      }, {
        "title" : "Character-word LSTM language models",
        "author" : [ "Lyan Verwimp", "Joris Pelemans", "Hugo Van hamme", "Patrick Wambacq." ],
        "venue" : "Proceedings of EACL (to appear).",
        "citeRegEx" : "Verwimp et al\\.,? 2017",
        "shortCiteRegEx" : "Verwimp et al\\.",
        "year" : 2017
      }, {
        "title" : "Is \"universal syntax\" universally useful for learning distributed word representations? In Proceedings of ACL",
        "author" : [ "Ivan Vulić", "Anna Korhonen." ],
        "venue" : "pages 518–524. http://anthology.aclweb.org/P16-2084.",
        "citeRegEx" : "Vulić and Korhonen.,? 2016a",
        "shortCiteRegEx" : "Vulić and Korhonen.",
        "year" : 2016
      }, {
        "title" : "On the role of seed lexicons in learning bilingual word embeddings",
        "author" : [ "Ivan Vulić", "Anna Korhonen." ],
        "venue" : "Proceedings of ACL. pages 247–257. http://www.aclweb.org/anthology/P16-1024.",
        "citeRegEx" : "Vulić and Korhonen.,? 2016b",
        "shortCiteRegEx" : "Vulić and Korhonen.",
        "year" : 2016
      }, {
        "title" : "Knowledge graph embedding by translating on hyperplanes",
        "author" : [ "Zhen Wang", "Jianwen Zhang", "Jianlin Feng", "Zheng Chen." ],
        "venue" : "Proceedings of AAAI. pages 1112–1119.",
        "citeRegEx" : "Wang et al\\.,? 2014",
        "shortCiteRegEx" : "Wang et al\\.",
        "year" : 2014
      }, {
        "title" : "A networkbased end-to-end trainable task-oriented dialogue system",
        "author" : [ "Tsung-Hsien Wen", "David Vandyke", "Nikola Mrkšić", "Milica Gašić", "Lina M. Rojas-Barahona", "Pei-Hao Su", "Stefan Ultes", "Steve Young." ],
        "venue" : "Proceedings of EACL (to appear).",
        "citeRegEx" : "Wen et al\\.,? 2017",
        "shortCiteRegEx" : "Wen et al\\.",
        "year" : 2017
      }, {
        "title" : "From paraphrase database to compositional paraphrase model and back",
        "author" : [ "John Wieting", "Mohit Bansal", "Kevin Gimpel", "Karen Livescu." ],
        "venue" : "Transactions of the ACL 3:345–358.",
        "citeRegEx" : "Wieting et al\\.,? 2015",
        "shortCiteRegEx" : "Wieting et al\\.",
        "year" : 2015
      }, {
        "title" : "Charagram: Embedding words and sentences via character n-grams",
        "author" : [ "John Wieting", "Mohit Bansal", "Kevin Gimpel", "Karen Livescu." ],
        "venue" : "Proceedings of EMNLP. pages 1504–1515. https://aclweb.org/anthology/D16-1157.",
        "citeRegEx" : "Wieting et al\\.,? 2016",
        "shortCiteRegEx" : "Wieting et al\\.",
        "year" : 2016
      }, {
        "title" : "The Dialog State Tracking Challenge series: A review",
        "author" : [ "Jason D. Williams", "Antoine Raux", "Matthew Henderson." ],
        "venue" : "Dialogue & Discourse 7(3):4–33.",
        "citeRegEx" : "Williams et al\\.,? 2016",
        "shortCiteRegEx" : "Williams et al\\.",
        "year" : 2016
      }, {
        "title" : "RC-NET: A general framework for incorporating knowledge into word representations",
        "author" : [ "Chang Xu", "Yalong Bai", "Jiang Bian", "Bin Gao", "Gang Wang", "Xiaoguang Liu", "Tie-Yan Liu." ],
        "venue" : "Proceedings of CIKM. pages 1219–1228.",
        "citeRegEx" : "Xu et al\\.,? 2014",
        "shortCiteRegEx" : "Xu et al\\.",
        "year" : 2014
      }, {
        "title" : "Cognitive User Interfaces",
        "author" : [ "Steve Young." ],
        "venue" : "IEEE Signal Processing Magazine .",
        "citeRegEx" : "Young.,? 2010",
        "shortCiteRegEx" : "Young.",
        "year" : 2010
      }, {
        "title" : "Improving lexical embeddings with semantic knowledge",
        "author" : [ "Mo Yu", "Mark Dredze." ],
        "venue" : "Proceedings of ACL. pages 545–550. http://www.aclweb.org/anthology/P14-2089.",
        "citeRegEx" : "Yu and Dredze.,? 2014",
        "shortCiteRegEx" : "Yu and Dredze.",
        "year" : 2014
      }, {
        "title" : "DErivBase: Inducing and evaluating a derivational morphology resource for German",
        "author" : [ "Britta Zeller", "Jan Šnajder", "Sebastian Padó." ],
        "venue" : "Proceedings of ACL. pages 1201–1211. http://www.aclweb.org/anthology/P13-1118.",
        "citeRegEx" : "Zeller et al\\.,? 2013",
        "shortCiteRegEx" : "Zeller et al\\.",
        "year" : 2013
      }, {
        "title" : "Bilingual word embeddings for phrase-based machine translation",
        "author" : [ "Will Y. Zou", "Richard Socher", "Daniel Cer", "Christopher D. Manning." ],
        "venue" : "Proceedings of EMNLP. pages 1393–1398. http://www.aclweb.org/anthology/D13-1141.",
        "citeRegEx" : "Zou et al\\.,? 2013",
        "shortCiteRegEx" : "Zou et al\\.",
        "year" : 2013
      } ],
      "referenceMentions" : [ {
        "referenceID" : 10,
        "context" : "Word representation learning has become a research area of central importance in natural language processing (NLP), with its usefulness demonstrated across many application areas such as parsing (Chen and Manning, 2014), machine translation (Zou et al.",
        "startOffset" : 195,
        "endOffset" : 219
      }, {
        "referenceID" : 77,
        "context" : "Word representation learning has become a research area of central importance in natural language processing (NLP), with its usefulness demonstrated across many application areas such as parsing (Chen and Manning, 2014), machine translation (Zou et al., 2013), and many others (Turian et al.",
        "startOffset" : 241,
        "endOffset" : 259
      }, {
        "referenceID" : 63,
        "context" : ", 2013), and many others (Turian et al., 2010; Collobert et al., 2011).",
        "startOffset" : 25,
        "endOffset" : 70
      }, {
        "referenceID" : 62,
        "context" : "is expressed at word level” (Tsarfaty et al., 2010), pose specific challenges for NLP.",
        "startOffset" : 28,
        "endOffset" : 51
      }, {
        "referenceID" : 21,
        "context" : "Formalised as an instance of the post-processing semantic specialisation paradigm (Faruqui et al., 2015; Mrkšić et al., 2016), morphfitting is steered by a set of linguistic constraints derived from simple language-specific rules which describe (a subset of) morphological processes in a language.",
        "startOffset" : 82,
        "endOffset" : 125
      }, {
        "referenceID" : 48,
        "context" : "Formalised as an instance of the post-processing semantic specialisation paradigm (Faruqui et al., 2015; Mrkšić et al., 2016), morphfitting is steered by a set of linguistic constraints derived from simple language-specific rules which describe (a subset of) morphological processes in a language.",
        "startOffset" : 82,
        "endOffset" : 125
      }, {
        "referenceID" : 31,
        "context" : "sian), yielding large and consistent improvements on benchmarking word similarity evaluation sets such as SimLex-999 (Hill et al., 2015), its multilingual extension (Leviant and Reichart, 2015), and SimVerb-3500 (Gerz et al.",
        "startOffset" : 117,
        "endOffset" : 136
      }, {
        "referenceID" : 37,
        "context" : ", 2015), its multilingual extension (Leviant and Reichart, 2015), and SimVerb-3500 (Gerz et al.",
        "startOffset" : 36,
        "endOffset" : 64
      }, {
        "referenceID" : 26,
        "context" : ", 2015), its multilingual extension (Leviant and Reichart, 2015), and SimVerb-3500 (Gerz et al., 2016).",
        "startOffset" : 83,
        "endOffset" : 102
      }, {
        "referenceID" : 70,
        "context" : "1 The ATTRACT-REPEL Model The ATTRACT-REPEL model is an extension of PARAGRAM, proposed by Wieting et al. (2015). It provides a generic framework for incorporating similarity (e.",
        "startOffset" : 91,
        "endOffset" : 113
      }, {
        "referenceID" : 50,
        "context" : "ReLU(x) = max(0, x) is the standard rectified linear unit (Nair and Hinton, 2010).",
        "startOffset" : 58,
        "endOffset" : 81
      }, {
        "referenceID" : 25,
        "context" : "These can be extracted from a variety of semantic databases such as WordNet (Fellbaum, 1998), the Paraphrase Database (Ganitkevitch et al., 2013; Pavlick et al., 2015), or BabelNet (Navigli and Ponzetto, 2012; Ehrmann et al.",
        "startOffset" : 118,
        "endOffset" : 167
      }, {
        "referenceID" : 54,
        "context" : "These can be extracted from a variety of semantic databases such as WordNet (Fellbaum, 1998), the Paraphrase Database (Ganitkevitch et al., 2013; Pavlick et al., 2015), or BabelNet (Navigli and Ponzetto, 2012; Ehrmann et al.",
        "startOffset" : 118,
        "endOffset" : 167
      }, {
        "referenceID" : 51,
        "context" : ", 2015), or BabelNet (Navigli and Ponzetto, 2012; Ehrmann et al., 2014) as done in prior work (Faruqui et al.",
        "startOffset" : 21,
        "endOffset" : 71
      }, {
        "referenceID" : 20,
        "context" : ", 2015), or BabelNet (Navigli and Ponzetto, 2012; Ehrmann et al., 2014) as done in prior work (Faruqui et al.",
        "startOffset" : 21,
        "endOffset" : 71
      }, {
        "referenceID" : 19,
        "context" : "We train all models for 10 epochs with AdaGrad (Duchi et al., 2011).",
        "startOffset" : 47,
        "endOffset" : 67
      }, {
        "referenceID" : 1,
        "context" : "We define two rules for English, widely recognised as morphologically simple (Avramidis and Koehn, 2008; Cotterell et al., 2016).",
        "startOffset" : 77,
        "endOffset" : 128
      }, {
        "referenceID" : 14,
        "context" : "We define two rules for English, widely recognised as morphologically simple (Avramidis and Koehn, 2008; Cotterell et al., 2016).",
        "startOffset" : 77,
        "endOffset" : 128
      }, {
        "referenceID" : 57,
        "context" : "On the other hand, the latter refers to the formation of new words with semantic shifts in meaning (Schone and Jurafsky, 2001; Haspelmath and Sims, 2013; Lazaridou et al., 2013; Zeller et al., 2013; Cotterell and Schütze, 2017).",
        "startOffset" : 99,
        "endOffset" : 227
      }, {
        "referenceID" : 27,
        "context" : "On the other hand, the latter refers to the formation of new words with semantic shifts in meaning (Schone and Jurafsky, 2001; Haspelmath and Sims, 2013; Lazaridou et al., 2013; Zeller et al., 2013; Cotterell and Schütze, 2017).",
        "startOffset" : 99,
        "endOffset" : 227
      }, {
        "referenceID" : 35,
        "context" : "On the other hand, the latter refers to the formation of new words with semantic shifts in meaning (Schone and Jurafsky, 2001; Haspelmath and Sims, 2013; Lazaridou et al., 2013; Zeller et al., 2013; Cotterell and Schütze, 2017).",
        "startOffset" : 99,
        "endOffset" : 227
      }, {
        "referenceID" : 76,
        "context" : "On the other hand, the latter refers to the formation of new words with semantic shifts in meaning (Schone and Jurafsky, 2001; Haspelmath and Sims, 2013; Lazaridou et al., 2013; Zeller et al., 2013; Cotterell and Schütze, 2017).",
        "startOffset" : 99,
        "endOffset" : 227
      }, {
        "referenceID" : 13,
        "context" : "On the other hand, the latter refers to the formation of new words with semantic shifts in meaning (Schone and Jurafsky, 2001; Haspelmath and Sims, 2013; Lazaridou et al., 2013; Zeller et al., 2013; Cotterell and Schütze, 2017).",
        "startOffset" : 99,
        "endOffset" : 227
      }, {
        "referenceID" : 24,
        "context" : "We use a standard set of EN “antonymy” prefixes: APen = {dis, il, un, in, im, ir, mis, non, anti} (Fromkin et al., 2013).",
        "startOffset" : 98,
        "endOffset" : 120
      }, {
        "referenceID" : 45,
        "context" : "Training Data and Setup For each of the four languages we train the skip-gram with negative sampling (SGNS) model (Mikolov et al., 2013) on the latest Wikipedia dump of each language.",
        "startOffset" : 114,
        "endOffset" : 136
      }, {
        "referenceID" : 55,
        "context" : "We test standard distributional models: Common-Crawl GloVe (Pennington et al., 2014), SGNS vectors (Mikolov et al.",
        "startOffset" : 59,
        "endOffset" : 84
      }, {
        "referenceID" : 45,
        "context" : ", 2014), SGNS vectors (Mikolov et al., 2013) with various contexts (BOW = bag-ofwords; DEPS = dependency contexts), and training data (PW = Polyglot Wikipedia from Al-Rfou",
        "startOffset" : 22,
        "endOffset" : 44
      }, {
        "referenceID" : 4,
        "context" : "Other SGNS parameters were set to standard values (Baroni et al., 2014; Vulić and Korhonen, 2016b): 15 epochs, 15 negative samples, global learning rate: .",
        "startOffset" : 50,
        "endOffset" : 98
      }, {
        "referenceID" : 67,
        "context" : "Other SGNS parameters were set to standard values (Baroni et al., 2014; Vulić and Korhonen, 2016b): 15 epochs, 15 negative samples, global learning rate: .",
        "startOffset" : 50,
        "endOffset" : 98
      }, {
        "referenceID" : 38,
        "context" : "(2013); 8B = 8 billion token word2vec corpus), following (Levy and Goldberg, 2014) and (Schwartz et al.",
        "startOffset" : 57,
        "endOffset" : 82
      }, {
        "referenceID" : 58,
        "context" : "(2013); 8B = 8 billion token word2vec corpus), following (Levy and Goldberg, 2014) and (Schwartz et al., 2015).",
        "startOffset" : 87,
        "endOffset" : 110
      }, {
        "referenceID" : 4,
        "context" : "(2016) (SymPat-Emb), count-based PMI-weighted vectors reduced by SVD (Baroni et al., 2014) (Count-SVD), a model which replaces the context modelling function from CBOW with bidirectional LSTMs (Melamud et al.",
        "startOffset" : 69,
        "endOffset" : 90
      }, {
        "referenceID" : 44,
        "context" : ", 2014) (Count-SVD), a model which replaces the context modelling function from CBOW with bidirectional LSTMs (Melamud et al., 2016) (Context2Vec), and two sets of EN vectors trained by injecting multilingual information: BiSkip (Luong et al.",
        "startOffset" : 110,
        "endOffset" : 132
      }, {
        "referenceID" : 42,
        "context" : ", 2016) (Context2Vec), and two sets of EN vectors trained by injecting multilingual information: BiSkip (Luong et al., 2015) and MultiCCA (Faruqui and Dyer, 2014).",
        "startOffset" : 104,
        "endOffset" : 124
      }, {
        "referenceID" : 22,
        "context" : ", 2015) and MultiCCA (Faruqui and Dyer, 2014).",
        "startOffset" : 21,
        "endOffset" : 45
      }, {
        "referenceID" : 17,
        "context" : "We also experiment with a selection of standard distributional spaces in other languages from prior work (Dinu et al., 2015; Luong et al., 2015; Vulić and Korhonen, 2016a).",
        "startOffset" : 105,
        "endOffset" : 171
      }, {
        "referenceID" : 42,
        "context" : "We also experiment with a selection of standard distributional spaces in other languages from prior work (Dinu et al., 2015; Luong et al., 2015; Vulić and Korhonen, 2016a).",
        "startOffset" : 105,
        "endOffset" : 171
      }, {
        "referenceID" : 66,
        "context" : "We also experiment with a selection of standard distributional spaces in other languages from prior work (Dinu et al., 2015; Luong et al., 2015; Vulić and Korhonen, 2016a).",
        "startOffset" : 105,
        "endOffset" : 171
      }, {
        "referenceID" : 35,
        "context" : "(2013); 8B = 8 billion token word2vec corpus), following (Levy and Goldberg, 2014) and (Schwartz et al., 2015). We also test the symmetricpattern based vectors of Schwartz et al. (2016) (SymPat-Emb), count-based PMI-weighted vectors reduced by SVD (Baroni et al.",
        "startOffset" : 58,
        "endOffset" : 186
      }, {
        "referenceID" : 48,
        "context" : "We also tried using another post-processing model (Mrkšić et al., 2016) in lieu of ATTRACT-REPEL.",
        "startOffset" : 50,
        "endOffset" : 71
      }, {
        "referenceID" : 37,
        "context" : "8 SimLex-999 was translated to DE, IT, and RU by Leviant and Reichart (2015), and they crowdsourced similarity scores from native speakers.",
        "startOffset" : 49,
        "endOffset" : 77
      }, {
        "referenceID" : 23,
        "context" : "Unlike other gold standard resources such as WordSim353 (Finkelstein et al., 2002) or MEN (Bruni et al.",
        "startOffset" : 56,
        "endOffset" : 82
      }, {
        "referenceID" : 8,
        "context" : ", 2002) or MEN (Bruni et al., 2014), SimLex and SimVerb provided explicit guidelines to discern between semantic similarity and association, so that related but non-similar words (e.",
        "startOffset" : 15,
        "endOffset" : 35
      }, {
        "referenceID" : 8,
        "context" : ", 2002) or MEN (Bruni et al., 2014), SimLex and SimVerb provided explicit guidelines to discern between semantic similarity and association, so that related but non-similar words (e.g. cup and coffee) have a low rating. Since Leviant and Reichart (2015) re-scored the original EN SimLex, we use their EN SimLex version for consistency.",
        "startOffset" : 16,
        "endOffset" : 254
      }, {
        "referenceID" : 45,
        "context" : "SG-BOW2-PW (300) (Mikolov et al., 2013) .",
        "startOffset" : 17,
        "endOffset" : 39
      }, {
        "referenceID" : 55,
        "context" : "GloVe-6B (300) (Pennington et al., 2014) .",
        "startOffset" : 15,
        "endOffset" : 40
      }, {
        "referenceID" : 4,
        "context" : "Count-SVD (500) (Baroni et al., 2014) .",
        "startOffset" : 16,
        "endOffset" : 37
      }, {
        "referenceID" : 38,
        "context" : "SG-DEPS-PW (300) (Levy and Goldberg, 2014) .",
        "startOffset" : 17,
        "endOffset" : 42
      }, {
        "referenceID" : 3,
        "context" : "SG-DEPS-8B (500) (Bansal et al., 2014) .",
        "startOffset" : 17,
        "endOffset" : 38
      }, {
        "referenceID" : 22,
        "context" : "MultiCCA-EN (512) (Faruqui and Dyer, 2014) .",
        "startOffset" : 18,
        "endOffset" : 42
      }, {
        "referenceID" : 42,
        "context" : "BiSkip-EN (256) (Luong et al., 2015) .",
        "startOffset" : 16,
        "endOffset" : 36
      }, {
        "referenceID" : 58,
        "context" : "SG-BOW2-8B (500) (Schwartz et al., 2015) .",
        "startOffset" : 17,
        "endOffset" : 40
      }, {
        "referenceID" : 59,
        "context" : "SymPat-Emb (500) (Schwartz et al., 2016) .",
        "startOffset" : 17,
        "endOffset" : 40
      }, {
        "referenceID" : 44,
        "context" : "Context2Vec (600) (Melamud et al., 2016) .",
        "startOffset" : 18,
        "endOffset" : 40
      }, {
        "referenceID" : 66,
        "context" : "439 DE: SG-DEPS-PW (300) (Vulić and Korhonen, 2016a) .",
        "startOffset" : 25,
        "endOffset" : 52
      }, {
        "referenceID" : 42,
        "context" : "325 DE: BiSkip-DE (256) (Luong et al., 2015) .",
        "startOffset" : 24,
        "endOffset" : 44
      }, {
        "referenceID" : 66,
        "context" : "421 IT: SG-DEPS-PW (300) (Vulić and Korhonen, 2016a) .",
        "startOffset" : 25,
        "endOffset" : 52
      }, {
        "referenceID" : 17,
        "context" : "391 IT: CBOW5-Wacky (300) (Dinu et al., 2015) .",
        "startOffset" : 26,
        "endOffset" : 45
      }, {
        "referenceID" : 74,
        "context" : "The DST model is the first component of modern dialogue pipelines (Young, 2010).",
        "startOffset" : 66,
        "endOffset" : 79
      }, {
        "referenceID" : 72,
        "context" : "The Dialogue State Tracking Challenge (DSTC) shared task series formalised the evaluation and provided labelled DST datasets (Henderson et al., 2014a,b; Williams et al., 2016).",
        "startOffset" : 125,
        "endOffset" : 175
      }, {
        "referenceID" : 68,
        "context" : ", handcrafted rules (Wang et al., 2014) or conditional random fields (Lee and Eskenazi, 2013), the recent DST methodology has seen a shift towards neuralnetwork architectures (Henderson et al.",
        "startOffset" : 20,
        "endOffset" : 39
      }, {
        "referenceID" : 36,
        "context" : ", 2014) or conditional random fields (Lee and Eskenazi, 2013), the recent DST methodology has seen a shift towards neuralnetwork architectures (Henderson et al.",
        "startOffset" : 37,
        "endOffset" : 61
      }, {
        "referenceID" : 48,
        "context" : "by reasoning purely over pre-trained word vectors (Mrkšić et al., 2016).",
        "startOffset" : 50,
        "endOffset" : 71
      }, {
        "referenceID" : 48,
        "context" : "0 release expanded the dataset to 1,200 dialogues (Mrkšić et al., 2016).",
        "startOffset" : 50,
        "endOffset" : 71
      }, {
        "referenceID" : 66,
        "context" : "0 Dataset Our DST evaluation is based on the WOZ dataset, released by Wen et al. (2017). In this Wizard-of-Oz setup, two Amazon Mechanical Turk workers assumed the role of the user and the system asking/providing restaurant information.",
        "startOffset" : 70,
        "endOffset" : 88
      }, {
        "referenceID" : 28,
        "context" : "Evaluation Setup The principal metric we use to measure DST performance is joint goal accuracy, which represents the proportion of test set dialogue turns where all user goals expressed up to that point of the dialogue were decoded correctly (Henderson et al., 2014a).",
        "startOffset" : 242,
        "endOffset" : 267
      }, {
        "referenceID" : 75,
        "context" : "Some models integrate such constraints into the training procedure, modifying the prior or the regularisation (Yu and Dredze, 2014; Xu et al., 2014; Bian et al., 2014; Kiela et al., 2015), or using a variant of the SGNS-style objective (Liu et al.",
        "startOffset" : 110,
        "endOffset" : 187
      }, {
        "referenceID" : 73,
        "context" : "Some models integrate such constraints into the training procedure, modifying the prior or the regularisation (Yu and Dredze, 2014; Xu et al., 2014; Bian et al., 2014; Kiela et al., 2015), or using a variant of the SGNS-style objective (Liu et al.",
        "startOffset" : 110,
        "endOffset" : 187
      }, {
        "referenceID" : 6,
        "context" : "Some models integrate such constraints into the training procedure, modifying the prior or the regularisation (Yu and Dredze, 2014; Xu et al., 2014; Bian et al., 2014; Kiela et al., 2015), or using a variant of the SGNS-style objective (Liu et al.",
        "startOffset" : 110,
        "endOffset" : 187
      }, {
        "referenceID" : 33,
        "context" : "Some models integrate such constraints into the training procedure, modifying the prior or the regularisation (Yu and Dredze, 2014; Xu et al., 2014; Bian et al., 2014; Kiela et al., 2015), or using a variant of the SGNS-style objective (Liu et al.",
        "startOffset" : 110,
        "endOffset" : 187
      }, {
        "referenceID" : 41,
        "context" : ", 2015), or using a variant of the SGNS-style objective (Liu et al., 2015; Osborne et al., 2016).",
        "startOffset" : 56,
        "endOffset" : 96
      }, {
        "referenceID" : 53,
        "context" : ", 2015), or using a variant of the SGNS-style objective (Liu et al., 2015; Osborne et al., 2016).",
        "startOffset" : 56,
        "endOffset" : 96
      }, {
        "referenceID" : 21,
        "context" : ", WordNet, PPDB) into pre-trained word vectors (Faruqui et al., 2015; Jauhar et al., 2015; Wieting et al., 2015; Nguyen et al., 2016; Mrkšić et al., 2016).",
        "startOffset" : 47,
        "endOffset" : 154
      }, {
        "referenceID" : 32,
        "context" : ", WordNet, PPDB) into pre-trained word vectors (Faruqui et al., 2015; Jauhar et al., 2015; Wieting et al., 2015; Nguyen et al., 2016; Mrkšić et al., 2016).",
        "startOffset" : 47,
        "endOffset" : 154
      }, {
        "referenceID" : 70,
        "context" : ", WordNet, PPDB) into pre-trained word vectors (Faruqui et al., 2015; Jauhar et al., 2015; Wieting et al., 2015; Nguyen et al., 2016; Mrkšić et al., 2016).",
        "startOffset" : 47,
        "endOffset" : 154
      }, {
        "referenceID" : 52,
        "context" : ", WordNet, PPDB) into pre-trained word vectors (Faruqui et al., 2015; Jauhar et al., 2015; Wieting et al., 2015; Nguyen et al., 2016; Mrkšić et al., 2016).",
        "startOffset" : 47,
        "endOffset" : 154
      }, {
        "referenceID" : 48,
        "context" : ", WordNet, PPDB) into pre-trained word vectors (Faruqui et al., 2015; Jauhar et al., 2015; Wieting et al., 2015; Nguyen et al., 2016; Mrkšić et al., 2016).",
        "startOffset" : 47,
        "endOffset" : 154
      }, {
        "referenceID" : 2,
        "context" : ", 2015) such as CELEX (Baayen et al., 1995) or as an external analyser such as Morfessor (Creutz and Lagus, 2007), along with distributional information jointly at training time in the language modelling (LM) objective (Luong et al.",
        "startOffset" : 22,
        "endOffset" : 43
      }, {
        "referenceID" : 15,
        "context" : ", 1995) or as an external analyser such as Morfessor (Creutz and Lagus, 2007), along with distributional information jointly at training time in the language modelling (LM) objective (Luong et al.",
        "startOffset" : 53,
        "endOffset" : 77
      }, {
        "referenceID" : 35,
        "context" : "The key idea is to learn a morphological composition function (Lazaridou et al., 2013; Cotterell and Schütze, 2017) which synthesises the representation of a word given the representations of its constituent morphemes.",
        "startOffset" : 62,
        "endOffset" : 115
      }, {
        "referenceID" : 13,
        "context" : "The key idea is to learn a morphological composition function (Lazaridou et al., 2013; Cotterell and Schütze, 2017) which synthesises the representation of a word given the representations of its constituent morphemes.",
        "startOffset" : 62,
        "endOffset" : 115
      }, {
        "referenceID" : 2,
        "context" : ", 2015) such as CELEX (Baayen et al., 1995) or as an external analyser such as Morfessor (Creutz and Lagus, 2007), along with distributional information jointly at training time in the language modelling (LM) objective (Luong et al., 2013; Botha and Blunsom, 2014; Qiu et al., 2014; Cotterell and Schütze, 2015; Bhatia et al., 2016, i.a.). The key idea is to learn a morphological composition function (Lazaridou et al., 2013; Cotterell and Schütze, 2017) which synthesises the representation of a word given the representations of its constituent morphemes. Contrary to our work, these models typically coalesce all lexical relations. Another class of models, operating at the character level, shares a similar methodology: such models compose token-level representations from subcomponent embeddings (subwords, morphemes, or characters) (dos Santos and Zadrozny, 2014; Ling et al., 2015; Cao and Rei, 2016; Kim et al., 2016; Wieting et al., 2016; Verwimp et al., 2017, i.a.). In contrast to prior work, our model decouples the use of morphological information, now provided in the form of inflectional and derivational rules transformed into linguistic constraints, from the actual training. This pipelined approach results in a simpler, more portable model. In spirit, our work is similar to Cotterell et al. (2016), who formulate the idea of post-training specialisation in a generative Bayesian framework.",
        "startOffset" : 23,
        "endOffset" : 1319
      }, {
        "referenceID" : 60,
        "context" : "Future work will focus on other potential sources of morphological knowledge (Soricut and Och, 2015), porting the framework to other morphologically rich languages and downstream tasks, and on further refinements of the post-processing algorithm and constraints selection.",
        "startOffset" : 77,
        "endOffset" : 100
      } ],
      "year" : 2017,
      "abstractText" : "Morphologically rich languages accentuate two properties of distributional vector space models: 1) the difficulty of inducing accurate representations for lowfrequency word forms; and 2) insensitivity to distinct lexical relations that have similar distributional signatures. These effects are detrimental for language understanding systems, which may infer that inexpensive is a rephrasing for expensive or may not associate acquire with acquires. In this work, we propose a novel morph-fitting procedure which moves past the use of curated semantic lexicons for improving distributional vector spaces. Instead, our method injects morphological constraints generated using simple language-specific rules, pulling inflectional forms of the same word close together and pushing derivational antonyms far apart. In intrinsic evaluation over four languages, we show that our approach: 1) improves low-frequency word estimates; and 2) boosts the semantic quality of the entire word vector collection. Finally, we show that morph-fitted vectors yield large gains in the downstream task of dialogue state tracking, highlighting the importance of morphology for tackling long-tail phenomena in language understanding tasks.",
      "creator" : "LaTeX with hyperref package"
    }
  }