For the automatic evaluation we used the BLEU  , METEOR  and chrF  metrics. BLEU (BiLingual Evaluation Understudy) is calculated for individual translated segments (n-grams) by comparing them with a data set of reference translations. The calculated scores, between 0 and 100 (perfect translation), are averaged over the whole evaluation data set to reach an estimate of the translation's overall quality. Considering the short length of the terms in WordNet, while we report scores based on the unigram overlap (BLEU-1), and as this is in most cases only precision, so in addition we also report other metrics. METEOR (Metric for Evaluation of Translation with Explicit ORdering) is based on the harmonic mean of precision and recall, whereby recall is weighted higher than precision. In addition to exact word (or phrase) matching it has additional features, i.e. stemming, paraphrasing and synonymy matching. In contrast to BLEU, the metric produces good correlation with human judgement at the sentence or segment level.

chrF3 is a character n-gram metric, which has shown very good correlations with human judgements on the WMT2015 shared metric task  , especially when translating from English into morphologically rich(er) languages. As there are multiple translations available for each sense in the target wordnet we use all translations as multiple references for BLEU, for the other two metrics we compare only to the most frequent member of the synset. Here we present the evaluation of the translated English WordNet words into 16 European languages. We evaluate the quality of translations of the WordNet entries against the existing entries in the non-English Wordnets (Table TABREF34 ).

This is done by assigning WordNet synsets to Wikipedia entries, and making these relations multilingual through the interlingual links. For languages, which do not have the corresponding Wikipedia entry, the authors use Google Translate to translate English sentences containing the synset in the sense annotated corpus. After that, the most frequent translation is included as a variant for the synset for the given language. The use of parallel corpora has been previously exploited for word sense disambiguation, for example to construct sense-tagged corpora in another language  or by using translations as a method to discriminate senses  . It has been shown that the combination of these techniques can improve supervised word sense disambiguation  . A similar approach to the one proposed in this paper is that of  , where they show that using the interlingual index of WordNet with the help of parallel text can improve word sense disambiguation of a monolingual approach and we generalize this result to generate wordnets for new languages.

Special Characteristics of Polylingual Wordnet	Polylingual WordNet is the largest multilingual resource released under an open license and the only resource that has been developed in a fully automatic manner. As such, the resource plays a number of useful roles in covering applications for languages or applications where the manually constructed wordnets do not have sufficient coverage. Moreover, this WordNet is intended to be a basis that can help in the translation of existing wordnets by providing a basis from which lexicographers can work.

Polylingual Wordnet	Princeton WordNet is one of the most important resources for natural language processing, but is only available for English. While it has been translated using the expand approach to many other languages, this is an expensive manual process. Therefore it would be beneficial to have a high-quality automatic translation approach that would support NLP techniques, which rely on WordNet in new languages. The translation of wordnets is fundamentally complex because of the need to translate all senses of a word including low frequency senses, which is very challenging for current machine translation approaches. For this reason we leverage existing translations of WordNet in other languages to identify contextual information for wordnet senses from a large set of generic parallel corpora. We evaluate our approach using 10 translated wordnets for European languages. Our experiment shows a significant improvement over translation without any contextual information. Furthermore, we evaluate how the choice of pivot languages affects performance of multilingual word sense disambiguation.

A different approach to expand English WordNet synsets with lexicalizations in other languages was proposed in  . The authors do not directly match concepts in the two different language resources, but demonstrate an approach that learns how to determine the best translation for English synsets by taking bilingual dictionaries, structural information of the English WordNet and corpus frequency information into account. With the growing amount of parallel data,  show an approach to acquire a set of synsets from parallel corpora. The synsets are obtained by comparing aligned words in parallel corpora in several languages. Similarly, the sloWNet for Slovene  and Wolf for French  are constructed using a multilingual corpus and word alignment techniques in combination with other existing lexical resources. Since all these approaches use word alignment information, they are not able to generate any translation equivalents for multi-word expressions (MWE). In contrast, our approach use an SMT system trained on a large amount of parallel sentences, which allows us to align possible MWEs, such as commercial loan or take a breath, between source and target language. Furthermore, we engage the idea of identifying relevant contextual information to support an SMT system translating short expressions, which showed better performance compared to approaches without a context.

Discussion and Future Plans	Polylingual WordNet is a wordnet that has been constructed fully automatically based on the English Princeton WordNet and as such represents a significantly different resource to the others described in this volume, yet a resource that will still be helpful in a wide number of applications. This resource was created by means of a novel machine translation approach, which uses disambiguated contexts to find the correct translation of a given sense, and has been shown  that this is significantly better than direct translation. Furthermore, we have also used automatic methods to provide links from this resource to other resources by means of semantic and structural similarity, which gives a high quality linking to encyclopaedic resources, in particular DBpedia/Wikipedia. Thus, while our results show that this resource is of noticeably lower quality than manually constructed resources, there are still many applications where the wide coverage of this resource would be preferred to smaller, high-quality wordnets. We intend to continue to refine our processes, in order to close the gap between this automatically constructed wordnet and manually constructed wordnets in terms of quality. Furthermore, we are working on expanding the coverage of this resource beyond European languages and in particular into under-resourced languages such as Dravidian and Gaelic languages.

We calculate a score as: INLINEFORM3  where INLINEFORM0 is the length of the WordNet text and INLINEFORM1 is the length of the Wikipedia text. We calculate a similarity using the LSTM approach described by  . As each of these features can be turned on or off individually we consider the effects of each for the quality of the alignment between Princeton WordNet and Wikipedia. For this we used the 200NS dataset from  , and computed the precision, recall and F-measure of the mapping for various settings, using 10-fold cross validation, as follows: Only Jaccard score of concatenated label Only smoothed Jaccard ( INLINEFORM0 ) of concatenated label As above with basic features (Jaccard, Dice, Containment, Length Ratio, Average Word Length Ratio, Negation and Number) for the most similar label As above with basic features for label and description As above with GloVe Similarity As above with LSTM score As 5, but using superterm labels As 7, but using superterm labels The results show that the combination of different factors improves the matching quality and we see large gains in the score by combining these features.

Bridging the Language Barrier	The construction of Polylingual WordNet is based on phrase-based SMT  , where we wish to find the best translation of a string, given by a log-linear model combining a set of features. The translation that maximizes the score of the log-linear model is obtained by searching all possible translations candidates. The decoder, which is a search procedure, provides the most probable translation based on a statistical translation model learned from the training data. For our translation task, we use the statistical translation toolkit Moses  , where word alignments, necessary for generating translation models, were built with the GIZA++ toolkit  . The Kenlm toolkit  was used to build a 5-gram language model. To ensure a broad lexical and domain coverage of our SMT system we merged the existing parallel corpora for each language pair from the OPUS web page into one parallel data set, i.e., Europarl  , DGT - translation memories generated by the Directorate-General for Translation  , MultiUN corpus  , EMEA, KDE4, OpenOffice  , OpenSubtitles2012  , among others. The automatic translation evaluation is based on the correspondence between the SMT output and reference translation (gold standard).

The most straightforward matcher is called the greedy matcher, which simply compares each pair of entities between the two datasets using a similarity function to compute the similarity between this pair. The similarity calculation is divided into three stages: firstly, the lens examines the entity pair and extracts information that can be more easily compared, for example a pair of labels one for each element. This is then fed into a feature extractor that analyses the facet to produce a numerical value that is assumed to be related to the similarity of this entity pair. Finally, a supervised similarity classifier is used to aggregate a number of features and is trained on existing training data. This then creates a single score between zero (totally dissimilar) and one (identical) for each pair of entities between the two datasets. The greedy matcher then proceeds by adding matches between the dataset that increase a topological constrained score (TCS), which is a function that both checks the validity and computes the score of a matching between two datasets. In our experiments we use the bipartite TCS, which constraints the matching such that no entity in either dataset is linked twice and then produces as a score, the sum of all similarity scores in the dataset.

Development of Polylingual WordNet	The Princeton WordNet is a large, publicly available lexical semantic database of English nouns, verbs, adjectives and adverbs, grouped into synsets ( INLINEFORM0 117,000), which are aligned in terms of semantic and lexical relations. While it has been translated using the expand approach to many other languages, this is an expensive manual process. Therefore it would be beneficial to have a high-quality automatic translation approach that would support NLP techniques, which rely on WordNet in new languages. The translation of wordnets is fundamentally complex because of the need to translate all senses of a word including low frequency senses, which is very challenging for current machine translation approaches. Our approach takes the advantage of the increasing amount of parallel corpora in combination with wordnets in languages other than English for sense disambiguation, which helps us to improve automatic translations of English WordNet entries.

This method uses a multi-feature approach to establish similarities between synsets and DBpedia entities based on analysis of the definitions using a variety of methods from simple string statistics, to methods based on explicit semantic analysis as well as deep learning methods including long short-term memory (LSTM) networks. These statistics are created based on the Princeton WordNet synset gloss as well as the neighbouring words in the WordNet graph. These are combined using a constraint-based solver that considers not only the semantic similarity of the synsets but also the overall structure of the alignment and its consistency, following the best practices in ontology alignment. This work has led to the development of a large multilingual WordNet in more than 20 European languages, which we call Polylingual WordNet  , which is available under an open (CC-BY) license. Finally, we describe how this resource is published, firstly as linked data in the linguistic linked open data cloud, as well as published in all the formats of the Global WordNet Association Interlingual Index.

Links to external resources	Polylingual WordNet contains links to DBpedia (and hence also to Wikipedia), which as with the nature of this resource have been automatically constructed using a dataset alignment approach. In order to link Polylingual WordNet to other resources, we really on a dataset alignment system called NAISC (Nearly Automatic Alignment of SChema), which is a system designed to create linking between two resources. As such NAISC takes two lists of entities as input that may have the following: Each entity may have multiple labels in multiple languages, which are simply strings. In the case of wordnets, these are the words in each synset. As labels these are strings grouped by language, and correspond to synset definitions in wordnet. A list of relations to other entities. The list of relation types is fixed, and corresponds well to those in wordnets, e.g., `broader' INLINEFORM0 `hypernym'. The type of an entity, in the case of wordnet this is a part-of-speech. The development of a match is performed by a matcher, which defines the process for finding the optimal match between two lists of entities.

Applications of Polylingual Wordnet	Polylingual WordNet was developed as part of the MixedEmotions project, which aims to develop an innovative, multilingual model platform for emotion analysis. As such, we included emotion information from two sources into Polylingual WordNet, namely the emotion analysis from WordNet Affect  and the Emo WordNet data from plWordNet 3.0  . These have been used for developing emotion classification tools that take the annotations and use them as features in a supervised classification task as part of the SenPy  system.

built small domain-specific translation models for ontology translation from relevant sentence pairs that were identified in a parallel corpus based on the ontology labels to be translated. With this approach they improve the translation quality over the usage of large generic translation models. Since the generation of translation models can be computational expensive,  use large generic translation models to translate ontology labels, which were placed into a disambiguated context. With this approach the authors demonstrate translation quality improvement over commercial systems, like Microsoft Translator. Different from this approach, which uses the hierarchical structure of the ontology for disambiguation, we engage a large number of different languages to identify the relevant context.   present a method for WordNet construction and enlargement with the help of sense tagged parallel corpora. Since parallel sense tagged data are not always available, they use Google Translate to translate a manually sense tagged corpus. In addition they apply automatic sense tagging of a manually translated parallel corpus, whereby they report worse performance compared to the previous approach. We try to overcome this issue by engaging up to ten languages to improve the performance of the automatic sense tagging. Similarly, BabelNet  aligns the lexicographic knowledge from WordNet to the encyclopaedic knowledge of Wikipedia.

Each of these contexts is given to the SMT system and the most frequent translation across these INLINEFORM4 contexts is used. Furthermore, the SMT system is configured to return the INLINEFORM5 highest scoring translations, according to its model, and we select the translation as the most frequent translation of the context among this INLINEFORM6 -best list. In our experiments, we combined this with INLINEFORM7 disambiguations to give INLINEFORM8 candidate translations from which the candidate is chosen. Since only WordNet synsets are linked across different languages, we first align them with its translation equivalents, which is performed with their appearance within several million parallel sentences. In the next step we identify English sentences, which contain an English WordNet entry. Due to the multilingual nature of a parallel corpus, we identify the non-English Wordnet sense on the target side of the parallel corpus. Our approach is based on the assumption that a sentence shares the same semantic information as the WordNet entry sysnset if its translation, with the same mining or synset respectively, appears in the parallel target sentence.

This is summarized in the algorithm in Figure FIGREF42 . In order to apply NAISC to the task of matching WordNet to Wikipedia, we need to define a set of lenses and feature extractors to calculate similarity between these entities. The following lenses were used in our experiments We extract every word from the target WordNet synset and for Wikipedia, the article title and all other titles that redirect to this article. We choose the pair that has the lowest Levenshtein distance between each other. We extract labels as above for WordNet and Wikipedia, but instead concatenate all the words extracted We compare the definition of the concept for WordNet with the first sentence of the Wikipedia article For each WordNet synset we extract all the words in every concept that is a hypernym (either directly or transitively). For Wikipedia we used the assigned categories in the DBpedia Ontology and YAGO Ontology  . We use the two labels extracted in this way that are closest in Levenshtein distance. All of the lenses that we extract are textual and as such we apply the following textual features, which were combined into a single similarity score using Weka's SMO classifier  .

Since most of the current non-English wordnets use the Princeton WordNet as a pivot resource, concepts, which are not in this English lexical resource cannot not be realized or aligned to it. Therefore the authors support the idea of a central platform of concepts, where new concepts may be added even if they are not represented (yet) in the Princeton WordNet or even lexicalized in English (e.g., many languages have distinct gendered role words, such as `male teacher' and `female teacher', but these meanings are not distinguished in English). Previous studies of generating non-English wordnets combined Wiktionary knowledge with existing wordnets to extend them or to create new ones  .   describe in their work the creation of the Open Multilingual Wordnet and its extension with other resources  . The resource is made by combining different wordnets together, knowledge from Wiktionary and the Unicode Common Locale Data Repository. Overall they obtained over 2 million senses for over 100 thousand concepts, linking over 1.4 million words in hundreds of languages. Since using existing lexical resources guarantees a high precision, it may also provide a low recall due to the limitedness of lexical resources in different languages and domains.

This disambiguation approach can be further strengthened, if translations of the targeted WordNet entry appear in several languages in the parallel corpus. Due to this assumption we use 16 different languages in our experiment, which requires 16 different non-English wordnets and parallel corpora. Besides the Princeton Wordnet, we engage wordnets, freely provided by the Open Multilingual Wordnet (OMW) web page, i.e.: Bulgarian   Croatian   Danish   Dutch   Finnish   French   Greek   Italian   Lithuanian   Polish   Portuguese   Romanian   Slovak   Slovene   Spanish   Swedish   Once we obtain a set of sense disambiguated sentences for each Wordnet entry, we start the translation approach. Our hunch is that correctly identified contextual information around the WordNet entry can guide the SMT system to correctly translate an ambiguous entry.

We assume that we have a multilingual parallel corpus consisting of sentences, INLINEFORM0 in a language INLINEFORM1 , grouped into parallel translations: INLINEFORM2  We also assume that we have a collection of wordnets consisting of a set of senses, INLINEFORM0 , grouped into synsets, for each language: INLINEFORM1  We say that a context INLINEFORM0 , in language INLINEFORM1 (in our case this is always English), is disambiguated in INLINEFORM2 languages for a word INLINEFORM3 if: INLINEFORM4  That is, a context is disambiguated in INLINEFORM0 languages for a word, if for each of its translations we have a context in the parallel corpus that contains one of the known synset translations. Furthermore, we assume we have an SMT system that can translate any context in INLINEFORM1 into our target language, INLINEFORM2 , and produces an alignment such that we know which word or phrase in the output corresponds to the input. Within the set of identified disambiguated contexts, the INLINEFORM3 top scoring contexts are used, with ties broken at random.

Related Work	The Princeton WordNet inspired many researchers to create similarly structured wordnets for other languages. The EuroWordNet project  linked wordnets in different languages through a so-called Inter-Lingual-Index (ILI) into a single multilingual lexical resource. Via this index, the languages are aligned between each other, which allows to go from a concept in one language to a concept with a similar meaning in any of the other languages. Further multilingual extensions were generated by the BalkaNet project  , focusing on the Balkan languages and MultiWordNet  , aligning Italian concepts to English equivalents. Due to the large interest in the multilingual extensions of the Princeton WordNet, several initiatives started with the aim to unifying and making these wordnets easily accessible. The KYOTO project  focused on the development of a language-independent module to which all existing wordnets can be connected, which would allow a better cross-lingual machine processing of lexical information. Recently this has been realized by a new Global WordNet Grid  that takes advantage of the Collaborative Inter-Lingual Index (CILI)  .

Introduction	Princeton WordNet  is one of the most important resources used in many different tasks across linguistics and natural language processing, however the resource is only available for English and is limited in its coverage of real world concepts. To cross the language barrier, huge efforts have been made to extend the Princeton WordNet with multilingual information in projects, such as EuroWordNet  , BalkaNet  and MultiWordNet  , mostly following the extend approach, where the structure of the Princeton WordNet is preserved and only the words in each synset are translated and new synsets are added for concepts. Furthermore, the Princeton WordNet has many fewer concepts than large scale encyclopedias such as Wikipedia and resources derived from it such as DBpedia  and BabelNet  . This problem is even worse for many non-English wordnets, due to the extend approach, as these resources have even fewer synsets than Princeton WordNet. Furthermore, there are still many languages for which a wordnet does not exist or is not available to all potential users due to licensing restrictions. To address these deficiencies we propose two approaches.

As such this was used as a matching algorithm to create the final mapping between Princeton WordNet and DBpedia.

Firstly, we apply high-quality statistical machine translation (SMT) to automatically translate the WordNet entries into several different European languages. While an SMT system can only return the most frequent translation when given a term by itself, we propose a novel method to provide strong word sense disambiguation when translating wordnet entries. In addition, our method can handle fundamental complexities such as the need to translate all senses of a word including low-frequency senses, which is very challenging for current SMT approaches. For these reasons, we leverage existing translations of Princeton WordNet entries in other languages to identify contextual information for wordnet senses from a large set of generic parallel corpora. The goal is to identify sentences that share the same semantic information in respect to the synset of the Princeton WordNet entry that we want to translate. Secondly, we describe a novel system based on state-of-the-art semantic textual similarity and ontology alignment to establish a new linking between Princeton WordNet and DBpedia.

The Languages covered in Polylingual Wordnet	The Princeton WordNet is one of the most important resources for natural language processing, but is only available for English. While it has been translated using the expand approach to many other languages, this process is a very time consuming and expensive process. Therefore we engage SMT to automatically translate WordNet entries in to 23 European languages, as seen in Table TABREF5 . With this amount of languages, Polylingual Wordnet covers eight different language families, i.e. Slavic, Germanic, Uralic, Romance, Hellenic, Celtic, Baltic and Semitic. Furthermore, the entries in the described wordnet are, besides the Latin script, represented in Cyrillic for Bulgarian and Greek alphabet for the Greek language.

Native Statistics	The left part of Table TABREF27 illustrates the size of the wordnets provided by the Open Multilingual Wordnet. The right part of the table shows statistics of the Polylingual Wordnet.

We consider the two strings both as a set of words and a set of characters and compute the following functions INLINEFORM0 , INLINEFORM1 , INLINEFORM2 . Smoothed Jaccard is calculated only on the word level for the concatenated labels facet. It is calculated as follows INLINEFORM0 where INLINEFORM1 . This is a variant of Jaccard that can be adjusted to distinguish matches on shorter texts; it tends to Jaccard at INLINEFORM2 . The ratio of the number of tokens in each sentence. For symmetry this ratio is defined as INLINEFORM0 . The average length of each word in the text are also compared as above. One if both texts or neither text contain a negation word (`not', `never', etc.), zero otherwise. One if all numbers (e.g, `6') in each text are found in the other, zero otherwise. For each word in each text we extract the GloVe vectors  and calculate the cosine similarity between these words, to give a value INLINEFORM0 between the INLINEFORM1 th and the INLINEFORM2 th word.

Current state of the Polylingual Wordnet	This wordnet is currently under active development and we plan to improve the quality of the resource over several iterations. The first release (1.0) was made in November 2016 and a second release has been made with this publication (1.1) in May 2017.

Question: Which metrics do they use to evaluate results?
Answer: BLEU  , METEOR  and chrF  metrics, precision, recall and F-measure