Question: What is specific about the specific embeddings?
Evidence: Recent studies in information extraction domain (but also in other natural language processing fields) show that deep learning models produce state-of-the-art results BIBREF0 . Deep architectures employ multiple layers to learn hierarchical representations of the input data. In the last few years, neural networks based on dense vector representations provided the best results in various NLP tasks, including named entities recognition BIBREF1 , semantic role labelling BIBREF2 , question answering BIBREF3 and multitask learning BIBREF4 . The core element of most deep learning solutions is the dense distributed semantic representation of words, often called word embeddings. Distributional vectors follow the distributional hypothesis that words with a similar meaning tend to appear in similar contexts. Word embeddings capture the similarity between words and are often used as the first layer in deep learning models. Two of the most common and very efficient methods to produce word embeddings are Continuous Bag-of-Words (CBOW) and Skip-gram (SG), which produce distributed representations of words in a vector space, grouping them by similarity BIBREF5 , BIBREF6 . With the progress of machine learning techniques, it is possible to train such models on much larger data sets, and these often outperform the simple ones. It is possible to use a set of text documents containing even billions of words as training data. Both architectures (CBOW and SG) describe how the neural network learns the vector word representations for each word. In CBOW architecture the task is predicting the word given its context and in SG the task in predicting the context given the word.
We created a new Polish word embeddings models using the KGR10 corpus. We built 16 models of word embeddings using the implementation of CBOW and Skip-gram methods in the FastText tool BIBREF9 . These models are available under an open license in the CLARIN-PL project DSpace repository. The internal encoding solution based on embeddings of n-grams composing each word makes it possible to obtain FastText vector representations, also for words which were not processed during the creation of the model. A vector representation is associated with character n-gram and each word is represented as the sum of its n-gram vector representations. Previous solutions ignored the morphology of words and were assigning a distinct vector to each word. This is a limitation for languages with large vocabularies and many rare words, like Turkish, Finnish or Polish BIBREF9 . Authors observed that using word representations trained with subword information outperformed the plain Skip-gram model and the improvement was most significant for morphologically rich Slavic languages such as Czech (8% reduction of perplexity over SG) and Russian (13% reduction) BIBREF9 . We expected that word embeddings created that way for Polish should also provide such improvements. There were also previous attempts to build KGR10 word vectors with other methods (including FastText), and the results are presented in the article BIBREF8 . We selected the best models from that article – with embedding ID prefix EP (embeddings, previous) in Table TABREF13 – to compare with new models, marked as embedding ID prefix EC in Table TABREF13 ).
In CBOW architecture the task is predicting the word given its context and in SG the task in predicting the context given the word.
We built 16 models of word embeddings using the implementation of CBOW and Skip-gram methods in the FastText tool BIBREF9 .
Answer: The answers are shown as follows:
* predicting the word given its context