Evaluating KGR10 Polish word embeddings in the recognition of temporal expressions using BiLSTM-CRF	The article introduces a new set of Polish word embeddings, built using KGR10 corpus, which contains more than 4 billion words. These embeddings are evaluated in the problem of recognition of temporal expressions (timexes) for the Polish language. We described the process of KGR10 corpus creation and a new approach to the recognition problem using Bidirectional Long-Short Term Memory (BiLSTM) network with additional CRF layer, where specific embeddings are essential. We presented experiments and conclusions drawn from them.

Experiments and Results	Experiments were carried out by the method proposed in  . The first part is described as Task A, the purpose of which is to identify the boundaries of timexes and assign them to one of the following classes: date, time, duration, set. We trained the final models using the train set and we evaluated it using the test set, which was the reproduction of analysis performed in articles  ,  . The division is presented in Table TABREF16 . We used BiLSTM+CRF classifier as in previous work  . We used precision, recall and F1 metrics from the classic NER task  , where true positive system answer has the same boundaries and type as annotation in gold data set. We evaluated all 17 word embeddings models using these metrics. The results are presented in Tables TABREF17 , TABREF18 and TABREF19 . We chose the best 3 results from each word embeddings group (EE, EP, EC) from Table TABREF19 presenting F1-scores for all models. Then we evaluated these results using more detailed measures for timexes, presented in  .

Conclusions	The analysis of results from Tables TABREF17 , TABREF18 and TABREF19 show that 12 of 15 best results were obtained using new word embeddings. The evaluation results presented in Table TABREF20 (the chosen best embeddings models from Table TABREF19 ) prove that the best group of word embeddings is EC. The highest type F1-score was obtained for EC1 model, built using binary FastText Skip-gram method utilising subword information, with vector dimension equal to 300 and negative sampling equal to 10. The ability of the model to provide vector representation for the unknown words seems to be the most important. Also, previous models built using KGR10 (EP) are probably less accurate due to an incorrect tokenisation of the corpus. We used WCRFT tagger  , which utilises Toki  to tokenise the input text before the creation of the embeddings model. The comparison of EC1 with previous results obtained using only CRF  show the significant improvement across all the tested metrics: 3.6pp increase in strict F1-score, 1.36pp increase in relaxed precision, 5.61pp increase in relaxed recall and 3.51pp increase in relaxed F1-score.

Temporal expressions	Temporal expressions (henceforth timexes) tell us when something happens, how long something lasts, or how often something occurs. The correct interpretation of a timex often involves knowing the context. Usually, a person is aware of their location in time, i.e., they know what day, month and year it is, and whether it is the beginning or the end of week or month. Therefore, they refer to specific dates, using incomplete expressions such as 12 November, Thursday, the following week, after three days. The temporal context is often necessary to determine to which specific date and time timexes refer. These examples do not exhaust the complexity of the problem of recognising timexes. TimeML  is a markup language for describing timexes that has been adapted to many languages. One of the best-known methods of recognition of timexes called HeidelTime  , which uses the TIMEX3 annotation standard, currently supports 13 languages (with the use of hand-crafted resources). PLIMEX is a specification for the description of Polish timexes. It is based on TIMEX3 used in TimeML. Classes proposed in TimeML are adapted, namely: date, time, duration, set.

Building a larger corpus	KGR7 corpus (also called plWordNet Corpus 7.0, PLWNC 7.0)  ,  was created at the Wroclaw University of Science and Technology by G4.19 Group. Due to the licences of documents in this corpus, this resource is not publicly available. Table TABREF8 contains KGR7 subcorpora and statistics  . One of the subcorpora in KGR7 is KIPI (the IPI PAN Corpus)  . KGR7 covers texts from a wide range of domains like: blogs, science, stenographic recordings, news, journalism, books and parliamentary transcripts. All texts come from the second half of the 20th century and represent the modern Polish language.

Last two steps are available only for Polish. In order to significantly expand the set of documents in KGR7, we utilised DMOZ (short for directory.mozilla.org) – a multilingual open content directory of World Wide Web links, also known as Open Directory Project (ODP). The website with directory was closed in 2017, but the database still can be found on the web. Polish part of this directory contains more than 30,000 links to Polish websites. We used these links as root URLs for CorpoGrabber, and we downloaded more than 7TB of HTML web pages. After the extraction of text from HTML pages, deduplication of documents (including texts from KGR7) and removing bad quality documents (containing more than 30% of words outside the Morfeusz  dictionary) the result is KGR10 corpus, which contains 4,015,569,051 tokens and 18,084,712 unique words. Due to component licenses, KGR10 corpus is not publicly available.

plWordNet Corpus 10.0 (KGR10)	KGR10, also known as plWordNet Corpus 10.0 (PLWNC 10.0), is the result of the work on the toolchain to automatic acquisition and extraction of the website content, called CorpoGrabber  . It is a pipeline of tools to get the most relevant content of the website, including all subsites (up to the user-defined depth). The proposed toolchain can be used to build a big Web corpus of text documents. It requires the list of the root websites as the input. Tools composing CorpoGrabber are adapted to Polish, but most subtasks are language independent. The whole process can be run in parallel on a single machine and includes the following tasks: download of the HTML subpages of each input page URL with HTTrack, extraction of plain text from each subpage by removing boilerplate content (such as navigation links, headers, footers, advertisements from HTML pages)  , deduplication of plain text  , bad quality documents removal utilising Morphological Analysis Converter and Aggregator (MACA)  , documents tagging using Wrocław CRF Tagger (WCRFT)  .

This work describes how we extended KGR7 1G corpus to become KGR10 with 4 billion words. Next, we present the different variants of word embeddings produced using this corpus. In the article about the recognition of named entities for Polish from the previous year, these embeddings were used in one of the three voting models to obtain the best results and the final system PolDeepNer  took the second place in PolEval2018 Task 2  . In this article, we evaluated KGR10 FastText word embeddings in recognition of timexes.

There were also previous attempts to build KGR10 word vectors with other methods (including FastText), and the results are presented in the article  . We selected the best models from that article – with embedding ID prefix EP (embeddings, previous) in Table TABREF13 – to compare with new models, marked as embedding ID prefix EC in Table TABREF13 ). The word embeddings models used in PolDeepNer for recognition of timexes and named entities were EE1, . It was built on a plain KGR10. The dimension of word embedding is 300, the method of constructing vectors was Skip-gram  , and the number of negative samples for each positive example was 10.

It is possible to use a set of text documents containing even billions of words as training data. Both architectures (CBOW and SG) describe how the neural network learns the vector word representations for each word. In CBOW architecture the task is predicting the word given its context and in SG the task in predicting the context given the word. Due to a significant increase of quality using deep learning methods together with word embeddings as the input layer for neural networks, many word vector sets have been created, using different corpora. The widest range of available word embeddings is available for English  and there were not so many options for less popular languages, e.g. Polish. There was a definite need within CLARIN-PL project and Sentimenti to increase the quality of NLP methods for Polish which were utilising available Polish word vectors  ,  ,  ,  but only FastText modification of Skip-gram  was able to produce vectors for unknown words, based on character n-grams. The observation was that even using a sophisticated deep neural structure, the result strongly depends on the initial distributional representation. There was a need to build a massive corpus of Polish and create high-quality word vectors from that corpus.

Acknowledgements	Work co-financed as part of the investment in the CLARIN-PL research infrastructure funded by the Polish Ministry of Science and Higher Education and in part by the National Centre for Research and Development, Poland, under grant no POIR.01.01.01-00-0472/16.

Available word embeddings	At the time we were testing word embeddings for different applications, there were 2 most popular sources of word vectors. The first one, called IPIPAN, is the result of the project Compositional distributional semantic models for identification, discrimination and disambiguation of senses in Polish texts, the process of creating word embeddings is described in article  and corpora used were National Corpus of Polish (NKJP)  and Wikipedia (Wiki). The second one, called FASTTEXT, is original FastText word embeddings set, created for 157 languages (including Polish). Authors used Wikipedia and Common Crawl as the linguistic data source. Table TABREF6 shows the number of tokens in each corpus and the name of the institution which prepared it. There is also information about the public availability of the resource. Table TABREF7 presents the most commonly used word embeddings in CLARIN-PL before the creation of our embeddings.

KGR10 word embeddings	We created a new Polish word embeddings models using the KGR10 corpus. We built 16 models of word embeddings using the implementation of CBOW and Skip-gram methods in the FastText tool  . These models are available under an open license in the CLARIN-PL project DSpace repository. The internal encoding solution based on embeddings of n-grams composing each word makes it possible to obtain FastText vector representations, also for words which were not processed during the creation of the model. A vector representation is associated with character n-gram and each word is represented as the sum of its n-gram vector representations. Previous solutions ignored the morphology of words and were assigning a distinct vector to each word. This is a limitation for languages with large vocabularies and many rare words, like Turkish, Finnish or Polish  . Authors observed that using word representations trained with subword information outperformed the plain Skip-gram model and the improvement was most significant for morphologically rich Slavic languages such as Czech (8% reduction of perplexity over SG) and Russian (13% reduction)  . We expected that word embeddings created that way for Polish should also provide such improvements.

Recognition of timexes	There are many methods for recognising timexes that are widely used in natural language engineering. For English (but not exclusively), in approaches based on supervised learning, sequence labelling methods are often used, especially Conditional Random Fields  . A review of the methods in the article  about the recognition of timexes for English and Spanish has shown a certain shift within the most popular solutions. As with the normalisation of timexes, the best results are still achieved with rule-based methods, many new solutions have been introduced in the area of recognition. The best systems listed in  , called TIPSem  and ClearTK  , use CRFs for recognition, so initially, we decided to apply the CRF-based approach for this task. The results were described in  ,  . In recent years, solutions based on deep neural networks, using word representation in the form of word embeddings, created with the use of large linguistic corpus, have begun to dominate in the field of recognition of word expressions.

The most popular solutions include bidirectional long short-term memory neural networks (henceforth Bi-LSTM), often in combination with conditional random fields, as presented in the paper  dedicated to the recognition of proper names. For the Polish language, deep networks have also recently been used to recognise word expressions. In the issue of recognition of timexes, a bidirectional gated recurrent unit network (GRU) has been used  ,  . GRU network is described in detail in the article  . In case of recognition of event descriptions using Bi-LSTM and Bi-GRU, where most of the Liner2 features were included in the input feature vector, better results were obtained  than for the Liner2 method (but without taking into account domain dictionaries). In last year's publication on the issue of named entities recognition using BiLSTM+CRF (together with G4.19 Group members), we received a statistically significant improvement in the quality of recognition compared to a solution using CRF only. The solution has been called PolDeepNer  .

The following measures were used to evaluate the quality of boundaries and class recognition, so-called strict match: strict precision (Str.P), strict recall (Str.R) and strict F1-score (Str.F1). A relaxed match (Rel.P, Rel.R, Rel.F1) evaluation has also been carried out to determine whether there is an overlap between the system entity and gold entity, e.g. [Sunday] and [Sunday morning]  . If there was an overlap, a relaxed type F1-score (Type.F1) was calculated  . The results are presented in Table TABREF20 .

Introduction	Recent studies in information extraction domain (but also in other natural language processing fields) show that deep learning models produce state-of-the-art results  . Deep architectures employ multiple layers to learn hierarchical representations of the input data. In the last few years, neural networks based on dense vector representations provided the best results in various NLP tasks, including named entities recognition  , semantic role labelling  , question answering  and multitask learning  . The core element of most deep learning solutions is the dense distributed semantic representation of words, often called word embeddings. Distributional vectors follow the distributional hypothesis that words with a similar meaning tend to appear in similar contexts. Word embeddings capture the similarity between words and are often used as the first layer in deep learning models. Two of the most common and very efficient methods to produce word embeddings are Continuous Bag-of-Words (CBOW) and Skip-gram (SG), which produce distributed representations of words in a vector space, grouping them by similarity  ,  . With the progress of machine learning techniques, it is possible to train such models on much larger data sets, and these often outperform the simple ones.

Question: What conclusions are drawn from these experiments?
Answer: best results were obtained using new word embeddings, best group of word embeddings is EC, The highest type F1-score was obtained for EC1 model, built using binary FastText Skip-gram method utilising subword information, ability of the model to provide vector representation for the unknown words seems to be the most important