Neural Architectures ::: POS Tagging, Lemmatization, and Dependency Parsing ::: Input Embeddings	In our baseline model, we use the end-to-end word embeddings and also character-level word embeddings (bidirectional GRUs, , ,  of dimension 256) trained specifically for the task. Our architecture can optionally employ the following additional inputs [noitemsep,topsep=0pt] pretrained word embeddings (WE): For the PDT experiments, we generate the word embeddings with word2vec on a concatenation of large raw Czech corpora available from the LINDAT/CLARIN repository. For UD Czech, we use FastText word embeddings  of dimension 300, which we pretrain on Czech Wikipedia using segmentation and tokenization trained from the UD data. BERT : Pretrained contextual word embeddings of dimension 768 from the Base model. We average the last four layers of the BERT model to produce the embeddings. Because BERT utilizes word pieces, we decompose UD words into appropriate subwords and then average the generated embeddings over subwords belonging to the same word. Flair : Pretrained contextual word embeddings of dimension 4096.

Results ::: POS Tagging and Lemmatization on PDT 3.5	The POS tagging and lemmatization results are presented in Table TABREF44. The word2vec word embeddings (WE) considerably increase performance compared to the baseline, especially in POS tagging. When only Flair embeddings are added to the baseline, we also observe an improvement, but not as high. We hypothesise that the lower performance (in contrast with the results reported in ) is caused by the size of the training data, because we train the word2vec WE on considerably larger dataset than the Czech Flair model. However, when WE and Flair embeddings are combined, performance moderately increases, demonstrating that the two embedding methods produce at least partially complementary representations. The BERT embeddings alone bring highest improvement in performance. Furthermore, combination with WE or Flair again yields performance increase. The best results are achieved by exploiting all three embedding methods, substantially exceeding state-of-the-art results. Utilization of morphological dictionary improves prediction accuracy.

As a regularization, we apply dropout with rate $0.5$ and the word dropout replaces $20\%$ of words by the unknown token to force the network to rely more on context. We did not perform any complex hyperparameter search. In this model, we use the following word- and character-level word embeddings: [noitemsep,topsep=0pt] pretrained word embeddings: We use the FastText  word embeddings of dimension 300 from the publicly available Czech model. end-to-end word embeddings: We embed the input forms and lemmas (256 dimensions) and POS tags (one-hot). end-to-end character-level word embeddings: We use bidirectional GRUs ,  of dimension 128 in line with : we represent every Unicode character with a vector of dimension 128, and concatenate GRU outputs for forward and reversed word characters. Optionally, we add the BERT  and the Flair  contextualized embeddings in the same way as in the UDPipe 2.0 (Section SECREF16).

Of these, 3 participant systems used (a slightly modified variant of) the only newly presented contextualized embeddings called ELMo , most notably one of the shared task winners . BERT and Flair were not available at the time. For the Czech NER, Straková et al. (2016)  use an artificial neural network with word- and character-level word embeddings to perform NER on the Czech Named Entity Corpus (CNEC) , , .

Results ::: POS Tagging, Lemmatization and Dependency Parsing on Universal Dependencies	Table TABREF47 shows the performance of analyzed embedding methods in a joint model performing POS tagging, lemmatization, and dependency parsing on Czech PDT UD 2.3 treebank. This treebank is derived from PDT 3.5 a-layer, with original POS tags kept in XPOS, and the dependency trees and lemmas modified according to UD guidelines. We observe that the word2vec WEs perform similarly to Flair embeddings in this setting. Our hypothesis is that the word2vec WEs performance loss (compared to WEs in Section SECREF43) is caused by using a considerably smaller raw corpus to pretrain the WEs (Czech Wikipedia with 785M words, compared to 4G words used in Section SECREF43), due to licensing reasons. BERT embeddings once more deliver the highest improvement, especially in dependency parsing, and our best model employs all three embedding methods. In the previous ablation experiments, we used the gold segmentation and tokenization in the Czech PDT UD 2.3 treebank.

Related Work	As for the Prague Dependency Treebank (PDT) , most of the previous works are non-neural systems with one exception of  who hold the state of the art for Czech POS tagging and lemmatization, achieved with the recurrent neural network (RNN) using end-to-end trainable word embeddings and character-level word embeddings. Otherwise, Spoustová et al. (2009)  used an averaged perceptron for POS tagging. For parsing the PDT, Holan and Zabokrtský (2006)  and Novák and Žabokrtský (2007)  used a combination of non-neural parsing techniques . In the multilingual shared task CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies , raw text is processed and the POS tagging, lemmatization and dependency parsing are evaluated on the Universal Dependencies (UD) . Czech is one of the 57 evaluated languages. Interestingly, all 26 participant systems employed the artificial neural networks in some way.

Using two of these methods, for which precomputed models for Czech are available, namely BERT and Flair, we present our models for four NLP tasks: part-of-speech (POS) tagging, lemmatization, dependency parsing and named entity recognition (NER). Adding the contextualized embeddings as optional inputs in strong artificial neural network baselines, we report state-of-the-art results in these four tasks.

Neural Architectures ::: POS Tagging, Lemmatization, and Dependency Parsing ::: Dependency Parsing	The dependency parsing is again predicted using UDPipe 2.0 architecture. After embedding input words, three bidirectional LSTM  layers are again performed, followed by a biaffine attention layer  producing labeled dependency trees. In our evaluation we do not utilize gold POS tags and lemmas on the test set for dependency parsing. Instead, we consider three ways of employing them during parsing: [noitemsep,topsep=0pt] not using them at all; adding predicted POS tags and lemmas on input; perform joint training of POS tags, lemmatization, and dependency parsing. In this case, we share first two bidirectional LSTM layers between the tagger and the parser.

Results ::: Dependency Parsing on PDT 3.5	The evaluation of the contextualized embeddings methods as well as various ways of POS tag utilization is presented in Table TABREF44. Without POS tags and lemmas, the Flair embeddings bring only a slight improvement in dependency parsing when added to WE. In contrast, BERT embeddings employment results in substantial gains, increasing UAS and LAS by 1.6% and 2.1%. A combination of BERT and Flair embeddings does not result in any performance improvement, demonstrating that BERT syntactic representations encompass the Flair embeddings. When introducing POS tags and lemmas predicted by the best model from Section SECREF43 as inputs for dependency parsing, the performance increases only slightly. A better way of POS tags and lemmas exploitation is achieved in a joint model, which predicts POS tags, lemmas, and dependency trees simultaneously. Again, BERT embeddings bring significant improvements, but in contrast to syntax parsing only, adding Flair embeddings to BERT results in moderate gain – we hypothesise that the increase is due to the complementary morphological information present in Flair embeddings (cf. Section SECREF43).

Introduction	Recently, a novel way of computing word embeddings has been proposed. Instead of computing one word embedding for each word which sums over all its occurrences, ignoring the appropriate word meaning in various contexts, the contextualized embeddings are computed for each word occurrence, taking into account the whole sentence. Three ways of computing such contextualized embeddings have been proposed: ELMo , BERT  and Flair , along with precomputed models. Peters et al. (2018)  obtain the proposed embeddings, called ELMo, from internal states of deep bidirectional language model, pretrained on a large corpus. Akbik et al. (2018)  introduced Flair, contextualized word embeddings obtained from internal states of a character-level bidirectional language model, thus significantly increasing state of the art of POS tagging, chunking and NER tasks. Last, but not least, Devlin et al. (2018)  employ a Transformer  to compute contextualized embeddings from preceeding and following context at the same time, at the cost of increased processing costs. The new BERT embeddings achieved state-of-the-art results in eleven natural language tasks.

Note that the joint model achieves better parsing accuracy than the one given gold POS tags and lemmas on input. However, the POS tags and lemmas predicted by the joint model are of slightly lower quality compared to a standalone tagger of the best configuration from Section SECREF43. Table TABREF44 compares our best model with state-of-the-art results on PDT 2.0 (note that some of the related work used only a subset of PDT 2.0 and/or utilized gold morphological annotation). To our best knowledge, research on PDT parsing was performed mostly in the first decade of this century, therefore even our baseline model substantially surpasses previous works. Our best model with contextualized embeddings achieves nearly 50% error reduction both in UAS and LAS.

Conclusion	We have presented an evaluation of two contextualized embeddings methods, namely BERT and Flair. By utilizing these embeddings as input to deep neural networks, we have achieved state-of-the-art results in several Czech text processing tasks, namely in POS tagging, lemmatization, dependency parsing and named entity recognition.

Czech Text Processing with Contextual Embeddings: POS Tagging, Lemmatization, Parsing and NER	Contextualized embeddings, which capture appropriate word meaning depending on context, have recently been proposed. We evaluate two meth ods for precomputing such embeddings, BERT and Flair, on four Czech text processing tasks: part-of-speech (POS) tagging, lemmatization, dependency pars ing and named entity recognition (NER). The first three tasks, POS tagging, lemmatization and dependency parsing, are evaluated on two corpora: the Prague Dependency Treebank 3.5 and the Universal Dependencies 2.3. The named entity recognition (NER) is evaluated on the Czech Named Entity Corpus 1.1 and 2.0. We report state-of-the-art results for the above mentioned tasks and corpora.

Results ::: Named Entity Recognition	Table TABREF47 shows NER results (F1 score) on CNEC 1.1 and CNEC 2.0. Our sequence-to-sequence (seq2seq) model which captures the nested entities, clearly surpasses the current Czech NER state of the art. Furthermore, significant improvement is gained when adding the contextualized word embeddings (BERT and Flair) as optional input to the LSTM encoder. The strongest model is a combination of the sequence-to-sequence architecture with both BERT and Flair contextual word embeddings.

Neural Architectures ::: POS Tagging, Lemmatization, and Dependency Parsing ::: POS Tagging and Lemmatization	The tagger employs a standard bi-LSTM architecture. After embedding input words, three bidirectional LSTM  layers are performed, followed by a softmax output layers for POS tags and lemmas. While a classification output layer is natural for POS tags, we also apply it to lemmatization and generate lemmas by classifying the input words into lemma generation rules, therefore considering lemmatization as another tagging task. We construct a lemma generation rule from a given form and lemma as follows: [noitemsep,topsep=0pt] We start by finding the longest continuous substring of the form and the lemma. If it is empty, we use the lemma itself as the class. If there is a common substring of the form and the lemma, we compute the shortest edit script converting the prefix of the form into the prefix of the lemma, and the shortest edit script converting the suffix of the form to the suffix of the lemma. The edit scripts permit the operations delete_current_char and insert_char(c). All above operations are performed case insensitively.

Neural Architectures ::: Named Entity Recognition	We use a novel approach  for nested named entity recognition (NER) to capture the nested entities in the Czech Named Entity Corpus. The nested entities are encoded in a sequence and the problem of nested NER is then viewed as a sequence-to-sequence (seq2seq) problem, in which the input sequence consists of the input tokens (forms) and the output sequence of the linearized entity labels. The system is a encoder-decoder architecture. The encoder is a bi-directional LSTM and the decoder is a LSTM. The encoded labels are predicted one by one by the decoder, until the decoder outputs the "<eow>" (end of word) label and moves to the next token. We use a hard attention on the word whose label(s) is being predicted. We train the network using the lazy variant of the Adam optimizer , which only updates accumulators for variables that appear in the current batch, with parameters $\beta _1=0.9$ and $\beta _2=0.98$. We use mini-batches of size 8.

However, as the performance of a model itself increases, the gains obtained by the morphological dictionary diminishes – for a model without any pretrained embeddings, morphological dictionary improves POS tagging by and lemmatization by $0.43\%$ and $0.45\%$, while the best performing model gains only $0.11\%$ and $0.23\%$.

Acknowledgements	The work described herein has been supported by OP VVV VI LINDAT/CLARIN project (CZ.02.1.01/0.0/0.0/16_013/0001781) and it has been supported and has been using language resources developed by the LINDAT/CLARIN project (LM2015071) of the Ministry of Education, Youth and Sports of the Czech Republic.

Datasets ::: Prague Dependency Treebank 3.5	The Prague Dependency Treebank 3.5  is a 2018 edition of the core Prague Dependency Treebank. The Prague Dependency Treebank 3.5 contains the same texts as the previous versions since 2.0, and is divided into train, dtest, and etest subparts, where dtest is used as a development set and etest as a test set. The dataset consists of several layers – the morphological m-layer is the largest and contains morphological annotations (POS tags and lemmas), the analytical a-layer contains labeled dependency trees, and the t-layer is the smallest and contains tectogrammatical trees. The statistics of PDT 3.5 sizes is presented in Table TABREF7. A detailed description of the morphological system can be found in , a specification of the syntactic annotations has been presented in . We note that in PDT, lemmas with the same word form are disambiguated using a number suffix – for example, English lemmas for the word forms can (noun) and can (verb) would be annotated as can-1 and can-2.

Neural Architectures ::: POS Tagging, Lemmatization, and Dependency Parsing ::: POS Tags and Lemmas Decoding	Optionally, we employ a morphological dictionary MorfFlex  during decoding. If the morphological dictionary is used, it may produce analyses for an input word as (POS tag, lemma) pairs. If any are generated, we choose the pair with maximum likelihood given by both the POS tag and lemmatization model.

Neural Architectures ::: POS Tagging, Lemmatization, and Dependency Parsing	We perform POS tagging, lemmatization and dependency parsing using UDPipe 2.0 , one of the three winning systems of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies  and an overall winner of The 2018 Shared Task on Extrinsic Parser Evaluation . An overview of this architecture is presented in Figure FIGREF17 and the full details of the architecture and the training procedure are available in .

To compute the evaluation scores, we use the official CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies  evaluation script, which produces the following metrics: [noitemsep,topsep=0pt] UPOS – universal POS tags accuracy, XPOS – language-specific POS tags accuracy, UFeats – universal subset of morphological features accuracy, Lemmas – lemmatization accuracy, UAS – unlabeled attachment score, LAS – labeled attachment score, MLAS – morphology-aware LAS, BLEX – bi-lexical dependency score.

Datasets ::: Universal Dependencies	The Universal Dependencies project  seeks to develop cross-linguistically consistent treebank annotation of morphology and syntax for many languages. We evaluate the Czech PDT treebank of UD 2.3 , which is an automated conversion of PDT 3.5 a-layer to Universal Dependencies annotation. The original POS tags are used to generate UPOS (universal POS tags), XPOS (language-specific POS tags, in this case the original PDT tags), and Feats (universal morphological features). The UD lemmas are the raw textual lemmas, so the discriminative numeric suffix of PDT is dropped. The dependency trees are converted according to the UD guidelines, adapting both the unlabeled trees and the dependency labels.

For comparison with state of the art, Czech PDT UD 2.2 treebank without gold segmentation and tokenization is used in evaluation, according to the CoNLL 2018 shared task training and evaluation protocol. Our system reuses segmentation and tokenization produced by UDPipe 2.0 in the CoNLL 2018 shared task and surpasses previous works substantially in all metrics (bottom part of Table TABREF47). Comparing the results with a joint tagging and parsing PDT 3.5 model from Table TABREF7, we observe that the XPOS results are nearly identical as expected. Lemmatization on the UD treebank is performed without the discriminative numeric suffixes (see Section SECREF3) and therefore reaches better performance. Both UAS and LAS are also better on the UD treebank, which we assume is caused by the different annotation scheme.

To indicate correct casing of the lemma, we consider the lemma to be a concatenation of segments, where each segment is composed of either a sequence of lowercase characters, or a sequence of uppercase characters. We represent the lemma casing by encoding the beginning of every such segment, where the offsets in the first half of the lemma are computed relatively to the start of the lemma, and the offsets in the second half of the lemma are computed relatively to the end of the lemma.

Datasets ::: Czech Named Entity Corpus	The Czech Named Entity Corpus 1.1 ,  is a corpus of $5\,868$ Czech sentences with manually annotated $33\,662$ Czech named entities, classified according to a two-level hierarchy of 62 named entities. The Czech Named Entity Corpus 2.0  contains $8\,993$ Czech sentences with manually annotated $35\,220$ Czech named entities, classified according to a two-level hierarchy of 46 named entities. We evaluate the NER task with the official CNEC evaluation script. Similarly to previous literature ,  etc., the script only evaluates the first round annotation classes for the CNEC 1.1. For the CNEC 2.0, the script evaluates all annotated classes.

Neural Architectures	All our neural architectures are recurrent neural networks (RNNs). The POS tagging, lemmatization and dependency parsing is performed with the UDPipe 2.0 (Section SECREF16) and NER is performed with our new sequence-to-sequence model (Section SECREF36).

In evaluation, we compute: [noitemsep,topsep=0pt] POS tagging accuracy, lemmatization accuracy, unlabeled attachment score (UAS), labeled attachment score (LAS).

Question: What data is used to build the embeddings?
Answer: large raw Czech corpora available from the LINDAT/CLARIN repository, Czech Wikipedia