Introduction
Recently, a novel way of computing word embeddings has been proposed. Instead of computing one word embedding for each word which sums over all its occurrences, ignoring the appropriate word meaning in various contexts, the contextualized embeddings are computed for each word occurrence, taking into account the whole sentence. Three ways of computing such contextualized embeddings have been proposed: ELMo BIBREF0, BERT BIBREF1 and Flair BIBREF2, along with precomputed models.
Peters et al. (2018) BIBREF0 obtain the proposed embeddings, called ELMo, from internal states of deep bidirectional language model, pretrained on a large corpus. Akbik et al. (2018) BIBREF2 introduced Flair, contextualized word embeddings obtained from internal states of a character-level bidirectional language model, thus significantly increasing state of the art of POS tagging, chunking and NER tasks. Last, but not least, Devlin et al. (2018) BIBREF1 employ a Transformer BIBREF3 to compute contextualized embeddings from preceeding and following context at the same time, at the cost of increased processing costs. The new BERT embeddings achieved state-of-the-art results in eleven natural language tasks.
Using two of these methods, for which precomputed models for Czech are available, namely BERT and Flair, we present our models for four NLP tasks: part-of-speech (POS) tagging, lemmatization, dependency parsing and named entity recognition (NER). Adding the contextualized embeddings as optional inputs in strong artificial neural network baselines, we report state-of-the-art results in these four tasks.
Related Work
As for the Prague Dependency Treebank (PDT) BIBREF4, most of the previous works are non-neural systems with one exception of BIBREF5 who hold the state of the art for Czech POS tagging and lemmatization, achieved with the recurrent neural network (RNN) using end-to-end trainable word embeddings and character-level word embeddings. Otherwise, Spoustová et al. (2009) BIBREF6 used an averaged perceptron for POS tagging. For parsing the PDT, Holan and Zabokrtský (2006) BIBREF7 and Novák and Žabokrtský (2007) BIBREF8 used a combination of non-neural parsing techniques .
In the multilingual shared task CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies BIBREF9, raw text is processed and the POS tagging, lemmatization and dependency parsing are evaluated on the Universal Dependencies (UD) BIBREF10. Czech is one of the 57 evaluated languages. Interestingly, all 26 participant systems employed the artificial neural networks in some way. Of these, 3 participant systems used (a slightly modified variant of) the only newly presented contextualized embeddings called ELMo BIBREF0, most notably one of the shared task winners BIBREF11. BERT and Flair were not available at the time.
For the Czech NER, Straková et al. (2016) BIBREF12 use an artificial neural network with word- and character-level word embeddings to perform NER on the Czech Named Entity Corpus (CNEC) BIBREF13, BIBREF14, BIBREF15.
Datasets ::: Prague Dependency Treebank 3.5
The Prague Dependency Treebank 3.5 BIBREF4 is a 2018 edition of the core Prague Dependency Treebank. The Prague Dependency Treebank 3.5 contains the same texts as the previous versions since 2.0, and is divided into train, dtest, and etest subparts, where dtest is used as a development set and etest as a test set. The dataset consists of several layers – the morphological m-layer is the largest and contains morphological annotations (POS tags and lemmas), the analytical a-layer contains labeled dependency trees, and the t-layer is the smallest and contains tectogrammatical trees. The statistics of PDT 3.5 sizes is presented in Table TABREF7.
A detailed description of the morphological system can be found in BIBREF16, a specification of the syntactic annotations has been presented in BIBREF17. We note that in PDT, lemmas with the same word form are disambiguated using a number suffix – for example, English lemmas for the word forms can (noun) and can (verb) would be annotated as can-1 and can-2.
In evaluation, we compute:
[noitemsep,topsep=0pt]
POS tagging accuracy,
lemmatization accuracy,
unlabeled attachment score (UAS),
labeled attachment score (LAS).
Datasets ::: Universal Dependencies
The Universal Dependencies project BIBREF10 seeks to develop cross-linguistically consistent treebank annotation of morphology and syntax for many languages. We evaluate the Czech PDT treebank of UD 2.3 BIBREF18, which is an automated conversion of PDT 3.5 a-layer to Universal Dependencies annotation. The original POS tags are used to generate UPOS (universal POS tags), XPOS (language-specific POS tags, in this case the original PDT tags), and Feats (universal morphological features). The UD lemmas are the raw textual lemmas, so the discriminative numeric suffix of PDT is dropped. The dependency trees are converted according to the UD guidelines, adapting both the unlabeled trees and the dependency labels.
To compute the evaluation scores, we use the official CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies BIBREF9 evaluation script, which produces the following metrics:
[noitemsep,topsep=0pt]
UPOS – universal POS tags accuracy,
XPOS – language-specific POS tags accuracy,
UFeats – universal subset of morphological features accuracy,
Lemmas – lemmatization accuracy,
UAS – unlabeled attachment score, LAS – labeled attachment score,
MLAS – morphology-aware LAS, BLEX – bi-lexical dependency score.
Datasets ::: Czech Named Entity Corpus
The Czech Named Entity Corpus 1.1 BIBREF13, BIBREF14 is a corpus of $5\,868$ Czech sentences with manually annotated $33\,662$ Czech named entities, classified according to a two-level hierarchy of 62 named entities.
The Czech Named Entity Corpus 2.0 BIBREF15 contains $8\,993$ Czech sentences with manually annotated $35\,220$ Czech named entities, classified according to a two-level hierarchy of 46 named entities.
We evaluate the NER task with the official CNEC evaluation script. Similarly to previous literature BIBREF13, BIBREF12 etc., the script only evaluates the first round annotation classes for the CNEC 1.1. For the CNEC 2.0, the script evaluates all annotated classes.
Neural Architectures
All our neural architectures are recurrent neural networks (RNNs). The POS tagging, lemmatization and dependency parsing is performed with the UDPipe 2.0 (Section SECREF16) and NER is performed with our new sequence-to-sequence model (Section SECREF36).
Neural Architectures ::: POS Tagging, Lemmatization, and Dependency Parsing
We perform POS tagging, lemmatization and dependency parsing using UDPipe 2.0 BIBREF19, one of the three winning systems of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies BIBREF9 and an overall winner of The 2018 Shared Task on Extrinsic Parser Evaluation BIBREF20. An overview of this architecture is presented in Figure FIGREF17 and the full details of the architecture and the training procedure are available in BIBREF19.
Neural Architectures ::: POS Tagging, Lemmatization, and Dependency Parsing ::: POS Tagging and Lemmatization
The tagger employs a standard bi-LSTM architecture. After embedding input words, three bidirectional LSTM BIBREF21 layers are performed, followed by a softmax output layers for POS tags and lemmas. While a classification output layer is natural for POS tags, we also apply it to lemmatization and generate lemmas by classifying the input words into lemma generation rules, therefore considering lemmatization as another tagging task.
We construct a lemma generation rule from a given form and lemma as follows:
[noitemsep,topsep=0pt]
We start by finding the longest continuous substring of the form and the lemma. If it is empty, we use the lemma itself as the class.
If there is a common substring of the form and the lemma, we compute the shortest edit script converting the prefix of the form into the prefix of the lemma, and the shortest edit script converting the suffix of the form to the suffix of the lemma. The edit scripts permit the operations delete_current_char and insert_char(c).
All above operations are performed case insensitively. To indicate correct casing of the lemma, we consider the lemma to be a concatenation of segments, where each segment is composed of either a sequence of lowercase characters, or a sequence of uppercase characters. We represent the lemma casing by encoding the beginning of every such segment, where the offsets in the first half of the lemma are computed relatively to the start of the lemma, and the offsets in the second half of the lemma are computed relatively to the end of the lemma.
Neural Architectures ::: POS Tagging, Lemmatization, and Dependency Parsing ::: Dependency Parsing
The dependency parsing is again predicted using UDPipe 2.0 architecture. After embedding input words, three bidirectional LSTM BIBREF21 layers are again performed, followed by a biaffine attention layer BIBREF22 producing labeled dependency trees.
In our evaluation we do not utilize gold POS tags and lemmas on the test set for dependency parsing. Instead, we consider three ways of employing them during parsing:
[noitemsep,topsep=0pt]
not using them at all;
adding predicted POS tags and lemmas on input;
perform joint training of POS tags, lemmatization, and dependency parsing. In this case, we share first two bidirectional LSTM layers between the tagger and the parser.
Neural Architectures ::: POS Tagging, Lemmatization, and Dependency Parsing ::: Input Embeddings
In our baseline model, we use the end-to-end word embeddings and also character-level word embeddings (bidirectional GRUs, BIBREF23, BIBREF24, BIBREF25 of dimension 256) trained specifically for the task.
Our architecture can optionally employ the following additional inputs
[noitemsep,topsep=0pt]
pretrained word embeddings (WE): For the PDT experiments, we generate the word embeddings with word2vec on a concatenation of large raw Czech corpora available from the LINDAT/CLARIN repository. For UD Czech, we use FastText word embeddings BIBREF27 of dimension 300, which we pretrain on Czech Wikipedia using segmentation and tokenization trained from the UD data.
BERT BIBREF1: Pretrained contextual word embeddings of dimension 768 from the Base model. We average the last four layers of the BERT model to produce the embeddings. Because BERT utilizes word pieces, we decompose UD words into appropriate subwords and then average the generated embeddings over subwords belonging to the same word.
Flair BIBREF2: Pretrained contextual word embeddings of dimension 4096.
Neural Architectures ::: POS Tagging, Lemmatization, and Dependency Parsing ::: POS Tags and Lemmas Decoding
Optionally, we employ a morphological dictionary MorfFlex BIBREF28 during decoding. If the morphological dictionary is used, it may produce analyses for an input word as (POS tag, lemma) pairs. If any are generated, we choose the pair with maximum likelihood given by both the POS tag and lemmatization model.
Neural Architectures ::: Named Entity Recognition
We use a novel approach BIBREF29 for nested named entity recognition (NER) to capture the nested entities in the Czech Named Entity Corpus. The nested entities are encoded in a sequence and the problem of nested NER is then viewed as a sequence-to-sequence (seq2seq) problem, in which the input sequence consists of the input tokens (forms) and the output sequence of the linearized entity labels.
The system is a encoder-decoder architecture. The encoder is a bi-directional LSTM and the decoder is a LSTM. The encoded labels are predicted one by one by the decoder, until the decoder outputs the "<eow>" (end of word) label and moves to the next token. We use a hard attention on the word whose label(s) is being predicted.
We train the network using the lazy variant of the Adam optimizer BIBREF30, which only updates accumulators for variables that appear in the current batch, with parameters $\beta _1=0.9$ and $\beta _2=0.98$. We use mini-batches of size 8. As a regularization, we apply dropout with rate $0.5$ and the word dropout replaces $20\%$ of words by the unknown token to force the network to rely more on context. We did not perform any complex hyperparameter search.
In this model, we use the following word- and character-level word embeddings:
[noitemsep,topsep=0pt]
pretrained word embeddings: We use the FastText BIBREF27 word embeddings of dimension 300 from the publicly available Czech model.
end-to-end word embeddings: We embed the input forms and lemmas (256 dimensions) and POS tags (one-hot).
end-to-end character-level word embeddings: We use bidirectional GRUs BIBREF23, BIBREF24 of dimension 128 in line with BIBREF25: we represent every Unicode character with a vector of dimension 128, and concatenate GRU outputs for forward and reversed word characters.
Optionally, we add the BERT BIBREF1 and the Flair BIBREF2 contextualized embeddings in the same way as in the UDPipe 2.0 (Section SECREF16).
Results ::: POS Tagging and Lemmatization on PDT 3.5
The POS tagging and lemmatization results are presented in Table TABREF44. The word2vec word embeddings (WE) considerably increase performance compared to the baseline, especially in POS tagging. When only Flair embeddings are added to the baseline, we also observe an improvement, but not as high. We hypothesise that the lower performance (in contrast with the results reported in BIBREF2) is caused by the size of the training data, because we train the word2vec WE on considerably larger dataset than the Czech Flair model. However, when WE and Flair embeddings are combined, performance moderately increases, demonstrating that the two embedding methods produce at least partially complementary representations.
The BERT embeddings alone bring highest improvement in performance. Furthermore, combination with WE or Flair again yields performance increase. The best results are achieved by exploiting all three embedding methods, substantially exceeding state-of-the-art results.
Utilization of morphological dictionary improves prediction accuracy. However, as the performance of a model itself increases, the gains obtained by the morphological dictionary diminishes – for a model without any pretrained embeddings, morphological dictionary improves POS tagging by and lemmatization by $0.43\%$ and $0.45\%$, while the best performing model gains only $0.11\%$ and $0.23\%$.
Results ::: Dependency Parsing on PDT 3.5
The evaluation of the contextualized embeddings methods as well as various ways of POS tag utilization is presented in Table TABREF44. Without POS tags and lemmas, the Flair embeddings bring only a slight improvement in dependency parsing when added to WE. In contrast, BERT embeddings employment results in substantial gains, increasing UAS and LAS by 1.6% and 2.1%. A combination of BERT and Flair embeddings does not result in any performance improvement, demonstrating that BERT syntactic representations encompass the Flair embeddings.
When introducing POS tags and lemmas predicted by the best model from Section SECREF43 as inputs for dependency parsing, the performance increases only slightly. A better way of POS tags and lemmas exploitation is achieved in a joint model, which predicts POS tags, lemmas, and dependency trees simultaneously. Again, BERT embeddings bring significant improvements, but in contrast to syntax parsing only, adding Flair embeddings to BERT results in moderate gain – we hypothesise that the increase is due to the complementary morphological information present in Flair embeddings (cf. Section SECREF43). Note that the joint model achieves better parsing accuracy than the one given gold POS tags and lemmas on input. However, the POS tags and lemmas predicted by the joint model are of slightly lower quality compared to a standalone tagger of the best configuration from Section SECREF43.
Table TABREF44 compares our best model with state-of-the-art results on PDT 2.0 (note that some of the related work used only a subset of PDT 2.0 and/or utilized gold morphological annotation). To our best knowledge, research on PDT parsing was performed mostly in the first decade of this century, therefore even our baseline model substantially surpasses previous works. Our best model with contextualized embeddings achieves nearly 50% error reduction both in UAS and LAS.
Results ::: POS Tagging, Lemmatization and Dependency Parsing on Universal Dependencies
Table TABREF47 shows the performance of analyzed embedding methods in a joint model performing POS tagging, lemmatization, and dependency parsing on Czech PDT UD 2.3 treebank. This treebank is derived from PDT 3.5 a-layer, with original POS tags kept in XPOS, and the dependency trees and lemmas modified according to UD guidelines.
We observe that the word2vec WEs perform similarly to Flair embeddings in this setting. Our hypothesis is that the word2vec WEs performance loss (compared to WEs in Section SECREF43) is caused by using a considerably smaller raw corpus to pretrain the WEs (Czech Wikipedia with 785M words, compared to 4G words used in Section SECREF43), due to licensing reasons. BERT embeddings once more deliver the highest improvement, especially in dependency parsing, and our best model employs all three embedding methods.
In the previous ablation experiments, we used the gold segmentation and tokenization in the Czech PDT UD 2.3 treebank. For comparison with state of the art, Czech PDT UD 2.2 treebank without gold segmentation and tokenization is used in evaluation, according to the CoNLL 2018 shared task training and evaluation protocol. Our system reuses segmentation and tokenization produced by UDPipe 2.0 in the CoNLL 2018 shared task and surpasses previous works substantially in all metrics (bottom part of Table TABREF47).
Comparing the results with a joint tagging and parsing PDT 3.5 model from Table TABREF7, we observe that the XPOS results are nearly identical as expected. Lemmatization on the UD treebank is performed without the discriminative numeric suffixes (see Section SECREF3) and therefore reaches better performance. Both UAS and LAS are also better on the UD treebank, which we assume is caused by the different annotation scheme.
Results ::: Named Entity Recognition
Table TABREF47 shows NER results (F1 score) on CNEC 1.1 and CNEC 2.0. Our sequence-to-sequence (seq2seq) model which captures the nested entities, clearly surpasses the current Czech NER state of the art. Furthermore, significant improvement is gained when adding the contextualized word embeddings (BERT and Flair) as optional input to the LSTM encoder. The strongest model is a combination of the sequence-to-sequence architecture with both BERT and Flair contextual word embeddings.
Conclusion
We have presented an evaluation of two contextualized embeddings methods, namely BERT and Flair. By utilizing these embeddings as input to deep neural networks, we have achieved state-of-the-art results in several Czech text processing tasks, namely in POS tagging, lemmatization, dependency parsing and named entity recognition.
Acknowledgements
The work described herein has been supported by OP VVV VI LINDAT/CLARIN project (CZ.02.1.01/0.0/0.0/16_013/0001781) and it has been supported and has been using language resources developed by the LINDAT/CLARIN project (LM2015071) of the Ministry of Education, Youth and Sports of the Czech Republic.

Question: What previous approaches did this method outperform?
Answer: Table TABREF44, Table TABREF44, Table TABREF47, Table TABREF47

Question: How big is the Universal Dependencies corpus?
Answer: Unanswerable

Question: What data is the Prague Dependency Treebank built on?
Answer: Unanswerable

Question: What data is used to build the embeddings?
Answer: large raw Czech corpora available from the LINDAT/CLARIN repository, Czech Wikipedia