CoNLL 2018 Shared Task: Multilingual Parsing
from Raw Text to Universal Dependencies
Daniel Zeman1 , Jan Hajič1 , Martin Popel1 , Martin Potthast2 ,Milan Straka1 , Filip Ginter3 , Joakim Nivre4 , and Slav Petrov5
1

Charles University, Faculty of Mathematics and Physics
2
Universität Leipzig, 3 University of Turku
4
Uppsala University, 5 Google AI Language{zeman|hajic|popel|straka}@ufal.mff.cuni.cz,
martin.potthast@uni-leipzig.de, figint@utu.fi,
joakim.nivre@lingfil.uu.se, slav@google.com

Abstractand 2007 (Buchholz and Marsi, 2006; Nivre et al.,2007) and evolved it in two ways: (1) the parsing process started from raw text rather than gold
standard tokenization and part-of-speech tagging,and (2) the syntactic representations were consistent across languages thanks to the Universal Dependencies framework (Nivre et al., 2016). The
2018 CoNLL shared task on universal dependency
parsing starts from the same premises but adds a
focus on morphological analysis as well as datafrom new languages.

Every year, the Conference on Computational Natural Language Learning(CoNLL) features a shared task, in which
participants train and test their learning
systems on the same data sets. In 2018,
one of two tasks was devoted to learning dependency parsers for a large number of languages, in a real-world setting
without any gold-standard annotation on
the input. All test sets followed the unified annotation scheme of Universal Dependencies (Nivre et al., 2016). This
shared task constitutes a 2nd edition—thefirst one took place in 2017 (Zeman et al.,2017); the main metric from 2017 was
kept, allowing for easy comparison, andtwo new main metrics were introduced.
New datasets added to the Universal Dependencies collection between mid-2017
and the spring of 2018 contributed to theincreased difficulty of the task this year.
In this overview paper, we define the task
and the updated evaluation methodology,
describe data preparation, report and analyze the main results, and provide a brief
categorization of the different approachesof the participating systems.

1

Like last year, participating systems minimally
had to find labeled syntactic dependencies between words, i.e., a syntactic head for each word,
and a label classifying the type of the dependency
relation. In addition, this year’s task featured new
metrics that also scored a system’s capacity to predict a morphological analysis of each word, including a part-of-speech tag, morphological features, and a lemma. Regardless of metric, the assumption was that the input should be raw text,
with no gold-standard word or sentence segmentation, and no gold-standard morphological annotation. However, for teams who wanted to concentrate on one or more subtasks, segmentationand morphology predicted by the baseline UDPipe
system (Straka et al., 2016) was made available
just like last year.
There are eight new languages this year:
Afrikaans, Armenian, Breton, Faroese, Naija, Old
French, Serbian, and Thai; see Section 2 for moredetails. The two new evaluation metrics are described in Section 3.

Introduction

The 2017 CoNLL shared task on universal dependency parsing (Zeman et al., 2017) picked up thethread from the influential shared tasks in 2006
1

Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 1–21
Brussels, Belgium, October 31 – November 1, 2018. c 2018 Association for Computational Linguistics
https://doi.org/10.18653/v1/K18-2001

2

Data

from different sources, often also from different
domains.3 See Table 1 for an overview.
61 treebanks contain designated development
data. Participants were asked not to use it
for training proper but only for evaluation, development, tuning hyperparameters, doing error
analysis etc. Seven treebanks have reasonablysized training data but no development data;
only two of them, Irish and North Sámi, arethe sole treebanks of their respective languages.
For those treebanks cross-validation had to be
used during development, but the entire dataset
could be used for training once hyperparameters were determined. Five treebanks consistof extra test sets: they have no training or development data of their own, but large training data exist in other treebanks of the samelanguages (Czech-PUD, English-PUD, FinnishPUD, Japanese-Modern and Swedish-PUD, respectively). The remaining nine treebanks are
low-resource languages. Their “training data” waseither a tiny sample of a few dozen sentences (Armenian, Buryat, Kazakh, Kurmanji, Upper Sorbian), or there was no training data at all (Breton,
Faroese, Naija, Thai). Unlike in the 2017 task,these languages were not “surprise languages”,
that is, the participants knew well in advance what
languages to expect. The last two languages are
particularly difficult: Naija is a pidgin spoken in
Nigeria; while it can be expected to bear somesimilarity to English, its spelling is significantly
different from standard English, and no resourceswere available to learn it. Even harder was Thaiwith a writing system that does not separate words
by spaces; the Facebook word vectors were probably the only resource among the approved additional data where participants could learn something about words in Thai (Rosa and Mareček,
2018; Smith et al., 2018). It was also possible toexploit the fact that there is a 1-1 sentence mapping between the Thai test set and the other four
PUD test sets.4
Participants received the training and development data with gold-standard tokenization, sentence segmentation, POS tags and dependency re-In general, we wanted the participating systems tobe able to use any data that is available free ofcharge for research and educational purposes (sothat follow-up research is not obstructed). We deliberately did not place upper bounds on data sizes(in contrast to e.g. Nivre et al. (2007)), despite thefact that processing large amounts of data may bedifficult for some teams. Our primary objective
was to determine the capability of current parsers
provided with large amounts of freely available
data.
In practice, the task was formally closed, i.e.,
we listed the approved data resources so that all
participants were aware of their options. However, the selection was rather broad, ranging from
Wikipedia dumps over the OPUS parallel corpora(Tiedemann, 2012) to morphological transducers.
Some of the resources were proposed by the participating teams.
We provided dependency-annotated training
and test data, and also large quantities of crawled
raw texts. Other language resources are available
from third-party servers and we only referred to
the respective download sites.
2.1

Training Data: UD 2.2

Training and development data came from the
Universal Dependencies (UD) 2.2 collection(Nivre et al., 2018). This year, the official UD release immediately followed the test phase of theshared task. The training and development data
were available to the participating teams as a prerelease; these treebanks were then released exactly
in the state in which they appeared in the task.1
The participants were instructed to only use the
UD data from the package released for the sharedtask. In theory, they could locate the (yet unreleased) test data in the development repositories
on GitHub, but they were trusted that they would
not attempt to do so.
82 UD treebanks in 57 languages were includedin the shared task;2 however, nine of the smaller
treebanks consisted solely of test data, with nodata at all or just a few sentences available fortraining. 16 languages had two or more treebanks

3
We distinguish treebanks of the same language by theirshort names or acronyms. Hence, the two treebanks of Ancient Greek are identified as Perseus and PROIEL, the threetreebanks of Latin are ITTB, Perseus and PROIEL, etc.
4
While the test datasets were not available to the teams
when they developed their systems, the documentation of thetreebanks was supplied together with the training data, hence
the teams could learn that the PUD treebanks were parallel.

1

UD 2.2 also contains other treebanks that were not included in the task for various reasons, and that may have beenfurther developed even during the duration of the task.
2Compare with the 81 treebanks and 49 languages in the2017 task.

2

Language
Tbk Code
2017
TrWrdsAfrikaans
af afribooms NA
34 K
Ancient Greek grc perseus grc
160 K
Ancient Greek grc proiel
grc proiel
187 K
Arabicar padtar
224 K
Armenianhy armtdp
NA
1K
Basqueeu bdteu
73 K
Bretonbr kebNA
0K
Bulgarianbg btbbg
124 K
Buryatbxr bdt
bxr
0K
Catalanca ancoraca
418 K
Chinesezh gsdzh
97 K
Croatianhr set
hr
154 K
Czech
cs caccs cac473 K
Czech
cs fictree
NA
134 K
Czech
cs pdtcs
1,173 KCzech
cs pudcs pud0K
Danishda ddt
da
80 K
Dutch
nl alpino
nl
186 K
Dutch
nl lassysmall nl lassysmall
75 K
Englishen ewten
205 K
Englishen gumNA
54 K
Englishen linesen lines
50 K
Englishen puden pud0K
Estonianet edtet
288 K
Faroesefo oftNA
0K
Finnishfi ftbfi ftb128 K
Finnishfi pudfi pud0K
Finnishfi tdtfi
163 K
Frenchfr gsdfr
357 K
Frenchfr sequoia
fr sequoia
51 K
Frenchfr spoken
NA
15 K
Galiciangl ctg
gl
79 K
Galician
gl treegal
gl treegal
15 K
Germande gsdde
264 K
Gothicgot proiel
got
35 K
Greek
el gdtel
42 K
Hebrew
he htb
he
138 K
Hindi
hi hdtbhi
281 K
Hungarianhu szegedhu
20 K
Indonesianid gsd
id
98 K
Irish
ga idtga
14 K

Language
Tbk Code
2017
TrWrds
Italianit isdt
it276 K
Italianit postwita
NA
99 K
Japaneseja gsdja
162 K
Japanese
ja modern
NA
0K
Kazakhkk ktbkk
1K
Koreanko gsdko
57 K
Koreanko kaist
NA
296 K
Kurmanji
kmr mg
kmr
0K
Latin
la ittb
la ittb
270 K
Latin
la perseus
la
18 K
Latin
la proiel
la proiel
172 K
Latvianlv lvtb
lv
81 K
Naija
pcm nsc
NA
0K
North Sámisme giella
sme
17 K
Norwegianno bokmaal no bokmaal
244 K
Norwegianno nynorsk
no nynorsk
245 K
Norwegianno nynorsklia NA
4K
Old Church Slavonic cu proielcu
37 K
Old French
fro srcmf
NA
136 K
Persianfa serajifa
121 K
Polishpl lfgNA
105 K
Polish
pl sz
pl
63 K
Portuguese
pt bosquept
207 K
Romanianro rrtro
185 K
Russianru syntagrus ru syntagrus
872 K
Russianru taiga
NA
10 K
Serbiansr setNA
66 K
Slovaksk snksk
81 K
Sloveniansl ssjsl
113 K
Sloveniansl sstsl sst19 K
Spanishes ancoraes ancora
445 K
Swedishsv lines
sv lines
48 K
Swedishsv pudsv pud0K
Swedishsv talbanken sv
67 K
Thai
th pudNA
0K
Turkishtr imsttr
38 K
Ukrainian
uk iu
uk
75 K
Upper Sorbian
hsb ufal
hsb
0K
Urdu
ur udtbur
109 K
Uyghurug udtug
19 K
Vietnamesevi vtbvi
20 K

Table 1: Overview of the 82 test treebanks. TbkCode = Treebank identifier, consisting of the ISO 639
language code followed by a treebank-specific code. 2017 = Code of the corresponding treebank in
the 2017 task if applicable (“NA” otherwise). TrWrds = Size of training data, rounded to the nearest
thousand words.
lations; and for most languages also lemmas andmorphological features.
Cross-domain and cross-language training wasallowed and encouraged. Participants were free totrain models on any combination of the training
treebanks and apply it to any test set.
2.2

sentences and 90 G words in 45 languages areavailable in CoNLL-U format (Ginter et al., 2017);
the per-language sizes of the corpus are listed inTable 2.

See Zeman et al. (2017) for more details on how
the raw texts and embeddings were processed.
Note that the resource was originally prepared forthe 2017 task and it was not extended to include
the eight new languages; however, some of the
new languages are covered by the word vectors
provided by Facebook (Bojanowski et al., 2016)
and approved for the shared task.

Supporting Data

To enable the induction of custom embeddings andthe use of semi-supervised methods in general,the participants were provided with supporting resources primarily consisting of large text corpora
for many languages in the task, as well as embeddings pre-trained on these corpora. In total, 5.9 M
3Language
English (en)German (de)Portuguese (pt)Spanish (es)French (fr)
Polish (pl)Indonesian (id)Japanese (ja)
Italian (it)Vietnamese (vi)Turkish (tr)Russian (ru)Swedish (sv)Dutch (nl)Romanian (ro)Czech (cs)
Hungarian (hu)Danish (da)Chinese (zh)Norwegian-Bokmål (no)Persian (fa)Finnish (fi)Arabic (ar)Catalan (ca)Slovak (sk)Greek (el)Hebrew (he)Croatian (hr)Ukrainian (uk)
Korean (ko)Slovenian (sl)Bulgarian (bg)
Estonian (et)Latvian (lv)
Galician (gl)Latin (la)Basque (eu)
Hindi (hi)Norwegian-Nynorsk (no)Kazakh (kk)Urdu (ur)Irish (ga)Ancient Greek (grc)
Uyghur (ug)Kurdish (kmr)
Upper Sorbian (hsb)
Buryat (bxr)
North Sámi (sme)
Old Church Slavonic (cu)
Total

Words
9,441 M
6,003 M
5,900 M
5,721 M
5,242 M
5,208 M
5,205 M
5,179 M
5,136 M
4,066 M
3,477 M
3,201 M
2,932 M
2,914 M
2,776 M
2,005 M
1,624 M
1,564 M
1,530 M
1,305 M
1,120 M
1,008 M
963 M
860 M
811 M
731 M
615 M
583 M
538 M
527 M
522 M
370 M
328 M
276 M
262 M
244 M
155 M
91 M
76 M
54 M
46 M
24 M
7M
3M
3M
2M
413 K
331 K
28 K
90,669 M

2.3

Test Data: UD 2.2

Each of the 82 treebanks mentioned in Section 2.1
has a test set. Test sets from two different treebanks of one language were evaluated separately
as if they were different languages. Every test set
contains at least 10,000 words (including punctuation marks). UD 2.2 treebanks that were smaller
than 10,000 words were excluded from the sharedtask. There was no upper limit on the test data;
the largest treebank had a test set comprising 170K
words. The test sets were officially released as apart of UD 2.2 immediately after the shared task.5

3

Evaluation Metrics

There are three main evaluation scores, dubbed
LAS, MLAS and BLEX. All three metrics reflect
word segmentation and relations between content
words. LAS is identical to the main metric of the2017 task, allowing for easy comparison; the othertwo metrics include part-of-speech tags, morphological features and lemmas. Participants whowanted to decrease task complexity could concentrate on improvements in just one metric; however,all systems were evaluated with all three metrics,
and participants were strongly encouraged to output all relevant annotation, even if they just copy
values predicted by the baseline model.
When parsers are applied to raw text, the metricmust be adjusted to the possibility that the number of nodes in gold-standard annotation and in
the system output vary. Therefore, the evaluation
starts with aligning system nodes and gold nodes.
A dependency relation cannot be counted as correct if one of the nodes could not be aligned to agold node. See Section 3.4 and onward for moredetails on alignment.
The evaluation software is a Python script thatcomputes the three main metrics and a number of
additional statistics. It is freely available for download from the shared task website.6
3.1

LAS: Labeled Attachment Score

The standard evaluation metric of dependency
parsing is the labeled attachment score (LAS), i.e.,
the percentage of nodes with correctly assigned
reference to the parent node, including the label(type) of the relation. For scoring purposes, only

Table 2: Supporting data overview: Number ofwords (M = million; K = thousand) for each language.

5

http://hdl.handle.net/11234/1-2837
http://universaldependencies.org/
conll18/conll18_ud_eval.py
6

4

Content

Function
Ignorednsubj, obj, iobj, csubj, ccomp, xcomp, obl, vocative, expl,
dislocated, advcl, advmod, discourse, nmod, appos, nummod,
acl, amod, conj, fixed, flat, compound, list, parataxis,
orphan, goeswith, reparandum, root, dep
aux, cop, mark, det, clf, case, cc
punct

Table 3: Universal dependency relations considered as pertaining to content words and function words,
respectively, in MLAS. Content word relations are evaluated directly; words attached via functional
relations are treated as features of their parent nodes.
Features

PronType, NumType, Poss, Reflex, Foreign, Abbr, Gender,
Animacy, Number, Case, Definite, Degree, VerbForm, Mood,
Tense, Aspect, Voice, Evident, Polarity, Person, Polite

Table 4: Universal features whose values are evaluated in MLAS. Any other features are ignored.

The idea behind MLAS is that function words
often correspond to morphological features inother languages. Furthermore, languages withmany function words (e.g., English) have longer
sentences than morphologically rich languages(e.g., Finnish), hence a single error in Finnish
costs the parser significantly more than an error
in English according to LAS.
The core part is identical to LAS (Section 3.1):
for aligned system and gold nodes, their respective parent nodes are considered; if the systemparent is not aligned with the gold parent, or if
the universal relation label differs, the word is notcounted as correctly attached. Unlike LAS, certain types of relations (Table 3) are not evaluateddirectly. Words attached via such relations (in either system or gold data) are not counted as independent words. Instead, they are treated as features of the content words they belong to. Therefore, a system-produced word counts as correct if
it is aligned and attached correctly, its universal
POS tag and selected morphological features (Table 4) are correct, all its function words are attached correctly, and their POS tags and features
are also correct. Punctuation nodes are neither
content nor function words; their attachment is ignored in MLAS.

universal dependency labels were taken into account, which means that language-specific subtypes such as expl:pv (pronoun of a pronominal verb), a subtype of the universal relation expl
(expletive), were truncated to expl both in thegold standard and in the system output before
comparing them.
In the end-to-end evaluation of our task, LAS isre-defined as the harmonic mean (F1 ) of precision
P and recall R, whereP =#correctRelations#systemNodes

(1)

R=

#correctRelations#goldNodes

(2)

2P R
P +R

(3)

LAS =

Note that attachment of all nodes including punctuation is evaluated. LAS is computed separately
for each of the 82 test files and a macro-average of
all these scores is used to rank the systems.
3.2

MLAS: Morphology-Aware Labeled
Attachment Score

MLAS aims at cross-linguistic comparability ofthe scores. It is an extension of CLAS (Nivre andFang, 2017), which was tested experimentally inthe 2017 task. CLAS focuses on dependencies between content words and disregards attachment offunction words; in MLAS, function words are notignored, but they are treated as features of content
words. In addition, part-of-speech tags and morphological features are evaluated, too.

3.3

BLEX: Bilexical Dependency Score

BLEX is similar to MLAS in that it focuses on
relations between content words. Instead of morphological features, it incorporates lemmatizationin the evaluation. It is thus closer to semantic
content and evaluates two aspects of UD annota5

are involved. Fortunately, the CoNLL-U file format used in UD clearly marks all MWTs so we
can detect them both in system output and in gold
data. Whenever one or more MWTs have overlapping spans of surface character offsets, the longest
common subsequence algorithm is used to alignsyntactic words within these spans.

tion that are important for language understanding: dependencies and lexemes. The inclusion ofthis metric should motivate the competing teams
to model lemmas, the last important piece of annotation that is not captured by the other metrics.
A system that scores high in all three metrics willthus be a general-purpose language-analysis toolthat tackles segmentation, morphology and surface syntax.
Computation of BLEX is analogous to LAS and
MLAS. Precision and recall of correct attachments
is calculated, attachment of function words andpunctuation is ignored (Table 3). An attachment
is correct if the parent and child nodes are aligned
to the corresponding nodes in gold standard, if theuniversal dependency label is correct, and if thelemma of the child node is correct.
A few UD treebanks lack lemmatization (or, asin Uyghur, have lemmas only for some words andnot for others). A system may still be able topredict the lemmas if it learns them in other treebanks. Such system should not be penalized justbecause no gold standard is available; therefore,
if the gold lemma is a single underscore character(“ ”), any system-produced lemma is consideredcorrect.
3.4

3.6

Words are aligned and dependencies are evaluatedin the entire file without considering sentence segmentation. Still, the accuracy of sentence boundaries has an indirect impact on attachment scores:
any missing or extra sentence boundary necessarily makes one or more dependency relations incorrect.
3.7

Invalid Output

If a system fails to produce one of the 82 files orif the file is not valid CoNLL-U format, the scoreof that file (counting towards the system’s macroaverage) is zero.
Formal validity is defined more leniently thanfor UD-released treebanks. For example, a nonexistent dependency type does not render thewhole file invalid, it only costs the system one incorrect relation. However, cycles and multi-root
sentences are disallowed. A file is also invalid
if there are character mismatches that could makethe token-alignment algorithm fail.

Token Alignment

UD defines two levels of token/word segmentation. The lower level corresponds to what is usually understood as tokenization. However, unlike
some popular tokenization schemes, it does notinclude any normalization of the non-whitespacecharacters. We can safely assume that any two tokenizations of a text differ only in whitespace while
the remaining characters are identical. There isthus a 1-1 mapping between gold and system nonwhitespace characters, and two tokens are aligned
if all their characters match.
3.5

Sentence Segmentation

3.8

Extrinsic Parser Evaluation

The metrics described above are all intrinsic measures: they evaluate the grammatical analysis task
per se, with the hope that better scores correspond to output that is more useful for downstream
NLP applications. Nevertheless, such correlations
are not automatically granted. We thus seek tocomplement our task with an extrinsic evaluation,
where the output of parsing systems is exploited
by applications like biological event extraction,
opinion analysis and negation scope resolution.
This optional track involves English only. It
is organized in collaboration with the EPE initiative;7 for details see Fares et al. (2018).

Syntactic Word Alignment

The higher segmentation level is based on the notion of syntactic word. Some languages containmulti-word tokens (MWT) that are regarded ascontractions of multiple syntactic words. For example, the German token zum is a contraction of
the preposition zu “to” and the article dem “the”.
Syntactic words constitute independent nodes independency trees. As shown by the example, it
is not required that the MWT is a pure concatenation of the participating words; the simple token alignment thus does not work when MWTs

4

TIRA: The System Submission
Platform

Similarly to our 2017 task and to some other recent CoNLL shared tasks, we employed the cloud7

6

http://epe.nlpl.eu/

based evaluation platform TIRA (Potthast et al.,2014),8 which implements the evaluation as a service paradigm (Hanbury et al., 2015). Instead ofprocessing test data on their own hardware andsubmitting the outputs, participants submit working software. Naturally, software submissions
bring about additional overhead for both organizers and participants, whereas the goal of an evaluation platform like TIRA is to reduce this overhead
to a bearable level.

dition for as long as possible. TIRA supplies
participants with a virtual machine, offering a
range of commonly used operating systems. Once
deployed and tested, the virtual machines arearchived to preserve the software within.
In addition, some participants agreed to share
their code so that we decided to collect the respective projects in an open source repository hosted
on GitHub.9

5
4.1

Blind Evaluation

We prepared a set of baseline models using UDPipe 1.2 (Straka and Straková, 2017).
The baseline models were released together
with the UD 2.2 training data. For each of the
73 treebanks with non-empty training data wetrained one UDPipe model, utilizing training data
for training and development data for hyperparameter tuning. If a treebank had no development data,
we cut 10% of the training sentences and considered it as development data for the purpose of tuning hyperparameters of the baseline model (employing only the remainder of the original training
data for the actual training in that case).
In addition to the treebank-specific models, we
also trained a “mixed model” on samples from all
treebanks. Specifically, we utilized the first 200
training sentences of each treebank (or less in caseof small treebanks) as training data, and at most 20
sentences from each treebank’s development set as
development data.
The baseline models, together with all information needed to replicate them (hyperparameters,
the modified train-dev split where applicable, andpre-computed word embeddings for the parser) areavailable from http://hdl.handle.net/11234/
1-2859.
Additionally, the released archive also contains
the training and development data with predicted
morphology. Morphology in development data
was predicted using the baseline models, morphology in training data via “jack-knifing” (split thetraining set into 10 parts, train a model on 9 parts,
use it to predict morphology in the tenth part, repeat for all 10 target parts). The same hyperparameters were used as those used to train the baselinemodel on the entire training set.
The UDPipe baseline models are able to reconstruct nearly all annotation from CoNLL-U files– they can generate segmentation, tokenization,

Traditionally, evaluations in shared tasks are halfblind (the test data are shared with participants
while the ground truth is withheld). TIRA enables fully blind evaluation, where the software islocked in a datalock together with the test data, its
output is recorded but all communication channels
to the outside are closed or tightly moderated. The
participants do not even see the input to their software. This feature of TIRA was not too important in the present task, as UD data is not secret,and the participants were simply trusted that theywould not exploit any knowledge of the test data
they might have access to.
However, closing down all communication
channels also has its downsides, since participantscannot check their running software; before thesystem run completes, even the task moderator
does not see whether the system is really producing output and not just sitting in an endlessloop. In order to alleviate this extra burden, we
made two modifications compared to the previous year: 1. Participants were explicitly advised
to invoke shorter runs that process only a subsetof the test files. The organizers would then stitch
the partial runs into one set of results. 2. Participants were able to see their scores on the test set
rounded to the nearest multiple of 5%. This way
they could spot anomalies possibly caused by illselected models. The exact scores remained hidden because we did not want the participants tofine-tune their systems against the test data.
4.2

Replicability

It is desirable that published experiments can be
re-run yielding the same results, and that the algorithms can be tested on alternative test data in
the future. Ensuring both requires that a to-beevaluated software is preserved in working con8

Baseline System9

http://www.tira.io/7

https://github.com/CoNLL-UD-2018

Treebank without
training data
Breton KEB
Czech PUD
English PUD
Faroese OFT
Finnish PUD
Japanese Modern
Naija NSC
Swedish PUD
Thai PUD

Substitution
model
mixed model
Czech PDT
English EWT
mixed model
Finnish TDT
Japanese GSD
mixed model
Swedish Talbanken
mixed modelTeam
1. HIT-SCIR (Che et al.)2. TurkuNLP (Kanerva et al.)3. UDPipe Future (Straka)
LATTICE (Lim et al.)ICS PAS (Rybak and Wróblewska)
6. CEA LIST (Duthoo and Mesnard)
7. Uppsala (Smith et al.)Stanford (Qi et al.)
9. AntNLP (Ji et al.)NLP-Cube (Boros, et al.)
11. ParisNLP (Jawahar et al.)
12. SLT-Interactions (Bhat et al.)
13. IBM NY (Wan et al.)
14. UniMelb (Nguyen and Verspoor)
15. LeisureX (Li et al.)
16. KParse (Kırnap et al.)17. Fudan (Chen et al.)18. BASELINE UDPipe 1.2
19. Phoenix (Wu et al.)20. CUNI x-ling (Rosa and Mareček)
21. BOUN (Özateş et al.)
22. ONLP lab (Seker et al.)
23. iParse (no paper)
24. HUJI (Hershcovich et al.)
25. ArmParser (Arakelyan et al.)
26. SParse (Önder et al.)Table 5: Substitution models of the baseline systems for treebanks without training data.
multi-word token splitting, morphological annotation (lemmas, UPOS, XPOS and FEATS) and dependency trees. Participants were free to use any
part of the model in their systems – for all test sets,
we provided UDPipe processed variants in addition to raw text inputs.
Baseline UDPipe Shared Task System The
shared task baseline system employs the UDPipe
1.2 baseline models. For the nine treebanks without their own training data, a substitution modelaccording to Table 5 was used.

6

Results

6.1

Official Parsing Results

LAS
75.84
73.28
73.11
73.02
73.02
72.56
72.37
72.29
70.90
70.82
70.64
69.98
69.11
68.66
68.31
66.58
66.34
65.80
65.61
64.87
63.54
58.35
55.83
53.69
47.02
1.95

Table 6: Ranking of the participating systems by
the labeled attachment F1 -score (LAS), macroaveraged over 82 test sets. Pairs of systems withsignificantly (p < 0.05) different LAS are separated by a line. Citations refer to the corresponding system-description papers in this volume.

Table 6 gives the main ranking of participating
systems by the LAS F1 score macro-averaged overall 82 test files. The table also shows the performance of the baseline UDPipe system; 17 of the
25 systems managed to outperform it. The baseline is comparatively weaker than in the 2017 task
(only 12 out of 32 systems beat the baseline there).
The ranking of the baseline system by MLAS issimilar (Table 7) but in BLEX, the baseline jumps
to rank 13 (Table 8). Besides the simple explanation that UDPipe 1.2 is good at lemmatization, we
could also hypothesize that some teams put lesseffort in building lemmatization models (see also
the last column in Table 10).
Each ranking has a different winning system, although the other two winners are typically closely
following. The same 8–10 systems occupy best
positions in all three tables, though with variablemutual ranking. Some teams seem to have deliberately neglected some of the evaluated attributes:
Uppsala is rank 7 in LAS and MLAS, but 24 in

BLEX; IBM NY is rank 13 in LAS but 24 in
MLAS and 23 in BLEX.
While the LAS scores on individual treebanks
are comparable to the 2017 task, the macro average is not, because the set of treebanks is different,
and the impact of low-resource languages seems tobe higher in the present task.
We used bootstrap resampling to compute 95%
confidence intervals: they are in the range ±0.11
to ±0.16 (% LAS/MLAS/BLEX) for all systems
except SParse (where it is ±0.00).
8

Team
MLAS
1. UDPipe Future (Praha)
61.25
2. TurkuNLP (Turku)
60.99
Stanford (Stanford)
60.92
4. ICS PAS (Warszawa)
60.25
5. CEA LIST (Paris)
59.92
6. HIT-SCIR (Harbin)
59.78
7. Uppsala (Uppsala)
59.20
8. NLP-Cube (Bucureşti)
57.32
9. LATTICE (Paris)
57.01
10. AntNLP (Shanghai)
55.92
11. ParisNLP (Paris)
55.74
12. SLT-Interactions (Bengaluru) 54.52
13. LeisureX (Shanghai)
53.70
UniMelb (Melbourne)
53.62
15. KParse (İstanbul)
53.25
16. Fudan (Shanghai)
52.69
17. BASELINE UDPipe 1.2
52.42
Phoenix (Shanghai)
52.26
19. BOUN (İstanbul)
50.40
CUNI x-ling (Praha)
50.35
21. ONLP lab (Ra’anana)
46.09
22. iParse (Pittsburgh)
45.65
23. HUJI (Yerushalayim)
44.60
24. IBM NY (Yorktown Heights) 40.61
25. ArmParser (Yerevan)
36.28
26. SParse (İstanbul)
1.68

Team
BLEX
1. TurkuNLP (Turku)
66.09
2. HIT-SCIR (Harbin)
65.33
3. UDPipe Future (Praha)
64.49
ICS PAS (Warszawa)
64.44
5. Stanford (Stanford)
64.04
6. LATTICE (Paris)
62.39
CEA LIST (Paris)
62.23
8. AntNLP (Shanghai)
60.91
9. ParisNLP (Paris)
60.70
10. SLT-Interactions (Bengaluru) 59.68
11. UniMelb (Melbourne)
58.67
12. LeisureX (Shanghai)
58.42
13. BASELINE UDPipe 1.2
55.80
Phoenix (Shanghai)
55.71
15. NLP-Cube (Bucureşti)
55.52
16. KParse (İstanbul)
55.26
17. CUNI x-ling (Praha)
54.07
Fudan (Shanghai)
54.03
19. BOUN (İstanbul)
53.45
20. iParse (Pittsburgh)
48.71
21. HUJI (Yerushalayim)
48.05
22. ArmParser (Yerevan)
39.18
23. IBM NY (Yorktown Heights) 32.55
24. Uppsala (Uppsala)
32.09
25. ONLP lab (Ra’anana)
28.29
26. SParse (İstanbul)
1.71

Table 7: Ranking of the participating systems by
MLAS, macro-averaged over 82 test sets. Pairsof systems with significantly (p < 0.05) different
MLAS are separated by a line.

Table 8: Ranking of the participating systems by
BLEX, macro-averaged over 82 test sets. Pairsof systems with significantly (p < 0.05) different
BLEX are separated by a line.

We used paired bootstrap resampling to compute whether the difference between two neighboring systems is significant (p < 0.05).10
6.2

marked by their teams as primary runs, or wereeven run after the official evaluation phase closed
and test data were unblinded. The difference fromthe official results is much less dramatic than in
2017, with the exception of the team SParse, whomanaged to fix their software and produce more
valid output files.

Secondary Metrics

In addition to the main LAS ranking, we evaluatedthe systems along multiple other axes, which may
shed more light on their strengths and weaknesses.
This section provides an overview of selected secondary metrics for systems matching or surpassing
the baseline; a large number of additional results
are available at the shared task website.11
The website also features a LAS ranking of
unofficial system runs, i.e. those that were not

As an experiment, we also applied the 2017 system submissions to the 2018 test data. This allows
us to test how many systems can actually be used
to produce new data without a glitch, as well as
to see to what extent the results change over one
year and two releases of UD. Here it should be
noted that not all of the 2018 task languages andtreebanks were present in the 2017 task, therefore
causing many systems fail due to an unknown language or treebank code. The full results of this10
Using Udapi (Popel et al., 2017) eval.Conll18, marked
by the presence or absence of horizontal lines in Tables 6–8.
11
http://universaldependencies.org/
conll18/results.html

9

Team
1. Uppsala
2. HIT-SCIR
3. CEA LIST
4. CUNI x-ling
5. TurkuNLP
6. SLT-Interactions
7. UDPipe Future
8. Phoenix
9. BASELINE UDPipe
ParisNLPAntNLP
UniMelb
BOUN
ICS PAS
LATTICE
LeisureX
KParse18. Fudan
19. IBM NY
20. ONLP lab
21. NLP-Cube
22. Stanford
23. HUJI
24. ArmParser
25. iParse
26. SParse

Toks Wrds
97.60 98.18
98.42 98.12
98.16 97.78
98.09 97.74
97.83 97.42
97.51 97.09
97.46 97.04
97.46 97.03
97.39 96.97
97.39 96.97
97.39 96.97
97.39 96.97
97.39 96.97
97.39 96.97
97.39 96.97
97.39 96.97
97.39 96.97
97.38 96.96
97.30 96.92
97.28 96.86
97.36 96.80
96.19 95.99
94.95 94.61
79.75 79.41
78.45 78.11
2.32 2.32

Sents
83.80
83.87
82.79
82.80
83.03
83.01
83.64
82.91
83.01
83.01
83.01
83.01
83.01
83.01
83.01
83.01
83.01
82.85
83.51
83.00
82.55
76.55
80.84
13.33
68.37
2.34

Team
UPOS
1. Uppsala
90.91
2. HIT-SCIR
90.19
3. CEA LIST
89.97
4. TurkuNLP
89.81
5. LATTICE
89.53
6. UDPipe Future
89.37
7. Stanford
89.01
8. ICS PAS
88.70
9. CUNI x-ling
88.68
10. NLP-Cube
88.50
11. SLT-Interactions
88.12
12. IBM NY
88.02
13. UniMelb
87.90
14. KParse
87.62
15. Phoenix
87.49
16. ParisNLP
87.35
17. BASELINE UDPipe 87.32
AntNLP
87.32
19. ONLP lab
87.25
20. Fudan
87.25
21. BOUN
87.19
22. LeisureX
87.15
23. HUJI
85.06
24. ArmParser
72.99
25. iParse
71.38
26. SParse
2.25

Table 9: Tokenization, word segmentation andsentence segmentation (ordered by word F1scores; out-of-order scores in the other twocolumns are bold).

Feats Lemm
87.59 58.50
84.24 88.82
86.83 88.90
86.70 91.24
83.74 87.84
86.67 89.32
85.47 88.32
85.14 87.99
84.56 88.96
85.08 81.21
83.72 87.51
59.11 59.51
83.74 87.84
84.32 86.26
83.87 87.69
83.74 87.84
83.74 87.84
83.74 87.84
83.67 57.10
83.47 85.91
83.73 87.68
83.46 87.77
81.51 85.61
69.91 72.22
68.64 71.68
2.29
2.28

Table 10: Universal POS tags, features and lemmas (ordered by UPOS F1 scores; out-of-order
scores in the other two columns are bold).
tion scores (better than their respective macroaverages) on Arabic. They were able to translate it
into better LAS, but not MLAS and BLEX, where
there were too many other chances to make an error.
The complexity of the new metrics, especially
MLAS, is further underlined by Table 10: Uppsalais the clear winner in both UPOS tags and morphological features, but 6 other teams had better dependency relations and better MLAS. Note that aswith segmentation, morphology predicted by thebaseline system was available, though only a few
systems seem to have used it without attempting
to improve it.

experiment are available on the shared task website.12
Table 9 evaluates detection of tokens, syntactic
words and sentences. About a third of the systems trusted the baseline segmentation; this is lessthan in 2017. For most languages and in aggregate, the segmentation scores are very high andtheir impact on parsing scores is not easy to prove;
but it likely played a role in languages where segmentation is hard. For example, HIT-SCIR’s word
segmentation in Vietnamese surpasses the second
system by a margin of 6 percent points; likewise,
the system’s advantage in LAS and MLAS (butnot in BLEX!) amounts to 7–8 points. Similarly,
Uppsala and ParisNLP achieved good segmenta-

6.3

Partial Results

Table 11 gives the three main scores averaged overthe 61 “big” treebanks (training data larger than12
http://universaldependencies.org/
conll18/results-2017-systems.html

10

Team
1. HIT-SCIR
2. Stanford
3. TurkuNLP
4. UDPipe Future
5. ICS PAS
6. CEA LIST
7. LATTICE
8. NLP-Cube
9. ParisNLP
10. Uppsala
11. SLT-Interactions
12. AntNLP
13. LeisureX
14. UniMelb
15. IBM NY
16. Fudan
17. KParse
18. BASELINE UDPipe
19. Phoenix
20. BOUN
21. CUNI x-ling
22. ONLP lab
23. iParse
24. HUJI
25. ArmParser
26. SParse

LAS MLAS BLEX
84.37 70.12 75.05
83.03 72.67 75.46
81.85 71.27 75.83
81.83 71.71 74.67
81.72 70.30 74.42
81.66 70.89 72.32
80.97 66.27 71.50
80.48 67.79 64.76
80.29 65.88 70.95
80.25 68.81 36.02
79.67 64.95 69.77
79.61 65.43 70.34
77.98 63.79 68.55
77.69 63.17 68.25
77.55 47.34 36.68
75.42 62.28 62.90
74.84 62.40 63.84
74.14 61.27 64.67
73.93 61.12 64.47
72.85 60.00 62.99
71.54 58.33 61.63
67.08 55.20 33.08
66.55 55.37 58.80
62.07 53.20 56.90
58.14 45.87 49.25
2.63
2.26
2.30

Table 11: Average LAS on the 61 “big” treebanks(ordered by LAS F1 scores; out-of-order scores in
the other two columns are bold).

Team
1. CUNI x-ling
2. Uppsala
3. CEA LIST
4. HIT-SCIR
5. LATTICE
6. TurkuNLP
7. IBM NY
8. UDPipe Future
9. ICS PAS
10. AntNLP
11. KParse
12. SLT-Interactions
13. Stanford
14. BASELINE UDPipe
UniMelb
16. LeisureX
17. Phoenix
18. NLP-Cube
19. ParisNLP
20. ONLP lab
21. Fudan
22. BOUN
23. HUJI
24. ArmParser
25. iParse
26. SParse

LAS MLAS BLEX
27.89
6.13 13.98
25.87
5.16
9.03
23.90
3.75 10.99
23.88
2.88 10.50
23.39
4.38 10.01
22.91
3.59 11.40
21.88
2.62
7.17
21.75
2.82
8.80
19.26
1.89
6.17
18.59
3.43
8.61
17.84
3.32
6.58
17.47
1.79
6.95
17.45
2.76
7.63
17.17
3.44
7.63
17.17
3.44
7.63
17.16
3.43
7.63
16.99
3.02
8.00
16.85
3.39
7.05
16.52
2.53
6.75
15.98
3.58
4.96
15.45
2.98
6.61
14.78
2.59
6.43
8.53
0.92
2.77
7.47
1.86
3.54
2.82
0.23
0.97
0.00
0.00
0.00

Table 12: Average LAS, MLAS and BLEX on the
9 low-resource languages: Armenian (hy), Breton (br), Buryat (bxr), Faroese (fo), Kazakh (kk),Kurmanji (kmr), Naija (pcm), Thai (th) and Upper
Sorbian (hsb) (ordered by LAS F1 scores; out-oforder scores in the other two columns are bold).

test data, development data available). Higherscores reflect the fact that models for these test
sets are easier to learn: enough data is available,
no cross-lingual or cross-domain learning is necessary (the extra test sets are not included here).
Regarding ranking, the Stanford system makes aremarkable jump when it does not have to carrythe load of underresourced languages: from rank
8 to 2 in LAS, from 3 to 1 in MLAS and from 5 to
2 in BLEX.

ogy is almost impossible to learn from foreign languages, hence the much lower values of MLAS
and BLEX. BLEX is a bit better than MLAS,
which could be explained by cases where a wordform is identical to its lemma. However, there
are significant language-by-language differences;the best LAS on Faroese and Upper Sorbian surpassing 45%. This probably owes to the presenceof many Germanic and Slavic treebanks in training data, including some of the largest datasets in
UD. Three languages, Buryat, Kurmanji and Upper Sorbian, were introduced in the 2017 task as

Table 12 gives the LAS F1 score on the nine
low-resource languages only. Here we have a true
specialist: The team CUNI x-ling lives up to its
name and wins in all three scores, although in theoverall ranking they fall even slightly behind thebaseline. On the other hand, the scores are extremely low and the outputs are hardly useful forany downstream application. Especially morphol11

Team
1. HIT-SCIR
2. LATTICE
3. ICS PAS
4. TurkuNLP
5. UDPipe Future
6. AntNLP
7. Uppsala
8. ParisNLP
9. CEA LIST
10. KParse
11. NLP-Cube
12. SLT-Interactions
13. IBM NY
14. UniMelb
15. BASELINE UDPipe
LeisureX
17. Phoenix
Fudan
19. CUNI x-ling
20. BOUN
21. Stanford
22. ONLP lab
23. iParse
24. HUJI
25. ArmParser
26. SParse

LAS MLAS BLEX
69.53 45.94 53.30
68.12 45.03 51.71
66.90 49.24 54.89
64.48 47.63 53.54
64.21 47.53 49.53
63.73 42.24 48.31
63.60 46.00 29.25
60.84 40.71 46.08
57.34 39.97 43.39
57.32 39.20 43.61
56.78 37.13 38.30
56.74 35.73 42.90
56.13 26.51 25.23
56.12 36.09 42.09
55.01 38.80 41.06
55.01 38.80 41.06
54.63 38.38 40.72
54.63 38.15 40.07
54.33 38.10 40.70
50.18 34.29 36.75
48.56 34.86 38.55
47.49 32.74 22.39
38.79 28.03 29.62
36.74 24.47 27.70
34.54 22.94 25.26
0.00
0.00
0.00

Table 13: Average attachment score on the 7
small treebanks: Galician TreeGal, Irish, Latin
Perseus, North Sámi, Norwegian Nynorsk LIA,
Russian Taiga and Slovenian SST (ordered by
LAS F1 scores; out-of-order scores in the othertwo columns are bold).

Team
1. HIT-SCIR
2. Stanford
3. LATTICE
4. Uppsala
5. ICS PAS
6. TurkuNLP
7. UDPipe Future
8. CEA LIST
9. NLP-Cube
10. IBM NY
11. AntNLP
12. UniMelb
13. Phoenix
14. BASELINE UDPipe
15. KParse
16. SLT-Interactions
17. CUNI x-ling
18. ParisNLP
19. Fudan
20. LeisureX
21. BOUN
22. HUJI
23. iParse
24. ONLP lab
25. ArmParser
SParseLAS MLAS BLEX
74.20 55.52 62.34
73.14 58.75 61.96
72.34 55.60 60.42
72.27 57.80 29.73
72.18 58.07 60.97
71.78 57.54 63.25
71.57 57.93 61.52
70.45 54.99 57.83
69.83 55.01 54.15
69.40 46.59 38.12
68.87 53.47 57.71
68.72 52.05 56.77
66.97 52.26 55.69
66.63 51.75 54.87
66.55 51.29 54.45
64.73 48.47 54.90
64.70 49.71 52.72
64.09 48.79 53.16
63.54 45.54 50.73
61.05 41.95 50.60
56.46 41.91 45.12
56.35 46.52 50.10
44.20 33.43 38.18
43.33 30.20 20.08
0.00
0.00
0.00
0.00
0.00
0.00

Table 14: Average attachment score on the 5 additional test sets for high-resource languages: Czech
PUD, English PUD, Finnish PUD, Japanese Modern and Swedish PUD (ordered by LAS F1 scores;
out-of-order scores in the other two columns arebold).

surprise languages and had higher scores there.13
This is because in 2017, the segmentation, POS
tags and morphology UDPipe models were trainedon the test data, applied to it via cross-validation,and made available to the systems. Such an approach makes the conditions unrealistic, therefore
it was not repeated this year. Consequently, parsing these languages is now much harder.
In contrast, the results on the 7 treebanks with“small” training data and no development data
(Table 13) are higher on average, but again thevariance is significant. The smallest treebank

in the group, Norwegian Nynorsk LIA, has only
3583 training words. There are two larger Norwegian treebanks that could be used as additional
training sources. However, the LIA treebank consists of spoken dialects and is probably quite dissimilar to the other treebanks. The same can besaid about Slovenian SST and the other Slovenian treebank; SST is the most difficult dataset
in the group, despite of having almost 20K of its
own training words. Other treebanks, like Russian Taiga and Galician TreeGal, have much better scores (74% LAS, about 61% MLAS and 64%
BLEX). There are also two treebanks that are thesole representatives of their languages: Irish andNorth Sámi. Their best LAS is around 70%: com-

13

The fourth surprise language, North Sámi, has now additional training data and does not fall in the low-resource
category.

12

parable to Nynorsk LIA but much better than SST.
ICS PAS is the most successful system in the domain of small treebanks, especially when judged
by MLAS and BLEX.

Treebank
LAS Best system
Avg StDev
1. pl lfg
94.86 HIT-SCIR
85.89 ± 6.97
2. ru syntagrus 92.48 HIT-SCIR
79.68 ± 9.09
3. hi hdtb
92.41 HIT-SCIR
85.16 ± 5.32
4. pl sz
92.23 HIT-SCIR
81.47 ± 7.27
5. cs fictree
92.02 HIT-SCIR
82.10 ± 7.26
6. it isdt
92.00 HIT-SCIR
87.61 ± 4.12
7. cs pdt
91.68 HIT-SCIR
82.18 ± 6.91
8. ca ancora
91.61 HIT-SCIR
83.61 ± 6.01
9. cs cac91.61 HIT-SCIR
82.69 ± 6.93
10. sl ssj
91.47 HIT-SCIR
75.00 ± 9.13
11. no bokmaal 91.23 HIT-SCIR
79.80 ± 7.29
12. bg btb
91.22 HIT-SCIR
82.52 ± 5.88
13. no nynorsk
90.99 HIT-SCIR
78.55 ± 7.88
14. es ancora
90.93 HIT-SCIR
82.84 ± 6.17
15. fi pud
90.23 HIT-SCIR
68.87 ±15.61
16. fr sequoia
89.89 LATTICE
80.55 ± 5.91
17. el gdt
89.65 HIT-SCIR
80.65 ± 6.05
18. nl alpino
89.56 HIT-SCIR
77.76 ± 7.42
19. sk snk
88.85 HIT-SCIR
76.53 ± 7.24
20. fi tdt
88.73 HIT-SCIR
73.55 ± 9.39
21. sr set
88.66 Stanford
79.84 ± 6.57
22. sv talbanken 88.63 HIT-SCIR
77.71 ± 6.50
23. fi ftb
88.53 HIT-SCIR
76.89 ± 7.60
24. uk iu
88.43 HIT-SCIR
72.47 ± 8.25
25. fa seraji
88.11 HIT-SCIR
78.71 ± 6.04
26. en pud
87.89 LATTICE
74.51 ± 8.28
27. pt bosque
87.81 Stanford
80.49 ± 5.46
28. hr set
87.36 HIT-SCIR
78.37 ± 6.42
29. fro srcmf
87.12 UDPipe Future 74.38 ±16.74
30. la ittb
87.08 HIT-SCIR
77.00 ± 7.42
31. ko kaist
86.91 HIT-SCIR
77.10 ± 8.72
32. fr gsd
86.89 HIT-SCIR
79.43 ± 5.47
33. ro rrt
86.87 HIT-SCIR
75.77 ± 7.66
34. nl lassysmall 86.84 HIT-SCIR
75.08 ± 6.59
35. da ddt
86.28 HIT-SCIR
75.02 ± 6.47
36. cs pud
86.13 HIT-SCIR
73.24 ± 9.97
37. af afribooms 85.47 HIT-SCIR
76.61 ± 6.17
38. et edt
85.35 HIT-SCIR
72.08 ± 8.71
39. ko gsd
85.14 HIT-SCIR
71.88 ±10.53
40. en gum
85.05 LATTICE
74.20 ± 6.27
41. en ewt
84.57 HIT-SCIR
75.99 ± 5.40
42. eu bdt
84.22 HIT-SCIR
72.08 ± 8.83
43. sv lines
84.08 HIT-SCIR
73.76 ± 5.98
44. lv lvtb
83.97 HIT-SCIR
67.76 ± 9.01
45. ur udtb
83.39 HIT-SCIR
75.89 ± 4.69
46. ja gsd
83.11 HIT-SCIR
73.68 ± 4.55
47. gl ctg
82.76 Stanford
72.46 ± 7.13
48. hu szeged
82.66 HIT-SCIR
67.05 ± 8.63
49. en lines
81.97 HIT-SCIR
72.28 ± 5.59
50. de gsd
80.36 HIT-SCIR
70.13 ± 7.14
51. sv pud
80.35 HIT-SCIR
67.02 ± 9.23
52. id gsd
80.05 HIT-SCIR
73.05 ± 4.69
53. it postwita
79.39 HIT-SCIR
64.95 ± 6.88
54. grc perseus
79.39 HIT-SCIR
59.01 ±15.56
55. grc proiel
79.25 HIT-SCIR
65.02 ±14.58
56. ar padt
77.06 Stanford
64.07 ± 6.41
57. zh gsd
76.77 HIT-SCIR
60.32 ± 6.14
58. he htb
76.09 Stanford
58.73 ± 5.29
59. fr spoken
75.78 HIT-SCIR
64.66 ± 5.35
60. cu proiel
75.73 Stanford
62.64 ± 6.98
61. gl treegal
74.25 UDPipe Future 64.65 ± 5.61
62. ru taiga
74.24 ICS PAS
56.27 ± 9.16
63. la proiel
73.61 HIT-SCIR
61.25 ± 6.87
64. la perseus
72.63 HIT-SCIR
46.91 ±11.12
65. ga idt
70.88 TurkuNLP
58.37 ± 7.05
66. no nynorsklia 70.34 HIT-SCIR
50.33 ± 9.28
67. sme giella69.87 LATTICE
51.10 ±14.32
68. got proiel
69.55 Stanford
60.55 ± 4.93
69. ug udt
67.05 HIT-SCIR
54.27 ± 6.90
70. tr imst
66.44 HIT-SCIR
55.61 ± 6.49
71. sl sst
61.39 HIT-SCIR
47.07 ± 5.84
72. vi vtb
55.22 HIT-SCIR
40.40 ± 4.43
73. fo oft
49.43 CUNI x-ling
27.87 ± 9.75
74. hsb ufal
46.42 SLT-Interactions 26.48 ± 8.90
75. br keb
38.64 CEA LIST
13.27 ± 8.77
76. hy armtdp
37.01 LATTICE
22.39 ± 7.91
77. kk ktb
31.93 Uppsala
19.11 ± 6.34
78. kmr mg
30.41 IBM NY
20.27 ± 6.14
79. pcm nsc
30.07 CUNI x-ling
13.19 ± 5.76
80. ja modern
28.33 Stanford
18.92 ± 5.14
81. bxr bdt
19.53 AntNLP
11.45 ± 4.28
82. th pud
13.70 CUNI x-ling
1.38 ± 2.83

Table 14 gives the average LAS on the 5 extra test sets (no own training data, but other treebanks of the same language exist). Four of themcome from the Parallel UD (PUD) collection introduced in the 2017 task (Zeman et al., 2017).
The fifth, Japanese Modern, turned out to be one of
the toughest test sets in this shared task. There isanother Japanese treebank, GSD, with over 160K
training tokens, but the Modern dataset seems almost inapproachable with models trained on GSD.
A closer inspection reveals why: despite its name,it is actually a corpus of historical Japanese, although from the relatively recent Meiji and Taishō
periods (1868–1926). An average sentence in
GSD is about 1.3× longer than in Modern. GSD
has significantly more tokens tagged as auxiliaries,
but more importantly, the top ten AUX lemmas
in the two treebanks are completely disjoint sets.
Some other words are out-of-vocabulary because
their preferred spelling changed. For instance, thedemonstrative pronoun sore is written using hiragana in GSD, but a kanji character is used in Modern. Striking differences can be observed also in
dependency relations: in GSD, 3.7% relations arensubj (subject), and 1.2% are cop (copula). In
Modern, there is just 0.13% of subjects, and not asingle occurrence of a copula.
See Tables 15, 16 and 17 for a ranking of all
test sets by the best scores achieved on them byany parser. Note that this cannot be directly interpreted as a ranking of languages by their parsing
difficulty: many treebanks have high ranks simply
because the corresponding training data is large.
Table 18 compares average LAS and MLAS foreach treebank.
Finally, Tables 19 and 20 show the treebanks
where word and sentence segmentation was extremely difficult (judged by the average parserscore). Not surprisingly, word segmentation is difficult for the low-resource languages and for languages like Chinese, Vietnamese, Japanese andThai, where spaces do not separate words. Notably the Japanese GSD set is not as difficult, but
whoever trusted it, crashed on the “Modern” set.
Sentence segmentation was particularly hard fortreebanks without punctuation, i.e., most of theclassical languages and spoken data.

Table 15: Treebank ranking by best parser LAS(Avg=average LAS over all systems, out-of-order
scores in bold).
13

Treebank
MLAS Best system
Avg StDev
1. pl lfg
86.93 UDPipe Future 73.73 ± 7.29
2. ru syntagrus
86.76 UDPipe Future 71.63 ± 9.36
3. cs pdt
85.10 UDPipe Future 73.61 ± 6.32
4. cs fictree
84.23 ICS PAS
69.91 ± 7.77
5. ca ancora
84.07 UDPipe Future 74.62 ± 7.69
6. es ancora
83.93 Stanford
74.61 ± 7.43
7. it isdt
83.89 Stanford
77.14 ± 8.89
8. fi pud
83.78 Stanford
62.38 ±14.83
9. no bokmaal
83.68 UDPipe Future 70.75 ± 8.92
10. cs cac
83.42 UDPipe Future 71.39 ± 6.89
11. bg btb
83.12 UDPipe Future 73.18 ± 7.15
12. fr sequoia
82.55 Stanford
70.42 ± 9.04
13. sl ssj
82.38 Stanford
62.41 ± 9.18
14. no nynorsk
81.86 UDPipe Future 68.62 ± 9.45
15. ko kaist
81.29 HIT-SCIR
70.18 ± 9.36
16. ko gsd
80.85 HIT-SCIR
63.73 ±16.02
17. fi tdt
80.84 Stanford
65.27 ± 9.22
18. fa seraji
80.83 UDPipe Future 71.23 ± 7.77
19. pl sz
80.77 Stanford
64.80 ± 8.49
20. fro srcmf
80.28 UDPipe Future 65.19 ±16.58
21. la ittb
79.84 ICS PAS
67.77 ± 8.37
22. fi ftb
79.65 TurkuNLP
66.11 ± 8.86
23. sv talbanken
79.32 Stanford
68.05 ± 8.49
24. ro rrt
78.68 TurkuNLP
67.43 ± 7.24
25. el gdt
78.66 Stanford
64.29 ± 8.28
26. fr gsd
78.44 Stanford
69.33 ± 8.59
27. hi hdtb
78.30 UDPipe Future 68.48 ± 5.88
28. sr set
77.73 UDPipe Future 67.33 ± 5.96
29. da ddt
77.31 Stanford
65.00 ± 6.89
30. et edt
76.97 TurkuNLP
63.59 ± 8.34
31. nl alpino
76.52 Stanford
62.82 ± 9.81
32. en ewt
76.33 Stanford
66.84 ± 5.86
33. pt bosque
75.94 Stanford
66.22 ± 6.76
34. cs pud
75.81 UDPipe Future 60.47 ±11.36
35. af afribooms
75.67 UDPipe Future 63.76 ± 7.06
36. sk snk
75.01 Stanford
56.82 ± 8.32
37. en pud
74.86 Stanford
63.05 ± 7.89
38. nl lassysmall 74.11 Stanford
61.95 ± 9.12
39. hr set
73.44 Stanford
60.08 ± 7.07
40. en gum
73.24 ICS PAS
61.72 ± 7.69
41. ja gsd
72.62 HIT-SCIR
59.52 ± 6.20
42. uk iu
72.27 UDPipe Future 55.45 ± 8.08
43. en lines
72.25 ICS PAS
62.35 ± 8.04
44. eu bdt
71.73 UDPipe Future 58.49 ± 8.62
45. gl ctg
70.92 Stanford
57.92 ±14.10
46. ar padt
68.54 Stanford
53.28 ± 6.12
47. it postwita
68.50 Stanford
51.72 ± 8.80
48. id gsd
68.36 Stanford
61.03 ± 6.49
49. lv lvtb
67.89 Stanford
53.31 ± 7.96
50. hu szeged
67.13 UDPipe Future 53.08 ± 8.01
51. zh gsd
66.62 HIT-SCIR
50.42 ± 5.87
52. sv lines
66.58 Stanford
57.40 ± 7.43
53. fr spoken
64.67 HIT-SCIR
53.17 ± 5.61
54. he htb
63.38 Stanford
45.22 ± 4.94
55. cu proiel
63.31 Stanford
50.28 ± 6.69
56. ru taiga
61.59 ICS PAS
37.16 ± 7.53
57. gl treegal
60.63 UDPipe Future 47.35 ± 5.93
58. grc proiel
60.27 Stanford
47.62 ±11.82
59. la proiel
59.36 Stanford
47.79 ± 6.90
60. de gsd
58.04 TurkuNLP
39.13 ±10.35
61. ur udtb
57.98 TurkuNLP
49.64 ± 4.21
62. no nynorsklia 57.51 ICS PAS
37.08 ± 7.78
63. sme giella57.47 TurkuNLP
38.29 ±12.37
64. got proiel
56.45 UDPipe Future 46.18 ± 5.36
65. tr imst
55.73 Stanford
45.26 ± 6.15
66. grc perseus
54.98 HIT-SCIR
35.65 ±12.31
67. sv pud
51.74 TurkuNLP
39.41 ± 7.78
68. la perseus
49.77 ICS PAS
28.67 ± 8.06
69. vi vtb
47.61 HIT-SCIR
32.45 ± 7.28
70. sl sst
45.93 ICS PAS
33.12 ± 5.33
71. ga idt
45.79 TurkuNLP
33.70 ± 5.18
72. ug udt
45.78 UDPipe Future 35.08 ± 5.96
73. br keb
13.91 Uppsala1.52 ± 3.34
74. hy armtdp
13.36 CUNI x-ling
5.94 ± 2.92
75. ja modern
11.82 Uppsala6.45 ± 2.59
76. hsb ufal
9.09 LATTICE
4.66 ± 2.37
77. kk ktb
8.93 CUNI x-ling
5.04 ± 2.34
78. kmr mg
7.98 IBM NY
4.01 ± 1.96
79. th pud
6.29 CUNI x-ling
0.42 ± 1.27
80. pcm nsc
5.30 KParse
3.00 ± 1.30
81. bxr bdt
2.98 AntNLP
1.33 ± 0.72
82. fo oft
1.07 CUNI x-ling
0.37 ± 0.21

Treebank
BLEX Best system
Avg StDev
1. pl lfg
90.42 TurkuNLP
72.81 ±16.96
2. ru syntagrus
88.65 TurkuNLP
68.57 ±18.07
3. cs pdt
87.91 HIT-SCIR
74.41 ±14.88
4. cs fictree
87.81 ICS PAS
71.10 ±16.26
5. cs cac
86.79 TurkuNLP
71.61 ±18.18
6. hi hdtb
86.74 HIT-SCIR
75.80 ± 9.28
7. pl sz
86.29 TurkuNLP
67.33 ±17.15
8. no bokmaal
85.82 UDPipe Future 69.52 ±13.54
9. ca ancora
85.47 UDPipe Future 72.60 ±12.31
10. es ancora
84.92 HIT-SCIR
72.10 ±12.71
11. it isdt
84.76 ICS PAS
75.42 ±10.72
12. fr sequoia
84.67 ICS PAS
70.63 ±11.66
13. no nynorsk
84.44 TurkuNLP
67.43 ±14.10
14. la ittb
84.37 TurkuNLP
68.10 ±17.85
15. bg btb
84.31 TurkuNLP
68.13 ±15.02
16. fro srcmf
84.11 UDPipe Future 70.46 ±16.40
17. sr set
83.28 TurkuNLP
65.62 ±17.61
18. sl ssj
83.23 Stanford
62.54 ±17.20
19. fi ftb
82.44 TurkuNLP
59.66 ±16.50
20. fi pud
82.44 TurkuNLP
52.25 ±18.50
21. sv talbanken 81.44 TurkuNLP
66.45 ±13.18
22. fi tdt
81.24 TurkuNLP
54.70 ±17.25
23. fr gsd
81.18 HIT-SCIR
69.61 ±10.58
24. ro rrt
80.97 TurkuNLP
63.53 ±15.84
25. sk snk
80.74 TurkuNLP
58.35 ±15.07
26. pt bosque
80.62 TurkuNLP
68.71 ±11.27
27. en pud
80.53 LATTICE
64.73 ±10.88
28. cs pud
80.53 ICS PAS
64.62 ±16.03
29. hr set
80.50 TurkuNLP
64.64 ±17.13
30. fa seraji
80.44 Stanford
68.38 ± 7.39
31. el gdt
80.09 TurkuNLP
63.26 ±15.60
32. ko kaist
79.55 TurkuNLP
57.32 ±20.78
33. et edt
79.37 TurkuNLP
57.06 ±16.14
34. nl alpino
79.15 HIT-SCIR
64.29 ±10.83
35. en ewt
78.44 HIT-SCIR
67.53 ± 8.47
36. uk iu
78.38 TurkuNLP
57.78 ±15.95
37. eu bdt
78.15 TurkuNLP
60.52 ±15.24
38. da ddt
78.07 TurkuNLP
63.16 ±11.41
39. sv lines
77.01 ICS PAS
63.13 ±11.72
40. id gsd
76.56 Stanford
62.52 ± 7.89
41. nl lassysmall 76.54 HIT-SCIR
60.92 ±11.93
42. af afribooms 76.44 TurkuNLP
63.87 ± 9.62
43. ko gsd
76.31 TurkuNLP
54.13 ±17.78
44. en lines
75.29 HIT-SCIR
62.29 ± 9.27
45. gl ctg
75.14 Stanford
60.86 ±10.82
46. ur udtb
73.79 TurkuNLP
62.93 ± 6.42
47. ja gsd
73.79 HIT-SCIR
60.87 ± 6.04
48. en gum
73.57 ICS PAS
61.02 ± 8.59
49. hu szeged
73.17 TurkuNLP
55.42 ±10.95
50. zh gsd
72.97 HIT-SCIR
55.66 ± 6.26
51. lv lvtb
72.40 TurkuNLP
53.42 ±14.56
52. de gsd
71.40 HIT-SCIR
54.86 ±14.99
53. cu proiel
71.31 Stanford
51.27 ±15.35
54. ar padt
70.06 Stanford
49.13 ±18.98
55. it postwita
69.34 HIT-SCIR
50.97 ± 8.76
56. grc proiel
69.03 TurkuNLP
48.58 ±19.91
57. la proiel
67.60 TurkuNLP
51.03 ±14.56
58. sv pud
66.12 TurkuNLP
50.20 ±11.30
59. fr spoken
65.63 HIT-SCIR
52.57 ± 7.29
60. he htb
65.04 Stanford
47.22 ± 6.60
61. ru taiga
64.36 ICS PAS
39.32 ±10.49
62. gl treegal
64.29 UDPipe Future 49.38 ± 8.18
63. got proiel
63.98 Stanford
48.79 ±13.77
64. no nynorsklia 60.98 ICS PAS
41.20 ± 8.64
65. tr imst
60.13 TurkuNLP
45.39 ±10.38
66. sme giella60.10 TurkuNLP
35.76 ±12.68
67. grc perseus
58.68 TurkuNLP
36.48 ±16.03
68. ug udt
55.42 HIT-SCIR
41.64 ± 8.09
69. ga idt
55.18 TurkuNLP
37.83 ± 7.61
70. la perseus
52.75 ICS PAS
30.16 ±11.05
71. sl sst
50.94 ICS PAS
37.20 ± 6.87
72. vi vtb
44.02 Stanford
35.50 ± 3.74
73. pcm nsc
26.04 CUNI x-ling
12.07 ± 5.63
74. hsb ufal
21.09 LATTICE
11.26 ± 4.97
75. br keb
20.70 TurkuNLP
4.19 ± 4.93
76. hy armtdp
19.04 CUNI x-ling
10.68 ± 4.37
77. fo oft
14.40 CUNI x-ling
7.32 ± 3.33
78. ja modern
13.79 Stanford
7.70 ± 2.86
79. kmr mg
13.66 LATTICE
8.44 ± 3.11
80. kk ktb
11.33 CUNI x-ling
6.75 ± 2.95
81. th pud
10.77 CUNI x-ling
0.91 ± 2.11
82. bxr bdt
6.65 AntNLP
3.39 ± 1.61

Table 16: Treebank ranking by best parser MLAS.

Table 17: Treebank ranking by best parser BLEX.

14

Treebank Best Best system Avg StDev
70. bxr bdt
99.24 IBM NY
88.64 ± 8.09
71. fi pud
99.69 Uppsala88.13 ±10.81
72. zh gsd
96.71 HIT-SCIR
86.91 ± 3.83
73. fo oft
99.47 CUNI x-ling 86.76 ±10.68
74. ar padt
96.81 Stanford
86.62 ± 7.00
75. kmr mg 96.97 Uppsala
86.61 ± 7.16
76. kk ktb
97.40 Uppsala85.55 ± 7.45
77. br keb
92.45 TurkuNLP 83.76 ± 7.37
78. he htb
93.98 Stanford
82.45 ± 3.80
79. vi vtb
93.46 HIT-SCIR
81.71 ± 3.73
80. pcm nsc 99.71 CEA LIST 79.94 ±10.69
81. ja modern 75.69 HIT-SCIR
59.40 ± 7.70
82. th pud
69.93 Uppsala
17.16 ±20.57

Treebank
LAS MLAS Diff Language
1. de gsd
70.13 39.13 31.01 German
2. sv pud
67.02 39.41 27.61 Swedish
3. fo oft
27.87
0.37 27.50 Faroese
4. ur udtb
75.89 49.64 26.25 Urdu
5. ga idt
58.37 33.70 24.66 Irish
6. grc perseus
59.01 35.65 23.36 Ancient Greek
7. hsb ufal
26.48
4.66 21.82 Upper Sorbian
8. sk snk
76.53 56.82 19.71 Slovak
9. ug udt
54.27 35.08 19.20 Uyghur
10. ru taiga
56.27 37.16 19.12 Russian
11. hr set
78.37 60.08 18.29 Croatian
12. la perseus
46.91 28.67 18.24 Latin
13. grc proiel
65.02 47.62 17.40 Ancient Greek
14. gl treegal
64.65 47.35 17.30 Galician
15. uk iu
72.47 55.45 17.01 Ukrainian
16. hi hdtb
85.16 68.48 16.68 Hindi
17. pl sz
81.47 64.80 16.67 Polish
18. hy armtdp
22.39
5.94 16.45 Armenian
19. el gdt
80.65 64.29 16.36 Greek
20. sv lines
73.76 57.40 16.36 Swedish
21. kmr mg
20.27
4.01 16.26 Kurmanji
22. nl alpino
77.76 62.82 14.95 Dutch
23. gl ctg
72.46 57.92 14.55 Galician
24. lv lvtb
67.76 53.31 14.45 Latvian
25. got proiel
60.55 46.18 14.37 Gothic
26. pt bosque
80.49 66.22 14.27 Portuguese
27. ja gsd
73.68 59.52 14.16 Japanese
28. kk ktb
19.11
5.04 14.07 Kazakh
29. hu szeged
67.05 53.08 13.96 Hungarian
30. sl sst
47.07 33.12 13.95 Slovenian
31. eu bdt
72.08 58.49 13.59 Basque32. he htb
58.73 45.22 13.51 Hebrew
33. la proiel
61.25 47.79 13.46 Latin
34. no nynorsklia 50.33 37.08 13.25 Norwegian
35. it postwita
64.95 51.72 13.22 Italian
36. nl lassysmall 75.08 61.95 13.14 Dutch
37. af afribooms 76.61 63.76 12.84 Afrikaans
38. sme giella51.10 38.29 12.82 North Sámi
39. cs pud
73.24 60.47 12.77 Czech
40. sl ssj
75.00 62.41 12.59 Slovenian
41. sr set
79.84 67.33 12.50 Serbian
42. en gum
74.20 61.72 12.48 English
43. ja modern
18.92
6.45 12.47 Japanese
44. cu proiel
62.64 50.28 12.36 Old Church Slavonic45. cs fictree
82.10 69.91 12.19 Czech
46. pl lfg
85.89 73.73 12.17 Polish
47. id gsd
73.05 61.03 12.02 Indonesian
48. br keb
13.27
1.52 11.75 Breton
49. fr spoken
64.66 53.17 11.49 French
50. en pud
74.51 63.05 11.46 English
51. cs cac
82.69 71.39 11.29 Czech
52. ar padt
64.07 53.28 10.79 Arabic
53. fi ftb
76.89 66.11 10.78 Finnish
54. it isdt
87.61 77.14 10.47 Italian
55. tr imst
55.61 45.26 10.34 Turkish
56. pcm nsc
13.19
3.00 10.19 Naija
57. fr sequoia
80.55 70.42 10.13 French
58. bxr bdt
11.45
1.33 10.12 Buryat
59. fr gsd
79.43 69.33 10.10 French
60. da ddt
75.02 65.00 10.02 Danish
61. no nynorsk
78.55 68.62 9.93 Norwegian62. en lines
72.28 62.35 9.93 English
63. zh gsd
60.32 50.42 9.90 Chinese
64. sv talbanken 77.71 68.05 9.66 Swedish
65. bg btb
82.52 73.18 9.34 Bulgarian
66. la ittb
77.00 67.77 9.23 Latin
67. fro srcmf
74.38 65.19 9.18 Old French
68. en ewt
75.99 66.84 9.15 English
69. no bokmaal 79.80 70.75 9.05 Norwegian70. ca ancora
83.61 74.62 8.99 Catalan
71. cs pdt
82.18 73.61 8.57 Czech
72. et edt
72.08 63.59 8.50 Estonian
73. ro rrt
75.77 67.43 8.33 Romanian74. fi tdt
73.55 65.27 8.28 Finnish
75. es ancora
82.84 74.61 8.23 Spanish
76. ko gsd
71.88 63.73 8.15 Korean
77. ru syntagrus 79.68 71.63 8.05 Russian
78. vi vtb
40.40 32.45 7.95 Vietnamese
79. fa seraji
78.71 71.23 7.48 Persian
80. ko kaist
77.10 70.18 6.92 Korean
81. fi pud
68.87 62.38 6.49 Finnish
82. th pud
1.38
0.42 0.96 Thai

Table 19: Treebanks with most difficult word segmentation (by average parser F1 ).
Treebank Best Best system Avg StDev
73. grc proiel 51.84 HIT-SCIR 42.46 ± 7.33
74. cu proiel 48.67 Stanford
35.54 ± 4.02
75. la proiel 39.61 Stanford
33.40 ± 5.39
76. got proiel 38.23 Stanford
27.22 ± 4.47
77. it postwita 65.90 Stanford
25.25 ±14.30
78. sl sst
24.43 NLP-Cube 20.92 ± 4.70
79. fr spoken 24.17 Stanford
20.43 ± 2.89
80. th pud
12.37 TurkuNLP
1.75 ± 3.68
81. pcm nsc
0.93 Stanford
0.06 ± 0.19
82. ja modern 0.23 Stanford
0.01 ± 0.04

Table 20: Treebanks with most difficult sentence
segmentation (by average parser F1 ).

7

Analysis of Submitted Systems

Table 21 gives an overview of 24 of the systems
evaluated in the shared task. The overview isbased on a post-evaluation questionnaire to which
24 of 25 teams responded. Systems are orderedalphabetically by name and their LAS rank is indicated in the second column.
Looking first at word and sentence segmentation, we see that, while a clear majority of systems(19/24) rely on the baseline system for segmentation, slightly more than half (13/24) have developed their own segmenter, or tuned the baseline
segmenter, for at least a subset of languages. Thisis a development from 2017, where only 7 out of
29 systems used anything other than the baseline
segmenter.
When it comes to morphological analysis, including universal POS tags, features and lemmas,
all systems this year include some such component, and only 6 systems rely entirely on the base-Table 18: Treebank ranking by difference between
average parser LAS and MLAS.

15

System
R Segment Morph
Syntax
WEmb
Additional Data MultiLing
AntNLP
9Base
BaseSingle-G
FB
None
OwnS
ArmParser
25 Base
Own
Single
FB
None
None
BOUN
21 Base
Base
Single-T
Base
None
None
CEA LIST
6
Base BL /Own Single-G/T
B/FB
OPUS/Wikt
OwnL
CUNI x-ling
20 B/Own B/Own Single/Ens
FB/None O/UM/WALS/Wiki OwnL,S
Fudan
17 BaseBase
EnsembleNone
None
OwnL,S
HIT-SCIR
1 B/Own
Base
Ensemble B/FB/Crawl
None
OwnL,S
HUJI
24 Base
Base
Single-T
FB
None
OwnL
IBM NY
13 B/Own B/Joint Ensemble-T
B/FB
Wiki
OwnL,SICS PAS
3
Base
Own
Single-G
FB/NoneNone
None
KParse16 B/Own
Own
Single
Other
None
OwnL
LATTICE
3
Base
OwnU Single-G/Ens B/FB/Crawl
OPUS/Wiki
OwnL,S
LeisureX
15 Base
Own
Single
Base
None
OwnL
NLP-Cube
9
Own
Own
SingleFB
None
OwnL
ONLP lab
22 Base
Base
Single-T
None
UML
None
ParisNLP
11 B/Own B/Own
Single-G
FB
UML
OwnL
Phoenix
19 OwnOwnUSingle
Train
None
OwnL
SLT-Interactions 12 B/Own
Own
Single
Crawl
None
OwnL
SParse26 B/Own
Own
Single-G
Crawl
None
OwnL
Stanford
7
Own
Own
Single-G
B/FB
None
None
TurkuNLP
2 B/Own
Own
Single-G
B/FB
OPUS/Aper
OwnL
UDPipe Future 3
Own
Joint
Single-G
B/FB
None
None
UniMelb
14 Base
Joint
Single
Base
None
Base
Uppsala7
Own OwnU,F Single-T
B/FB/Wiki OPUS/Wiki/Aper OwnL,S
Table 21: Classification of participating systems. R = LAS ranking. Segment = word/sentence segmentation. Morph = morphological analysis, including universal POS tags [U], features [F] and lemmas [L],with subscripts for subsets [Joint = morphological component trained jointly with syntactic parser]. Syntax = syntactic parsing [Single = single parser; Ensemble (or Ens) = parser ensemble; G = graph-based;
T = transition-based]. WEmb = pre-trained word embeddings [FB = Facebook; Crawl = trained on web
crawl data provided by the organizers; Wiki = trained on Wikipedia data; Train = trained on treebank
training data]. Additional Data = data used in addition to treebank training sets [OPUS (or O) = OPUS,
Aper = Apertium morphological analysers, Wikt = Wiktionary, Wiki = Wikipedia, UM = UniMorph,
UML = Universal Morphological Lattices, WALS = World Atlas of Language Structures]. MultiLing =multilingal models used for low-resource (L) or small (S) languages. In all columns, Base (or B) refers
to the Baseline UDPipe system or the baseline word embeddings provided by the organizers, while Nonemeans that there is no corresponding component in the system.
line UDPipe system. This is again quite different
from 2017, where more than half the systems either just relied on the baseline tagger (13 systems)
or did not predict any morphology at all (3 systems). We take this to be primarily a reflectionof the fact that two out of three official metrics
included (some) morphological analysis this year,
although 3 systems did not predict the lemmas required for the BLEX metric (and 2 systems only
predicted universal POS tags, no features). As far
as we can tell from the questionnaire responses,

only 3 systems used a model where morphology
and syntax were predicted jointly.14
For syntactic parsing, most teams (19) use a single parsing model, while 5 teams, including thewinning HIT-SCIR system, build ensemble models, either for all languages or a subset of them.
When it comes to the type of parsing model, weobserve that graph-based models are more popular than transition-based models this year, while
the opposite was true in 2017. We hypothesize that
14
The ONLP lab system also has a joint model but in theend used the baseline morphology as it gave better results.

16this is due to the superior performance of the Stanford graph-based parser in last year’s shared task,
and many of the high-performing systems this year
either incorporate that parser or a reimplementation of it.15
The majority of parsers make use of pre-trained
word embeddings. Most popular are the Facebook
embeddings, which are used by 17 systems, followed by the baseline embeddings provided by theorganizers (11), and embeddings trained on web
crawl data (4).16 When it comes to additional data,over and above the treebank training sets and pretrained word embeddings, the most striking observation is that a majority of systems (16) did not use
any at all. Those that did primarily used OPUS(5), Wikipedia dumps (3), Apertium morphological analyzers (2), and Universal Morphological
Lattices (2). The CUNI x-ling system, which focused on low-resource languages, also exploited
UniMorph and WALS (in addition to OPUS and
Wikipedia).
Finally, we note that a majority of systems makeuse of models trained on multiple languages toimprove parsing for languages with little or no
training data. According to the questionnaire responses, 15 systems use multilingual models forthe languages classified as “low-resource”, while
7 systems use them for the languages classified as“small”.17 Only one system relied on the baselinedelexicalized parser trained on data from all languages.

8

the language in question was available for all of
the languages. The most extreme example of theseis Thai, where the only accessible resource was theFacebook Research Thai embeddings model andthe OPUS parallel corpora. This year’s task also
introduced two additional metrics that take into
account morphology and lemmatization. This encouraged the development of truly end-to-end fullparsers, producing complete parses including morphological features and lemmas in addition to thesyntactic tree. This also aimed to improve the utility of the systems developed in the shared task forlater downstream applications. For most UD languages, these parsers represent a new state of theart for end-to-end dependency parsing.
The analysis of the shared task results has so far
only scratched the surface, and we refer to the system description papers for more in-depth analysis
of individual systems and their performance. For
many previous CoNLL shared tasks, the task itself has only been the starting point of a long andfruitful research strand, enabled by the resources
created for the task. We hope and believe that the2017 and 2018 UD parsing tasks will join this tradition.

Acknowledgments
We are grateful to all the contributors to Universal
Dependencies; without their effort a task like thissimply wouldn’t be possible.
The work described herein, including
data preparation for the CoNLL 2018 UD
Shared Task, has been supported by the following grants and projects: OP PPR No.
CZ.07.1.02/0.0/0.0/16 023/0000108 of the City ofPrague, “CRACKER,” H2020 Project No. 645357
of the European Commission; “MANYLA,”
Grant No. GA15-10472S of the Grant Agencyof the Czech Republic; FIN-CLARIN; andthe LINDAT/CLARIN research infrastructure
project funded by the Ministry of Education,
Youth and Sports of the Czech Republic, Project.
No. LM2015071. The data for the CoNLL
2018 UD Shared Task are available also via the
LINDAT/CLARIN repository.

Conclusion

The CoNLL 2018 Shared Task on UD parsing, thesecond in the series, was novel in several respects.
Besides using cross-linguistically consistent linguistic representations, emphasizing end-to-end
processing of text, and in using a multiply parallel test set, as in 2017, it was unusual also in featuring an unprecedented number of languages andtreebanks and in integrating cross-lingual learning
for resource-poor languages. Compared to the firstedition of the task in 2017, this year several languages were provided with little-to-no resources,
whereas in 2017, predicted morphology trained on
15

This is true of at least 3 of the 5 best performing systems.
The baseline embeddings were the same as in 2017 andtherefore did not cover new languages, which may partly explain the greater popularity of the Facebook embeddings thisyear.
17
We know that some teams used them also for clusters
involving high-resource languages, but we have no detailed
statistics on this usage.

References

16

Gor Arakelyan, Karen Hambardzumyan, and Hrant
Khachatrian. 2018. Towards JointUD: Part-ofspeech tagging and lemmatization using recurrent
neural networks. In Proceedings of the CoNLL 2018
Shared Task: Multilingual Parsing from Raw Text to17automatically annotated raw texts and word embeddings. LINDAT/CLARIN digital library at the Institute of Formal and Applied Linguistics, Charles
University. http://hdl.handle.net/11234/1-1989.

Universal Dependencies. Association for Computational Linguistics.
Riyaz Ahmad Bhat, Irshad Ahmad Bhat, and Srinivas
Bangalore. 2018. The SLT-Interactions parsing system at the CoNLL 2018 shared task. In Proceedings of the CoNLL 2018 Shared Task: Multilingual
Parsing from Raw Text to Universal Dependencies.
Association for Computational Linguistics.

Allan Hanbury, Henning Müller, Krisztian Balog, Torben Brodt, Gordon V. Cormack, Ivan Eggel, Tim
Gollub, Frank Hopfgartner, Jayashree KalpathyCramer, Noriko Kando, Anastasia Krithara, Jimmy
Lin, Simon Mercer, and Martin Potthast. 2015.
Evaluation-as-a-Service: Overview and Outlook.
ArXiv e-prints http://arxiv.org/abs/1512.07454.

Piotr Bojanowski, Edouard Grave, Armand Joulin,
and Tomas Mikolov. 2016.
Enriching word
vectors with subword information arXiv preprint
arXiv:1607.04606.

Daniel Hershcovich, Omri Abend, and Ari Rappoport.
2018. Universal dependency parsing with a general transition-based DAG parser. In Proceedings of
the CoNLL 2018 Shared Task: Multilingual Parsing
from Raw Text to Universal Dependencies. Association for Computational Linguistics.

Tiberiu Boros, , Stefan Daniel Dumitrescu, and Ruxandra Burtica. 2018. NLP-Cube: End-to-end raw text
processing with neural networks. In Proceedings of
the CoNLL 2018 Shared Task: Multilingual Parsing
from Raw Text to Universal Dependencies. Association for Computational Linguistics.

Ganesh Jawahar, Benjamin Muller, Amal Fethi, Louis
Martin, Éric de La Clergerie, Benoı̂t Sagot, andDjamé Seddah. 2018.
ELMoLex: Connecting
ELMo and lexicon features for dependency parsing.
In Proceedings of the CoNLL 2018 Shared Task:
Multilingual Parsing from Raw Text to Universal Dependencies. Association for Computational Linguistics.

Sabine Buchholz and Erwin Marsi. 2006. CoNLL-X
shared task on multilingual dependency parsing. In
Proceedings of the 10th Conference on Computational Natural Language Learning (CoNLL-X). Association for Computational Linguistics, pages 149–
164.
http://anthology.aclweb.org/W/W06/W0629.pdf#page=165.

Tao Ji, Yufang Liu, Yijun Wang, Yuanbin Wu, and Man
Lan. 2018. AntNLP at CoNLL 2018 shared task: Agraph-based parser for universal dependency parsing. In Proceedings of the CoNLL 2018 Shared
Task: Multilingual Parsing from Raw Text to Universal Dependencies. Association for Computational
Linguistics.

Wanxiang Che, Yijia Liu, Yuxuan Wang, Bo Zheng,
and Ting Liu. 2018. Towards better UD parsing: Deep contextualized word embeddings, ensemble, and treebank concatenation. In Proceedings of
the CoNLL 2018 Shared Task: Multilingual Parsing
from Raw Text to Universal Dependencies. Association for Computational Linguistics.

Jenna Kanerva, Filip Ginter, Niko Miekka, Akseli
Leino, and Tapio Salakoski. 2018. Turku neural parser pipeline: An end-to-end system for the
CoNLL 2018 shared task. In Proceedings of the
CoNLL 2018 Shared Task: Multilingual Parsing
from Raw Text to Universal Dependencies. Association for Computational Linguistics.

Danlu Chen, Mengxiao Lin, Zhifeng Hu, and Xipeng
Qiu. 2018. A simple yet effective joint training
method for cross-lingual universal dependency parsing. In Proceedings of the CoNLL 2018 Shared
Task: Multilingual Parsing from Raw Text to Universal Dependencies. Association for Computational
Linguistics.

Ömer Kırnap, Erenay Dayanık, and Deniz Yuret. 2018.
Tree-stack LSTM in transition based dependency
parsing. In Proceedings of the CoNLL 2018 Shared
Task: Multilingual Parsing from Raw Text to Universal Dependencies. Association for Computational
Linguistics.

Elie Duthoo and Olivier Mesnard. 2018. CEA LIST
: Processing low-resource languages for CoNLL
2018. In Proceedings of the CoNLL 2018 Shared
Task: Multilingual Parsing from Raw Text to Universal Dependencies. Association for Computational
Linguistics.

Zuchao Li, Shexia He, Zhuosheng Zhang, and Hai
Zhao. 2018. Joint learning of pos and dependencies
for multilingual universal dependency parsing. In
Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. Association for Computational Linguistics.

Murhaf Fares, Stephan Oepen, Lilja Øvrelid, Jari
Björne, and Richard Johansson. 2018. The 2018
shared task on extrinsic parser evaluation: On thedownstream utility of English universal dependency
parsers. In Proceedings of the CoNLL 2018 Shared
Task: Multilingual Parsing from Raw Text to Universal Dependencies. Association for Computational
Linguistics, Brussels, Belgium.

KyungTae Lim, Cheoneum Park, Changki Lee, andThierry Poibeau. 2018. SEx BiST: A multi-source
trainable parser with deep contextualized lexical representations. In Proceedings of the CoNLL 2018
Shared Task: Multilingual Parsing from Raw Text toUniversal Dependencies. Association for Computational Linguistics.

Filip Ginter, Jan Hajič, Juhani Luotolahti, Milan Straka,
and Daniel Zeman. 2017. CoNLL 2017 shared task -18Lương Nguyễn Thị, Huyền Nguyễn Thị Minh,
Vitaly Nikolaev, Rattima Nitisaroj, Hanna Nurmi,
Stina Ojala, Adédayọ̀ Olúòkun, Mai Omura,
Petya Osenova, Robert Östling, Lilja Øvrelid,
Niko Partanen, Elena Pascual, Marco Passarotti,
Agnieszka Patejuk, Siyao Peng, Cenel-Augusto
Perez, Guy Perrier, Slav Petrov, Jussi Piitulainen,
Emily Pitler, Barbara Plank, Thierry Poibeau,
Martin Popel, Lauma Pretkalniņa, Sophie Prévost,
Prokopis Prokopidis, Adam Przepiórkowski, Tiina
Puolakainen, Sampo Pyysalo, Andriela Rääbis,
Alexandre Rademaker, Loganathan Ramasamy,
Taraka Rama, Carlos Ramisch, Vinit Ravishankar,
Livy Real, Siva Reddy, Georg Rehm, Michael
Rießler, Larissa Rinaldi, Laura Rituma, Luisa
Rocha, Mykhailo Romanenko, Rudolf Rosa, Davide Rovati, Valentin Ros, ca, Olga Rudina, Shoval
Sadde, Shadi Saleh, Tanja Samardžić, StephanieSamson, Manuela Sanguinetti, Baiba Saulı̄te,
Yanin Sawanakunanon, Nathan Schneider, Sebastian Schuster, Djamé Seddah, Wolfgang Seeker,
Mojgan Seraji, Mo Shen, Atsuko Shimada, Muh
Shohibussirri, Dmitry Sichinava, Natalia Silveira,Maria Simi, Radu Simionescu, Katalin Simkó,
Mária Šimková, Kiril Simov, Aaron Smith, Isabela
Soares-Bastos, Antonio Stella, Milan Straka, Jana
Strnadová, Alane Suhr, Umut Sulubacak, Zsolt
Szántó, Dima Taji, Yuta Takahashi, Takaaki Tanaka,
Isabelle Tellier, Trond Trosterud, Anna Trukhina,
Reut Tsarfaty, Francis Tyers, Sumire Uematsu,
Zdeňka Urešová, Larraitz Uria, Hans Uszkoreit,
Sowmya Vajjala, Daniel van Niekerk, Gertjan van
Noord, Viktor Varga, Veronika Vincze, Lars Wallin,
Jonathan North Washington, Seyi Williams, MatsWirén, Tsegay Woldemariam, Tak-sum Wong,
Chunxiao Yan, Marat M. Yavrumyan, Zhuoran Yu,
Zdeněk Žabokrtský, Amir Zeldes, Daniel Zeman,
Manying Zhang, and Hanzhi Zhu. 2018. Universal
dependencies 2.2. LINDAT/CLARIN digital libraryat the Institute of Formal and Applied Linguistics(ÚFAL), Faculty of Mathematics and Physics,
Charles University. http://hdl.handle.net/11234/12837.

Dat Quoc Nguyen and Karin Verspoor. 2018. An
improved neural network model for joint pos tagging and dependency parsing. In Proceedings of
the CoNLL 2018 Shared Task: Multilingual Parsing
from Raw Text to Universal Dependencies. Association for Computational Linguistics.
Joakim Nivre, Mitchell Abrams, Željko Agić, Lars
Ahrenberg, Lene Antonsen, Maria Jesus Aranzabe,
Gashaw Arutie, Masayuki Asahara, Luma Ateyah,
Mohammed Attia, Aitziber Atutxa, Liesbeth
Augustinus, Elena Badmaeva, Miguel Ballesteros, Esha Banerjee, Sebastian Bank, Verginica
Barbu Mititelu, John Bauer, Sandra Bellato, Kepa
Bengoetxea, Riyaz Ahmad Bhat, Erica Biagetti,
Eckhard Bick, Rogier Blokland, Victoria Bobicev,
Carl Börstell, Cristina Bosco, Gosse Bouma, Sam
Bowman, Adriane Boyd, Aljoscha Burchardt, Marie
Candito, Bernard Caron, Gauthier Caron, Gülşen
Cebiroğlu Eryiğit, Giuseppe G. A. Celano, Savas
Cetin, Fabricio Chalub, Jinho Choi, Yongseok Cho,
Jayeol Chun, Silvie Cinková, Aurélie Collomb,Çağrı Çöltekin, Miriam Connor, Marine Courtin,
Elizabeth Davidson, Marie-Catherine de Marneffe, Valeria de Paiva, Arantza Diaz de Ilarraza,
Carly Dickerson, Peter Dirix, Kaja Dobrovoljc,
Timothy Dozat, Kira Droganova, Puneet Dwivedi,
Marhaba Eli, Ali Elkahky, Binyam Ephrem, Tomaž
Erjavec, Aline Etienne, Richárd Farkas, Hector
Fernandez Alcalde, Jennifer Foster, Cláudia Freitas,
Katarína Gajdošová, Daniel Galbraith, Marcos
Garcia, Moa Gärdenfors, Kim Gerdes, Filip Ginter, Iakes Goenaga, Koldo Gojenola, MemduhGökırmak, Yoav Goldberg, Xavier Gómez Guinovart, Berta Gonzáles Saavedra, Matias Grioni,
Normunds Grūzı̄tis, Bruno Guillaume, Céline
Guillot-Barbance, Nizar Habash, Jan Hajič, Jan
Hajič jr., Linh Hà Mỹ, Na-Rae Han, Kim Harris,
Dag Haug, Barbora Hladká, Jaroslava Hlaváčová,
Florinel Hociung, Petter Hohle, Jena Hwang, Radu
Ion, Elena Irimia, Tomáš Jelínek, Anders Johannsen,
Fredrik Jørgensen, Hüner Kaşıkara, Sylvain Kahane,
Hiroshi Kanayama, Jenna Kanerva, Tolga Kayadelen, Václava Kettnerová, Jesse Kirchner, Natalia
Kotsyba, Simon Krek, Sookyoung Kwak, Veronika
Laippala, Lorenzo Lambertino, Tatiana Lando,
Septina Dian Larasati, Alexei Lavrentiev, John
Lee, Phương Lê Hồng, Alessandro Lenci, Saran
Lertpradit, Herman Leung, Cheuk Ying Li, Josie Li,
Keying Li, KyungTae Lim, Nikola Ljubešić, Olga
Loginova, Olga Lyashevskaya, Teresa Lynn, Vivien
Macketanz, Aibek Makazhanov, Michael Mandl,
Christopher Manning, Ruli Manurung, Cătălina
Mărănduc, David Mareček, Katrin Marheinecke,
Héctor Martínez Alonso, André Martins, Jan
Mašek, Yuji Matsumoto, Ryan McDonald, Gustavo
Mendonça, Niko Miekka, Anna Missilä, Cătălin
Mititelu, Yusuke Miyao, Simonetta Montemagni,
Amir More, Laura Moreno Romero, Shinsuke
Mori, Bjartur Mortensen, Bohdan Moskalevskyi,
Kadri Muischnek, Yugo Murawaki, Kaili Müürisep,
Pinkey Nainwani, Juan Ignacio Navarro Horñiacek,
Anna Nedoluzhko, Gunta Nešpore-Bērzkalne,

Joakim Nivre, Marie-Catherine de Marneffe, Filip
Ginter, Yoav Goldberg, Jan Hajič, Christopher
Manning, Ryan McDonald, Slav Petrov, Sampo
Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel
Zeman. 2016.
Universal Dependencies v1: A
multilingual treebank collection. In Proceedingsof the 10th International Conference on Language
Resources and Evaluation (LREC 2016). European Language Resources Association, Portorož,Slovenia, pages 1659–1666.
http://www.lrecconf.org/proceedings/lrec2016/summaries/348.html.
Joakim Nivre and Chiao-Ting Fang. 2017. Universal dependency evaluation. In Proceedings of the
NoDaLiDa 2017 Workshop on Universal Dependencies (UDW 2017). pages 86–95.
Joakim Nivre, Johan Hall, Sandra Kübler, Ryan
McDonald, Jens Nilsson, Sebastian Riedel,and Deniz Yuret. 2007.
The CoNLL 2007

19

Amit Seker, Amir More, and Reut Tsarfaty. 2018. Universal morpho-syntactic parsing and the contribution of lexica: Analyzing the ONLP submission to
the CoNLL 2018 shared task. In Proceedings of
the CoNLL 2018 Shared Task: Multilingual Parsing
from Raw Text to Universal Dependencies. Association for Computational Linguistics.

shared task on dependency parsing.
In Proceedings of the CoNLL Shared Task Sessionof EMNLP-CoNLL 2007. Association for
Computational Linguistics, pages 915–932.
http://www.aclweb.org/anthology/D/D07/D071.pdf#page=949.
Berkay Furkan Önder, Can Gümeli, and Deniz Yuret.
2018. SParse: Koç University graph-based parsing
system for the CoNLL 2018 shared task. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. Association for Computational Linguistics.

Aaron Smith, Bernd Bohnet, Miryam de Lhoneux,
Joakim Nivre, Yan Shao, and Sara Stymne. 2018. 82
treebanks, 34 models: Universal dependency parsing with multi-treebank models. In Proceedings of
the CoNLL 2018 Shared Task: Multilingual Parsing
from Raw Text to Universal Dependencies. Association for Computational Linguistics.

Şaziye Betül Özateş, Arzucan Özgür, Tunga Güngör,
and Balkız Öztürk. 2018. A morphology-based
representation model for LSTM-based dependency
parsing of agglutinative languages. In Proceedingsof the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. Association for Computational Linguistics.

Milan Straka. 2018. UDPipe 2.0 prototype at CoNLL
2018 UD shared task. In Proceedings of the CoNLL
2018 Shared Task: Multilingual Parsing from Raw
Text to Universal Dependencies. Association for
Computational Linguistics.
Milan Straka, Jan Hajič, and Jana Straková. 2016. UDPipe: trainable pipeline for processing CoNLL-U
files performing tokenization, morphological analysis, POS tagging and parsing. In Proceedingsof the 10th International Conference on Language
Resources and Evaluation (LREC 2016). European
Language Resources Association, Portorož, Slovenia.

Martin Popel, Zdeněk Žabokrtský, and Martin
Vojtek. 2017.
Udapi: Universal API for universal dependencies. In NoDaLiDa 2017 Workshop on Universal Dependencies. Göteborgs
universitet, Göteborg, Sweden, pages 96–
101.
http://aclweb.org/anthology/W/W17/W170412.pdf.
Martin Potthast, Tim Gollub, Francisco Rangel, Paolo
Rosso, Efstathios Stamatatos, and Benno Stein.
2014. Improving the reproducibility of PAN’s
shared tasks: Plagiarism detection, author identification, and author profiling.
In Evangelos
Kanoulas, Mihai Lupu, Paul Clough, Mark Sanderson, Mark Hall, Allan Hanbury, and Elaine Toms,
editors, Information Access Evaluation meets Multilinguality, Multimodality, and Visualization. 5thInternational Conference of the CLEF Initiative(CLEF 14). Springer, Berlin Heidelberg New York,pages 268–299. https://doi.org/10.1007/978-3-31911382-1_22.

Milan Straka and Jana Straková. 2017. Tokenizing,
POS tagging, lemmatizing and parsing UD 2.0 with
UDPipe. In Proceedings of the CoNLL 2017 Shared
Task: Multilingual Parsing from Raw Text to Universal Dependencies. Association for Computational
Linguistics.
Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck,
Mehmet Ugur Dogan, Bente Maegaard, Joseph
Mariani, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Eight International Conference on Language Resources and Evaluation(LREC’12). European Language Resources Association (ELRA), Istanbul, Turkey.

Peng Qi, Timothy Dozat, Yuhao Zhang, and Christopher D. Manning. 2018. Universal dependency parsing from scratch. In Proceedings of the CoNLL 2018
Shared Task: Multilingual Parsing from Raw Text toUniversal Dependencies. Association for Computational Linguistics.

Hui Wan, Tahira Naseem, Young-Suk Lee, Vittorio
Castelli, and Miguel Ballesteros. 2018. IBM Research at the CoNLL 2018 shared task on multilingual parsing. In Proceedings of the CoNLL 2018
Shared Task: Multilingual Parsing from Raw Text toUniversal Dependencies. Association for Computational Linguistics.

Rudolf Rosa and David Mareček. 2018. CUNI x-ling:
Parsing under-resourced languages in CoNLL 2018
UD shared task. In Proceedings of the CoNLL 2018
Shared Task: Multilingual Parsing from Raw Text toUniversal Dependencies. Association for Computational Linguistics.

Yingting Wu, Hai Zhao, and Jia-Jun Tong. 2018. Multilingual universal dependency parsing from raw text
with low-resource language enhancement. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. Association for Computational Linguistics.

Piotr Rybak and Alina Wróblewska. 2018. Semisupervised neural system for tagging, parsing andlematization. In Proceedings of the CoNLL 2018
Shared Task: Multilingual Parsing from Raw Text toUniversal Dependencies. Association for Computational Linguistics.

Daniel Zeman, Martin Popel, Milan Straka, Jan
Hajič, Joakim Nivre, Filip Ginter, Juhani Luoto-

20

lahti, Sampo Pyysalo, Slav Petrov, Martin Potthast, Francis Tyers, Elena Badmaeva, MemduhGökırmak, Anna Nedoluzhko, Silvie Cinková, Jan
Hajič jr., Jaroslava Hlaváčová, Václava Kettnerová,
Zdeňka Urešová, Jenna Kanerva, Stina Ojala, Anna
Missilä, Christopher Manning, Sebastian Schuster,Siva Reddy, Dima Taji, Nizar Habash, Herman Leung, Marie-Catherine de Marneffe, Manuela Sanguinetti, Maria Simi, Hiroshi Kanayama, Valeria
de Paiva, Kira Droganova, Héctor Martínez Alonso,Çağrı Çöltekin, Umut Sulubacak, Hans Uszkoreit, Vivien Macketanz, Aljoscha Burchardt, Kim
Harris, Katrin Marheinecke, Georg Rehm, Tolga
Kayadelen, Mohammed Attia, Ali Elkahky, Zhuoran Yu, Emily Pitler, Saran Lertpradit, Michael
Mandl, Jesse Kirchner, Hector Fernandez Alcalde,Jana Strnadova, Esha Banerjee, Ruli Manurung, Antonio Stella, Atsuko Shimada, Sookyoung Kwak,
Gustavo Mendonça, Tatiana Lando, Rattima Nitisaroj, and Josie Li. 2017. Conll 2017 shared task:
Multilingual parsing from raw text to universal dependencies. In Proceedings of the CoNLL 2017
Shared Task: Multilingual Parsing from Raw Text toUniversal Dependencies. Association for Computational Linguistics, Vancouver, Canada, pages 1–19.
http://www.aclweb.org/anthology/K17-3001.

21