言語処理学会 第30回年次大会 発表論文集（2024年3月）

Improving Zero-Shot Dependency Parsing
by Unsupervised Learning
Jiannan Mao1,2 Chenchen Ding2 Hour Kaing2 Hideki Tanaka2
Masao Utiyama2 Tadahiro Matsumoto1
1 Gifu University, Gifu, Japan
2 ASTREC, UCRI, NICT, Japan{mao, tad}@mat.info.gifu-u.ac.jp{chenchen.ding, hour kaing, hideki.tanaka, mutiyama}@nict.go.jp

Abstract
UDify [1] is a multilingual and multi-task parser ﬁnetuned on mBERT. It has demonstrated notable performance, even on few-shot languages. However, its performance saturates early and decreases gradually as training
progresses with zero-shot languages. This work focuses on
this phenomenon, which has not yet been suﬃciently studied. Data augmentation methods based on unsupervised
learning on monolingual data were designed to transform azero-shot case into an artiﬁcial few-shot case. Experiments
were conducted on the Breton language as a typical casestudy. For the Breton language, the unlabeled attachmentscore was signiﬁcantly improved. The parsing accuracies
for other languages were not noticeably aﬀected.

1

Introduction

A dependency parser can be eﬃciently trained on large
treebanks when available [2, 3]. For low-resource languages with limited treebanks, multilingual modeling has
emerged as an eﬃcient solution in which cross-lingual information is leveraged to compensate for the lack of dataon speciﬁc languages. Many research [4, 5] have demonstrated that the performance in part-of-speech (POS) or
dependency parsing can be improved by pairing languages
with similar. This multilingual approach reduces the costof training multiple models for a language group [6].
UDify [1] is a multi-task self-attention network ﬁnetuned on multilingual BERT (mBERT) [7] pre-trained embeddings, capable of producing annotations for any treebank from Universal Dependencies treebanks [8]. UDify
exhibits strong and consistent performance across a wide
range of languages and tasks such as lemmatization, POS

Figure 1 Change in the UAS of a model during the training
process on the Breton test set for both the baselines (Baseline andSelf) and the proposed method (Unsup).

tags, and dependency parsing. In addition to UDify, there
are several comparable state-of-the-art methods[5, 9]. A
problem highlighted by several related studies is the substantial discrepancy in the performance of these methods inzero-shot scenarios, even with identical training strategies,
datasets, models, and evaluation methods [9, 10].
This work investigates the underlying reason for the phenomenon regarding zero-shot cases by exhaustively examining the epochs during model training. To resolve thisproblem, a data augmentation strategy is proposed to improve the performance and stability of UDify. Speciﬁcally,
the original UDify provides an initialization based on unsupervised learning for a zero-shot language; the generated
results by unsupervised learning are then incorporated into
UDify’s training set so that the zero-shot language is converted into an artiﬁcial few-shot language.
Experiments on the Breton are taken as a case study.
Unsupervised learning-based data augmentation eﬃciently
boosted the unlabeled attachment score (UAS) from 68.4%
to 76.1%. Furthermore, the parsing accuracy for otherlanguages did not decrease, which suggests that the overallrobustness of multilingualism processing is still retained.

― 1645 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

2

Background3

Proposed Method

2.1

UDify

3.1

Motivation

The UDify model jointly predicts lemmas, POS tags,

In UDify’s training, the dependency structures of zero-dependency structures, and etc. The pre-trained mBERT

shot languages are learned through transfer learning. Inmodel is a self-attention network with 12 encoder layers.

contrast to the high-resource and few-shot languages, early

It is used in the UDify model for cross-lingual learning

saturation in the accuracy of the dependency parsing is

without additional tags to distinguish the languages. In ad-

exhibited during the learning process of all low-resource

dition, a strategy similar to that of ELMo [11] is adopted,

languages. The best performance was typically achieved

where a weighted sum of the outputs of all layers is com-around the 8th training epoch, as shown in Figure 2.

puted as follows and fed to a task-speciﬁc classiﬁer:
𝑒 𝑡𝑗𝑎𝑠𝑘 =∑𝑖 𝑚𝐵𝐸 𝑅𝑇𝑖 𝑗 .

Here, 𝑒 𝑡 𝑎𝑠𝑘 denotes the contextual output embeddings
for tasks such as the dependency parse, 𝑚𝐵𝐸 𝑅𝑇𝑖 𝑗 denotes
the 𝑚𝐵𝐸 𝑅𝑇 representation for layer 𝑖 at token position 𝑗.
In the task involving dependency structures, mBERT’s
subword tokenization process inputs words into multiple
subword units. Despite this, only the embeddings 𝑒 𝑡𝑗𝑎𝑠𝑘of the ﬁrst subword unit are used, serving as input to the

Figure 2 Changes in the UAS of zero-shot languages during
the training process.

graph-based bi-aﬃne attention classiﬁer [2]. The resulting

This phenomenon has been noticed [5, 9, 10] but not yet

outputs are combined using bi-aﬃne attention to produce

systematically investigated to the best of our knowledge.

a probability distribution of the arc-head for each word.

Given that the accuracy of dependency parsing for zero-

Finally, the dependency tree is decoded using the Chu–
Liu/Edmonds algorithm [12, 13].

shot languages tends to decrease as training progresses, the2.2

Unsupervised Dependency Learning

number of epochs becomes a crucial factor in the inconsistency observed in related work.
Considering the observed positive correlation between

Adhering to the properties of dependency syntax [14], aparsing accuracy and the number of training epochs forgeneral unsupervised algorithm for projective N-gram dependency learning (Unsupervised-Dep) was described inhigh-resource and few-shot languages, it is somewhat unexpected to encounter a substantial discrepancy between

Ding [15]. This method employs a CYK-style chart and dy-

the optimal and ﬁnal testing results for zero-shot tasks.

namic programming to construct the optimal dependency

This calls for specialized strategies to bridge this gap. In

tree, grounded in the non-constituent concepts of complete-

the following, we introduce a data augmentation techniquelink and complete-sequence. Due to the signiﬁcant time

grounded on Unsupervised-Dep, and aims to reduce thecomplexity associated with N-gram learning, this study

performance gap and thus improve the eﬀectiveness of UD-concentrates on bi-gram for practical applicability.

ify in dependency parsing for zero-shot languages.

When considering the bi-gram, the directionality of apair of words is set by the dependency relation, with(𝑤 𝑖 →𝑤 𝑗 ) indicating a rightward relation and (𝑤 𝑖 ←𝑤 𝑗 ) in-

3.2

Unsupervised Augmentation

To apply Unsupervised-Dep in data augmentation, it isdicating a leftward one. The bi-gram unsupervised learn-vital to ensure the generated data aligns with the UD for-ing update probabilities 𝑃(𝑤 𝑖 →𝑤 𝑗 ) and 𝑃(𝑤 𝑖 ←𝑤 𝑗 ) are

mat. Therefore, in addition to dependency arc-heads, othercalculated using the Inside–Outside algorithm [16]. Fi-

types of data must be created and combined with the results

nally, the Viterbi algorithm [17] is employed to determine

from Unsupervised-Dep. Given its high time complexityof 𝑂 (𝑛3 ), making the common practice in the original

the optimal tree construction.

― 1646 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

methods, which start training from a random probability,
somewhat ineﬃcient. To circumvent this, we decided toTable 1Division of the OPUS-br into training, validation, andtest sets, and the number of words in each set.

data setleverage the parsing results from UDify to initialize the#sentence

#word

train

90,000 1,023,292

valid

9,000

113,066

nal results consistently outperform those from other models

test300

300

2,663

[8, 3], providing a robust foundation for our initialization

test500

500

4,469

probabilities. Despite the potential decrease in UDify’s
accuracy on zero-shot languages during its training, the ﬁ-

approach.
We initiate the process with the raw corpus, 𝐷𝑎𝑡𝑎,
input into the trained UDify by the original UD treebanks, to generate the dependency arc-heads, represented
as 𝐷𝐸 𝑃 𝑎𝑟 𝑐 , and POS, lemmas, etc., denoted as 𝑂𝑡ℎ𝑒𝑟 𝑠.
Statistical computations on 𝐷𝐸 𝑃 𝑎𝑟 𝑐 generate initial probabilities 𝑃(𝑤 𝑖 →𝑤 𝑗 ) and 𝑃(𝑤 𝑖 ←𝑤 𝑗 ), serving as input for
Unsupervised-Dep alongside 𝐷𝑎𝑡𝑎.
Following several iterations of training throughUnsupervised-Dep, the re-estimated 𝑃(𝑤 𝑖 →𝑤 𝑗 )

′

and

′

𝑃(𝑤 𝑖 ←𝑤 𝑗 ) emerge. They become the parameters for theViterbi algorithm to determine the optimal dependency
arc-head as given by′

the results of Unsupervised-Dep learning. The test set,demonstrates the results of our data augmentation.
To evaluate our proposed method, we used the sameversion of the UD treebank that UDify uses for our experiments. During training, we concatenated all training sets,
mirroring the approach of [18]. We shuﬄed all sentences
before each epoch and fed the mixed batch of sentences
into the network, including sentences from any language
or treebank, whether they were original UD treebank sentences or those generated through Unsupervised-Dep.

4.2

Setup

To minimize the impact of potential experimental envi′′

𝐷𝐸 𝑃 𝑎𝑟 𝑐 = 𝑉𝑖𝑡𝑒𝑟𝑏𝑖(𝑥, 𝑃(𝑤 𝑖 →𝑤 𝑗 ) , 𝑃(𝑤 𝑖 ←𝑤 𝑗 ) ) ,′where 𝐷𝐸 𝑃 𝑎𝑟 𝑐 is the tree with the highest probability fora sentence 𝑥 from 𝐷𝑎𝑡𝑎.
′

We merged 𝐷𝐸 𝑃 𝑎𝑟 𝑐 with 𝑂𝑡ℎ𝑒𝑟 𝑠, ultimately generating
artiﬁcial data. The artiﬁcial data are then combined withronmental variations [19], we follow the parameters andre-implemented the model provided by UDify2）, which werefer to as the Baseline. To expedite the experiment, we
implemented multi-GPU parallel training3）by modifying
UDify using Horovod [20].

the existing UD treebanks for the subsequent training.

For our method, we initially computed the parsing results of the training data from OPUS-br using the Base-4line. These results served as the initial probabilities forExperiments𝑃(𝑤 𝑖 →𝑤 𝑗 ) and 𝑃(𝑤 𝑖 ←𝑤 𝑗 ). These probabilities were

4.1

Datasetcontinuously re-estimated throughout the unsupervised

We selected Breton from OPUS 1） as our target zeroshot language for the implementation of UnsupervisedDep. After cleaning the collected data, we obtained 99.5k
sentences, which we refer to as OPUS-br.
From this collection, we ﬁrst identiﬁed a subset of 300
sentences as our test set, referred to as test300 . To conduct
more detailed testing of the sentence structures generated
by Unsupervised-Dep, we later expanded this subset byincorporating an additional 200 sentences, resulting in a
total of 500 sentences, referred to as test500 . The divisionof the collected data in the experiment is summarized inTable 1. The training set is used for obtaining and updating
the probabilities. The validation set is applied to validate
1） opus.nlpl.eu

learning process of training data from OPUS-br. After the
10th training epoch, we employed the re-estimated probabilities to parse the test set from OPUS-br. In multilingual
parser experiments, two diﬀerent subsets of the OPUS-br
dataset were employed to reﬁne the dependency arc-heads
generated by Unsupervised-Dep, speciﬁcally marked as
Unsup300 and Unsup500 . Unsup300 integrates sentencesfrom the OPUS-br test300 into the UDify training set, while
Unsup500 utilizes sentences from the OPUS-br test500 .
Meanwhile, inspired by Rybak [21], we conducted anexperiment using a comparative method dubbed Self300 .
In this approach, we used the test300 , diverging from the2） github.com/Hyperparticle/udify
3） When parallelizing across seven V100s, replicating UDify requires close to 12 days.

― 1647 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

use of Unsupervised-Dep for sentence reﬁnement. The
sentences were directly parsed with the Baseline instead.
The parsing results were then converted into UD format,
merged with the original training set, and used for a newround of UDify model training.

4.3

Result and Discussion

A comparison with the experimental results reported
in the original UDify study conﬁrms that Baseline was reimplemented successfully, as demonstrated in Table 2. The
results in this table not only indicate a signiﬁcant improvement in UDify’s ability to avoid the decrease in dependency
arc-head accuracy for the Breton language at the end of the

Figure 3

Changes in the UAS for the Baseline and Unsup500 on
all test treebanks. The x-axis sorts the UD treebanks by the ascending improvement of the proposed method over the Baseline.

Breton and other languages have been compiled in Table 2. Given that UDify must balance the loss produced

training, regardless of the proposed method that was implemented but also indicate that while self300 causes only aby multiple decoders during training and Rybak’s work

minor increase in the UAS score, Unsup300 and Unsup500 ,reasonable. Broadly, our data augmentation method has

which incorporate data generated from Unsupervised-Dep,
signiﬁcantly augment the accuracy of the dependency archead. Additionally, Figure 1 and Table 2 illustrates that,though diﬀerent datasets have minimal impact on the peak[21], these variations in evaluation metrics are consideredalmost no negative impact on other languages and tasks,
maintaining their performance levels.
Considering all results, we argue that generating training
data for zero-shot languages through the application of

UAS scores for Breton during training, methods Unsup300

the Unsupervised-Dep is both essential and eﬀective inand Unsup500 notably narrow the gap between the best andmultilingual modeling.

last UAS scores, thereby enhancing training stability.
Table 2UD scores on Breton and other languages obtained by
diﬀerent methods. Rest(%) refers to the average score of UPOS,
UFeats, Lemma, and LAS in the UD scores. The UDify result
was reported by Kondratyuk [1].

Breton5Conclusion
This study investigated the issue of decreased parsing ac-

curacy exhibited by UDify in zero-shot language scenarios,
despite its generally outstanding performance in few-shot

Otherlanguage scenarios. To address this problem, we proposed

UAS

Rest

UAS

Rest

a method that applies an unsupervised algorithm to transi-

last

best

gap

UDify

63.5

-

-

-

-

-

tion a zero-shot language into a few-shot language context,

Baseline

68.4 76.2

-7.8

52.4

77.7

82.3

thereby eﬀectively expanding the dataset and enhancing

Self300

70.1 76.2

-6.1

55.3

77.7

82.3

Unsup300

75.4 76.5

-1.1

60.9

77.7

82.4

the model’s learning capability.
The eﬃcacy of our approach has been substanti-

Unsup500

76.1 76.9

-0.8

61.1

77.7

82.3

ated through our experimental results.

By incorporat-

ing sentence dependency arc-head structures produced byConsidering UDify’s role as a multilingual parser, it is

Unsupervised-Dep into UDify’s training data, we achieved

necessary to evaluate the impact of the proposed method

a substantial improvement in UDify’s performance withon other languages. To observe in detail the diﬀerences andzero-shot languages. This improvement was signiﬁcantchanges in the UAS between the Baseline and Unsup500 ,even when only a limited number of sentences, such as 300

we conducted tests across all treebanks and display the
results in Figure 3. From the ﬁgure, it is evident that whileor 500, were used. Although our constraints did not allow

Unsup500 has improved the UAS for Breton, it has had

tween the number of sentences generated by Unsupervised-virtually no impact on the parsing precision of dependency

Dep incorporated into the training data and the improve-

constructions in other languages.

ment of UDify in zero-shot languages, this correlation re-

For a comprehensive comparison, the UD scores of thefor a deﬁnitive demonstration of a positive correlation be-

mains a possibility.

― 1648 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.

References[1] Dan Kondratyuk and Milan Straka. 75 languages, 1 model:
Parsing Universal Dependencies universally. In Proceedings of the 2019 Conference on Empirical Methods
in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2779–2795, Hong Kong,

China, November 2019. Association for Computational
Linguistics.
[2] Timothy Dozat and Christopher D. Manning. Deep biaﬃne
attention for neural dependency parsing. In 5th International Conference on Learning Representations,
ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.

8503–8520, Dublin, Ireland, May 2022. Association for
Computational Linguistics.
[10] Ahmet Üstün, Arianna Bisazza, Gosse Bouma, and Gertjan van Noord. UDapter: Language adaptation for truly
Universal Dependency parsing. In Proceedings of the2020 Conference on Empirical Methods in Natural
Language Processing (EMNLP), pp. 2302–2315, On-line, November 2020. Association for Computational Linguistics.
[11] Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt
Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In
Proceedings of the 2018 Conference of the North
American Chapter of the Association for Computational Linguistics: Human Language Technologies,
Volume 1 (Long Papers), pp. 2227–2237, New Orleans,[3] Peng Qi, Timothy Dozat, Yuhao Zhang, and Christopher D.
Manning. Universal Dependency parsing from scratch. In
Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pp. 160–170, Brussels, Belgium, October 2018.

[4]

[5]

[6]

[7]

Association for Computational Linguistics.
Waleed Ammar, George Mulcaire, Miguel Ballesteros,
Chris Dyer, and Noah A. Smith. Many languages, oneparser. Transactions of the Association for Computational Linguistics, Vol. 4, pp. 431–444, 2016.
Ahmet Üstün, Arianna Bisazza, Gosse Bouma, and Gertjan van Noord. UDapter: Typology-based language
adapters for multilingual dependency parsing and sequencelabeling. Computational Linguistics, Vol. 48, No. 3, pp.
555–592, September 2022.
Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim
Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduﬀ
Hughes, and Jeﬀrey Dean. Google’s multilingual neuralmachine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, Vol. 5, pp. 339–351, 2017.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina
Toutanova. BERT: Pre-training of deep bidirectionaltransformers for language understanding. In Proceedings of the 2019 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1(Long and Short Papers), pp. 4171–4186, Minneapo-lis, Minnesota, June 2019. Association for Computational
Linguistics.
[8] Daniel Zeman, Jan Hajič, Martin Popel, Martin Potthast,
Milan Straka, Filip Ginter, Joakim Nivre, and Slav Petrov.
CoNLL 2018 shared task: Multilingual parsing from raw
text to Universal Dependencies. In Proceedings of the
CoNLL 2018 Shared Task: Multilingual Parsing fromRaw Text to Universal Dependencies, pp. 1–21, Brus-sels, Belgium, October 2018. Association for Computational Linguistics.
[9] Anna Langedijk, Verna Dankers, Phillip Lippe, SanderBos, Bryan Cardenas Guevara, Helen Yannakoudakis, andEkaterina Shutova. Meta-learning for fast cross-lingual
adaptation in dependency parsing. In Proceedings of

[12]
[13]

[14]

[15]

Louisiana, June 2018. Association for Computational Linguistics.
Yoeng-Jin Chu. On the shortest arborescence of a directed
graph. Scientia Sinica, Vol. 14, pp. 1396–1400, 1965.
Jack Edmonds, et al. Optimum branchings. Journal of Research of the national Bureau of Standards B, Vol. 71,
No. 4, pp. 233–240, 1967.
Jane J. Robinson. Dependency structures and transformational rules. Language, Vol. 46, No. 2, pp. 259–285,
1970.
Chenchen Ding and Mikio Yamamoto. An unsupervised
parameter estimation algorithm for a generative dependency n-gram language model. In Proceedings of the
Sixth International Joint Conference on Natural Language Processing, pp. 516–524, Nagoya, Japan, Octo-

ber 2013. Asian Federation of Natural Language Processing.
[16] Karim Lari and Steve J Young. The estimation of stochastic
context-free grammars using the inside-outside algorithm.
Computer speech & language, Vol. 4, No. 1, pp. 35–56,
1990.
[17] G David Forney. The viterbi algorithm. Proceedings ofthe IEEE, Vol. 61, No. 3, pp. 268–278, 1973.
[18] Ryan McDonald, Slav Petrov, and Keith Hall. Multi-source
transfer of delexicalized dependency parsers. In Proceedings of the 2011 Conference on Empirical Methods in
Natural Language Processing, pp. 62–72, Edinburgh,

Scotland, UK., July 2011. Association for Computational
Linguistics.
[19] Martin Popel and Ondrej Bojar. Training tips for the transformer model. Prague Bull. Math. Linguistics, Vol. 110,
pp. 43–70, 2018.
[20] Alexander Sergeev and Mike Del Balso. Horovod: Fastand easy distributed deep learning in TensorFlow. arXiv
preprint arXiv:1802.05799, 2018.
[21] Piotr Rybak and Alina Wróblewska. Semi-supervised neural system for tagging, parsing and lematization. In Pro-

― 1649 ―ceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pp. 45–54, Brussels, Belgium, October 2018. Asso-

ciation for Computational Linguistics.

This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).