Cross-Lingual Investigations on Small Treebanks: A Case Study on Parallel Universal Dependencies
Authors: Felice Dell'Orletta, Simonetta Montemagni, Giulia Venturi, Chiara Alzetta (CNR-ILC, 2021)
Topic: Syntax and Dependency Treebanks

Abstract
We illustrate an approach for multilingual treebanks explorations by introducing a novel adaptation to small treebanks of a methodology for identifying cross-lingual quantitative trends in the distribution of dependency relations. 
By relying on the principles of cross-validation, we reduce the amount of data required to execute the method, paving the way to expanding its use to low-resources languages. We validated the approach on 8 small treebanks, each containing less than 100,000 tokens and representing typologically different languages. We also show preliminary but promising evidence on the use of the proposed methodology for treebank expansion.

Introduction
Linguistically-annotated language resources like treebanks are fundamental for developing reliable models to train and test tools used to address Natural Language Processing (NLP) tasks acquiring linguistic evidence from corpora. Concerning the latter,  researchers frequently rely on multilingual or parallel resources in contrastive studies to quantify the similarities and differences between languages <cit.>. Over the past few years, the Universal Dependencies (UD) initiative[<https://universaldependencies.org>] <cit.> has further encouraged such studies. UD defines a universal inventory of categories and guidelines to facilitate consistent annotation of similar constructions across languages <cit.>, and, at present, the project includes about 200 treebanks representing over 100 languages. The consistent annotation of linguistic phenomena under a shared representation and across different languages makes UD treebanks exceptionally well suited for quantitative comparison of languages (see, for example, croft2017linguistic, berdicevskis2018using, vylomova2020proceedings and among our works, alzetta2019inferring and alzetta2020quantitative).    Despite their great relevance for linguistic investigations, large treebanks are available for only a tiny fraction of the world's languages <cit.>. Even within the UD project, around 60% of the treebanks can be considered small, i.e. containing less than 100,000 tokens. Treebank size, in fact, is generally identified as the bottleneck for obtaining high-quality representative models of language use to be employed in downstream NLP applications. In general terms, larger datasets allow for better generalisations of language constructions, leading to better performances of systems trained using such data <cit.>. In fact, ad-hoc strategies are generally needed when dealing with low-resourced languages <cit.>.    This paper illustrates a novel workflow specifically designed to adapt an existing methodology for treebank exploration to small treebanks.   The base method, extensively described by alzetta2020linguistically, relies on an unsupervised algorithm called LISCA (LInguistically–driven Selection of Correct Arcs) <cit.>. LISCA has been successfully employed in past works for performing quantitative cross-lingual analyses <cit.> and error detection on UD treebanks <cit.>. The algorithm works in two main steps. First, it acquires evidence about language use from the distributions of phenomena in annotated sentences. The algorithm   then uses such evidence to distinguish typical from atypical constructions in an unseen set of sentences. The typicality of a construction is determined with respect to the examples observed in a corpus used as a reference, and is encoded with a score. This score, in fact, reflects the probability of observing a dependency occurring in a given context (both sentence-level and corpus-level) on the basis of the constructions sharing common properties reported in the reference corpus. Hence, from our point of view, typicality and frequency are tightly related concepts, as non-standard constructions are also usually less frequent in natural language use.   As such, the LISCA methodology relies on large sets of automatically parsed sentences to collect the statistics about phenomena distributions: even if the data contains parsing errors[An assumption when producing automatically parsed data is that most of the errors made by a parser are consistent. As we showed in <cit.>, the LISCA-based method allows to spot these errors types in annotations.], the corpus size guarantees the collected statistics reflect the actual language use.  However, such an approach can be employed only for analysing languages for which large amounts of data are available, or at least for which the parser outputs are generally considered reliable.  To overcome such a limit, aaggarwalthesis suggested that if the statistics are acquired from gold annotations (such as treebanks), the algorithm could collect the statistics from fewer data since these resources are assumed to be error-free.    We implemented this proposal by adapting the original LISCA workflow as detailed in Section <ref>. Our variation to the original methodology is inspired by the k-fold approach commonly used for performing systems' cross-validation: according to this approach, a dataset is split into sub-sets of equal size, iteratively used for training and/or evaluating a system. We employ a similar strategy for evaluating the typicality of the dependency relations in each treebank split, acquiring the statistics from the sentences contained in the other splits rather than from an external reference corpus. This small but substantial change in the method workflow allows us to apply the LISCA algorithm to small treebanks, which is particularly relevant in the case of analyses performed on low-resource languages.   We tested the methodology in a case study, reported in Section <ref>, involving 8 languages represented using UD treebanks. Our goal is to test if our method can support linguistic investigations for exploring and quantifying similarities and differences between  typologically different languages. To this aim, we first validate the adaptation to the original LISCA approach proposed here in Section <ref>. Then, we exemplify how the obtained results can be employed for linguistic investigations in Section <ref>.  To improve the cross–linguistic comparability of the analysis, we relied on Parallel UD (PUD) treebanks: a collection of parallel treebanks developed for the CoNLL–2017 Shared Task on multilingual parsing <cit.> and linguistically annotated under the UD representation. Being parallel, PUDs are particularly well suited for carrying out multilingual studies since they contain only 1,000 sentences manually translated from English into the other languages, representing a perfect testbed for our approach.   Before concluding the paper in Section <ref>, we report  the results of preliminary investigations to explore whether our approach could also be employed for automatically identifying underrepresented phenomena in treebanks. leakiness and leakiness2 argue that some treebanks cover only a restricted sample of the structures commonly used in a language, leaving out less common phenomena. This leakiness might affect the performances of NLP systems even more than the system architecture. Thus, treebanks should be expanded not only to improve their representativeness but also to obtain more truthful performances of systems trained using them.  Section <ref> investigates if our methodology can contribute to this issue by exploring its application in automatic treebank expansion.  The contributions of the paper can be listed as: (i) a novel approach specifically designed for carrying out multilingual investigations on small treebanks; (ii) a case study involving eight typologically different languages to test the methodology; and (iii)  a novel formula, introduced in Section <ref>, to measure the distance between dependents and their syntactic head which improves the cross-lingual comparability of treebanks with respect to such property.

Conclusion
We proposed a novel workflow to adapt an existing approach for treebank exploration to small treebanks and low-resourced languages. Results of our analyses showed the effectiveness of the methodology in multiple scenarios. First, the adapted method allows obtaining reliable results on par with the original method workflow when performing linguistic explorations of the treebanks. Secondly, the results also show the potential of the method for automatically identifying underrepresented constructions in treebanks. The latter result paves the way for the automatic identification of cases required to expand the treebanks, which we plan to further investigate in future work.