SlowGuess/NAACL_2022_OCR huggingface dataset:
Şaziye Betül Özates $^{1,2}$ , Arzucan Özgür $^{1}$ , Tunga Gungor $^{1}$ , Özlem Çetinoğlu $^{2}$
<sup>1</sup> Computer Engineering, Bogazici University, Turkey
2 IMS, University of Stuttgart, Germany
{saziye.bilgin,arzucan.ozgur,gungort}@boun.edu.tr
ozlem@ims.uni-stuttgart.de
Code-switching dependency parsing stands as a challenging task due to both the scarcity of necessary resources and the structural difficulties embedded in code-switched languages. In this study, we introduce novel sequence labeling models to be used as auxiliary tasks for dependency parsing of code-switched text in a semi-supervised scheme. We show that using auxiliary tasks enhances the performance of an LSTM-based dependency parsing model and leads to better results compared to an XLM-R-based model with significantly less computational and space complexity. As the first study that focuses on multiple code-switching language pairs for dependency parsing, we acquire state-of-the-art scores on all of the studied languages. Our best models outperform the previous work by 7.4 LAS points on average.
Code-switching (CS) is the producing of utterances by combining phrases and word forms from multiple languages. This is a phenomenon observed frequently in utterances of bilingual speakers (Auer and Wei, 2007). Figure 1 shows an example to this type of utterance formation. Although much work has been done on the syntactic parsing of monolingual languages, CS language pairs are quite understudied in this regard. There have been only a few studies on CS dependency parsing (Bhat et al., 2017; Partanen et al., 2018b; Braggaar and van der Goot, 2021), each focusing only on a single CS language pair. Although CS dependency parsing also benefited from the recent rise of multilingual and cross-lingual natural language processing (NLP) models as shown by van der Goot et al. (2021), these models, which are usually trained on monolingual corpora, are insufficient on CS parsing. The poor performance on CS language pairs is not only due to the lack or scarcity of the training data but also because of the shortage on resources required
'After secondary school I went to high school.'
Figure 1: Dependency tree of a code-switched sentence from the Turkish-German SAGT Treebank. Language ID of each token is located below the token. TR stands for Turkish, DE for German, MIXED for tokens with intra-word code-switching, OTHER is for punctuation. German tokens and token parts are shown in bold.
by deep neural models such as pretrained embeddings, language models, or even raw data. In addition, each language composing a CS language pair inherits its own structural difficulties which contributes a good deal to the problem.
Recently, a small number of CS treebanks were manually annotated within Universal Dependencies (UD) (Nivre et al., 2016). Even though these treebanks have little to no training data, their existence provides an opportunity to study dependency parsing also on CS language pairs.
In such low-resource scenarios, utilizing raw data can be helpful in boosting the performance. A common method to benefit from raw data is self-training (McClosky et al., 2006), a semi-supervised approach where a small number of labeled data is used to train a model that is later used to predict labels for unlabeled data. This pseudo-labeled data is then combined with the initial data to re-train the model. This method is usually found successful in low-resource scenarios (Rybak and Wroblewska, 2018; Yu et al., 2020), although error propagation is a known problem when pseudo-labels are noisy.
With very restricted resources, we hypothesize that CS dependency parsing can also benefit from unlabeled data. Based on this hypothesis, we form our first research question: is using pseudo-labeled data directly beneficial for CS dependency pars
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1159 - 1171
ing or can we find better ways of integrating the knowledge from pseudo-labeled data?
Starting from this question, we follow a deep contextualized self-training approach (Rotman and Reichart, 2019) and integrate semi-supervised auxiliary tasks to the parsing architecture to enhance CS dependency parsing. Our method enhances a widely-used BiLSTM-based parser (Dozat and Manning, 2017) by training parsing-related auxiliary sequence labeling tasks on automatically labeled data and combining these trained auxiliary task models with the base parser through a gating mechanism. We introduce new sequence labeling tasks that are shown to be beneficial in improving the parsing performance. Seeing the success of our semi-supervised enhancement method on the BiLSTM-based parser, we form our second research question: can we reach even better parsing scores if we combine this enhancement method with XLM-R (Conneau et al., 2020), a state-of-the-art (SOTA) transformer-based language model that shows superior performance on many NLP tasks? Our experimental results demonstrate notable success of our proposed models over the previous state-of-the-art on these treebanks. Our contributions are as follows:
- We employ a semi-supervised learning approach based on auxiliary tasks for CS dependency parsing. We present the first study with a focus on parsing all CS UD treebanks and achieve SOTA results on all of them.
- We introduce novel sequence labeling tasks including a CS-specific one, that capture syntactic information better and hence improve dependency parsing.
- We adapt this method to the powerful XLM-R model and elaborate the effectiveness of this approach when combined with XLM-R-based word representation for dependency parsing. We demonstrate that the mighty transformer model remains inadequate for the case of low-resource CS parsing.
Code-switching dependency parsing is a newly-studied research area. The first CS UD treebank was created by Bhat et al. (2017) which included only a test set of Hindi-English sentences. In the absence of CS training data, the test set was split to monolingual fragments and existing Hindi and
English monolingual treebanks in UD were used to parse these fragments. Bhat et al. (2018) extended this dataset with a CS training set. They trained a BiLSTM architecture on this additional training data by also integrating syntactic knowledge extracted from monolingual treebanks.
Partanen et al. (2018b) laid the first foundations of a Komi-Russian UD treebank with 25 CS sentences. They adopted a multilingual parsing approach (Lim and Poibea, 2017) and used Russian and Komi monolingual training data with bilingual Komi-Russian word embeddings. Later, this treebank expanded into the Komi-Zyrian IKDP treebank (Partanen et al., 2018a).
Çetinoğlu and Çöltekin (2019) created a Turkish-German UD treebank from a Turkish-German spoken corpus. Seddah et al. (2020) introduced the Maghrebi Arabic-French treebank and performed parsing experiments on the treebank using UDPipe (Straka and Straková, 2017). This treebank is yet to be included in the UD. A Frisian-Dutch UD treebank which includes only test data was introduced by Braggaar and van der Goot (2021). The authors performed data selection from eight related monolingual treebanks using Latent Dirichlet Allocation (Blei et al., 2003) to create a training set. Their experiments performed using a deep biaffine parser (van der Goot et al., 2021) demonstrated no significant performance difference between training the parser on the selected training set and only on a Dutch monolingual treebank.
Lately, multilingual and cross-lingual parsing studies have begun to include CS treebanks in their experimental setups. van der Goot et al. (2021) presented a multi-task learning tool that utilizes multilingual BERT (Devlin et al., 2019) to perform several NLP tasks, including dependency parsing. Evaluation was done on all available UD treebanks which include CS UD treebanks mentioned above. The model was fine-tuned on training set of each treebank, which is also the case for Hindi-English and Turkish-German CS treebanks. For Frisian-Dutch and Komi-Russian CS treebanks with no training data, they used Dutch Alpino and Russian SynTagRus treebanks, respectively. Müller-Eberstein et al. (2021) applied a sentence level genre-based data selection from UD treebanks in a cross-lingual setup. They trained a multilingual BERT-based biaffine parser (van der Goot et al., 2021) for 12 low-resource UD treebanks including Hindi-English and Turkish-German CS treebanks.
Figure 2: The parser architecture with semi-supervised auxiliary task enhancement.  $E_{p}$  is the parser encoder,  $E_{AT}$  is the sequence labeler encoder trained on one of the auxiliary tasks. For a given token pair, the model calculates a weighted average of each token's hidden representation from  $E_{P}$  and  $E_{AT}$ . The resulting vectors are given to two multi-layer perceptrons (MLP) to produce an arc score  $S_{arc}$  and a label score  $S_{label}$  for the given token pair. The input tokens are taken from the Frisian-Dutch Fame Treebank.
Our study on CS dependency parsing differs from the previous work in the sense that none of the previous work utilized raw CS data to improve parsing in a semi-supervised scheme.
3.1 Base Parsing Model
Our base parser is a neural graph-based parser by Dozat and Manning (2017) that uses two biaffine classifiers, one to predict the head of a given token and the other to predict the resulting arc's label. For input representation, the model uses BiLSTM modules to compute learned word embeddings and add them to their corresponding pretrained word embeddings that are later concatenated with corresponding part-of-speech (POS) embeddings. To ensure a well-formed tree at test time, the maximum spanning tree (MST) algorithm is used.
3.2 Semi-supervised Enhancement through Auxiliary Sequence Labeling Tasks
We follow Rotman and Reichart (2019) to exploit unlabeled data for CS dependency parsing. Rather than directly using pseudo-labeled data as an additional source in training, the main idea is to extract and utilize parsing-related knowledge from automatically parsed data. This is achieved by training contextualized embedding models on a number of auxiliary sequence labeling tasks derived from the raw data parsed by the base parser and then com
bining encoders of these trained models with that of the base parser through a gating procedure (Sato et al., 2017) as described in Section 3.3. Figure 2 depicts this enhanced parser. The combined model is then re-trained on the gold labeled data.
For their experimental setup, Rotman and Reichart (2019) consider three token-level sequence labeling schemes to extract the structural information encoded in the parsed sentences. These are:
(i) Number of Children (NOC) The task is to predict the number of children each token has in a dependency tree.
(ii) Distance to the Root (DTR) Each token is tagged with its minimum distance to the root token of the dependency tree.
(iii) Relative POS-based Encoding (RPE) Each token in a sentence is tagged with its head's POS tag in a simplified form and its distance from the head. The distance calculation considers only the intermediate tokens that share the same POS tag with its head.
Although these three auxiliary tasks offer a comprehensive scheme in terms of extracting parsing-related knowledge from automatically parsed data, we search ways of channeling the embedded knowledge in parsed trees more thoroughly to the trained word embedding layers of the parser. We come up with three additional sequence labeling tasks:
Figure 3: The dependency tree of an example sentence from Hindi-English HIENCS Treebank. Each node in the tree is tagged with the five auxiliary task schemes depicted in Section 3.2. Tags for the case of the SMH scheme are not shown for this example since the HIENCS Treebank does not include morphology.
(iv) Language ID of Head (LIH) We start with CS-specific features of parsed trees. The most prominent of them is the language ID (LID) features of the tokens in CS treebanks. Considering the positive impact of LIDs in various other NLP tasks (Jamatia et al., 2015; Aguilar and Solorio, 2020; Ozates and Cetinoğlu, 2021), we design a simple auxiliary sequence labeling task that makes use of LIDs. Unlike previous work using token LIDs, LIH tags each token with the LID of its head. This way, information about the language of tokens with which each token tends to relate in terms of dependencies is conveyed to the learning model.
(v) Simplified Morphology of Head (SMH)
Morphological features are found to be beneficial in parsing morphologically-rich languages (Dehouck and Denis, 2018). This was our motivation to create a new auxiliary task based on morphology. In the SMH scheme, each token is assigned its head's morphological features. To reduce the number of labels, we use only a subset of the morphological features set, selected by considering the inclusiveness and the prevalence of the features across the data. The main idea of SMH is to provide morphological clues to the parser while also giving information about the structure of the tree.
A similar approach is also tried by Sandhan et al. (2021). They define a sequential task to predict the full set of morphological features for a given token.
In our preliminary experiments, we observed that using the full set of morphological features does not improve the accuracy. In CS treebanks the unique number of features is increased due to the combination of language-specific feature sets of the language pair, making the task more complex. To reduce the complexity, we design SMH as utilizing only a subset of the morphological features of (not the token itself, but) the head of the token.
(vi) Punctuation Count (PC) Lastly, we design the PC task that only needs root tokens unlike all other tasks that need parsed trees to function. PC is also not dependent on morphological, POS, or LID tags as SMH, RPE, and LIH tasks.
PC simply tags each token with the number of punctuations between that token and the root token in the sentence. We observe a connection between the position of punctuation and phrase boundaries in a sentence which goes in line with previous studies (Li et al., 2010; Spitkovsky et al., 2011). PC roughly groups tokens into phrases that usually constitute sub-trees in a dependency tree.
Figure 3 shows the outputs of these tasks on the dependency tree of an example CS sentence.
3.3 The Gating Procedure
To create the final parser, the trained auxiliary task models are combined with the base parser through a gating mechanism (Sato et al., 2017) which learns to scale between the encoders of the auxiliary sequence labelers and that of the parser (see Fig. 2).
1The selected UD features are Aspect, Case, Foreign, Mood, NumType, Person, and VerbForm.
Formally, the combined representation can be formulated as:
$$
\begin{array}{l} b ^ {t} = \sigma (W ^ {\text {g a t e}} \left(e ^ {\text {p a r s e r}} \oplus e ^ {\text {l a b e l e r}}\right) + w ^ {\text {g a t e}}) \\ \mathbf {g} ^ {t} = b ^ {t} \cdot e ^ {\text {p a r s e r}} + (1 - b ^ {t}) \cdot e ^ {\text {l a b e l e r}} \\ \end{array}
$$
where  $e^{parser}$  and  $e^{labeler}$  are the outputs of the parser and sequence labeler encoders, respectively.  $\oplus$  denotes concatenation.  $W^{gate}$  and  $w^{gate}$  are the learned parameters of the gating procedure and  $\sigma$  is the sigmoid function. The final combined vector  $g^t$  is then given to the biaffine classifiers.
3.4 Transformer-based Adaptation of the Model
Our base parser as described in Dozat and Manning (2017) has some shortcomings in the choice of the input representation, especially when the target language has very little or no training data and there is no accompanying pretrained word embeddings to represent the input. This is also the case with CS language pairs. In that situation, utilizing the expressive power of transformers can be a good solution. Pretrained on huge amounts of raw data in different languages, multilingual transformer-based language models have proven remarkably effective (Devlin et al., 2019; Sanh et al., 2019; Liu et al., 2019). One such model is XLM-R (Connieu et al., 2020). Pretrained on text data in 100 languages, XLM-R shows SOTA performance in many languages including low-resource ones.
To the best of our knowledge, such a deep contextualized semi-supervised scheme has not been incorporated with XLM-R before. So, we re-implement the auxiliary task modules and the combined parsing approach for an XLM-R-based encoding module. For this purpose we follow the XLM-R-based parsing architecture of Grunewald et al. (2021) which has the same biaffine parsing model described in Dozat and Manning (2017). Our aim is to observe how extracting parsing-related knowledge from semi-supervised auxiliary tasks affects a multilingual transformer model.
We perform experiments on all CS treebanks<sup>2</sup> in Universal Dependencies (v2.8).<sup>3</sup> These are Komi-
Zyrian IKDP (Kpv-Ru), Hindi-English HIENCS (Hi-En), Frisian-Dutch Fame (Fy-Nl), and Turkish-German SAGT (Tr-De) treebanks. All except Hi-En are based on spoken CS data. Hi-En is constructed from bilingual tweets. Table 1 states basic statistics and related resources for each treebank.
Due to lack of training data in some CS treebanks, we have two types of experimental setup. We train the parser models on in-domain data for Hi-En and Tr-De. In these experiments we use each treebank's own training set. However, Kpv-Ru and Fy-Nl consist of a test set only. Hence, training of the latter two treebanks are on out-of-domain data. For Kpv-Ru which includes Komi-Russian code-switching, we train the models on Komi-Zyrian Lattice UD Treebank (Partanen et al., 2018a) of monolingual Komi data. The first 562 sentences in Komi-Zyrian Lattice are used for training, the remaining 100 are used for development. For Fy-Nl, our training data is the Dutch Alpino UD Treebank (Van der Beek et al., 2002). We chose Dutch Alpino over the other Dutch UD treebank (LassySmall) as Alpino is found more effective in parsing Fy-Nl (Braggaar and van der Goot, 2021).
4.2.1 Unlabeled Data
Komi-Russian Komi Social Media Corpus<sup>4</sup> is part of a social media corpora project for minority Uralic languages (Arkhangelskiy, 2019). The data is crawled from vkontakte, a social media service mostly popular in Russia. Collected texts are automatically separated to monolingual segments of Komi, Russian, or Unknown via a dictionary-based method. For our purposes, we extract 3,862 CS sentences from the corpus by joining consecutive segments that alternate between Komi and Russian.
Hindi-English We employ the datasets in the LinCE CS benchmark $^{5}$  (Aguilar et al., 2020) for this language pair. The benchmark provides three different corpora with gold LID and POS labels for Hindi-English (Mave et al., 2018; Singh et al., 2018a,b). We combine these three corpora to use them as unlabeled data. The resulting data consists of 10,989 sentences.
2There is also the Maghrebi Arabic-French Treebank (Sed-dah et al., 2020) but its annotations are not yet compatible with the UD scheme and it is not included in the UD repository.
<sup>3</sup>https://lindat.mff.cuni.cz/ repository/xmlui/handle/11234/1-3687. All UD treebanks used in this paper are licensed with CC
BY-SA 4.0.
4Available for research purposes. We obtained the corpus by contacting Timofey Arkhangelskiy at http://komi-zyrian.web-corpora.net/index_en.html
5The datasets are publicly available for research purposes.
Kpv-Ru(Komi-Russian)	Hi-En(Hindi-English)	Fy-NI(Frisian-Dutch)	Tr-De(Turkish-German)
Train	-	1,448	-	578
Dev	-	225	150	801
Test	214	225	250	805
CMI	16.97	36.08	17.80	28.78
Morphology	yes	no	no	yes
Monolingual treebanks	both	both	only Dutch	both
Unlabeled CS data	Komi Social Media	LinCE	FAME!	TuGeBiC
XLM-R	only Russian	both	both	both
FastText	only Russian	both	both	both
Table 1: Some statistics and related resources for the CS treebanks. Fy-Nl is provided as a single test set of 400 utterances. As in Braggaar and van der Goot (2021), we split it into a development set (first 150 utterances) and a test set (remaining 250 utterances). CMI is the code-mixing index (Das and Gambäck, 2014) that shows how frequent code-switching happens in the text.
Frisian-Dutch We extract CS sentences from the FAME! Corpus $^6$  (Yilmaz et al., 2016) which contains radio broadcasts in Frisian-Dutch. From this corpus, which is also the source of the Fy-NI treebank, we select 2,170 sentences that include at least one CS point and are not already in the treebank.
Turkish-German TuGeBiC $^{7}$  (Trefders-Daller and Çetinoğlu, 2022) is a set of transcriptions, collected from interviews with Turkish-German bilinguals in the 90s (Trefders-Daller, 2020). It contains 16,950 sentences. We use the whole corpus, and only remove the speaker IDs and metadata from the files.
4.2.2 Sequence Labeler Training
Training auxiliary models on sequence labeling tasks is done on automatically parsed version of the corresponding unlabeled data for each treebank. Some of the sequence labeling tasks need specific labels on unlabeled data to function. These are POS tags for RPE, LID labels for LIH, and morphological annotation for SMH. In training of these tasks, we use gold labels when available (POS tags for Hi-En; LIDs for Kpv-Ru, Hi-En, and Fy-Nl) and train taggers in the absence of gold labels (POS tags for Kpv-Ru, Fy-Nl, and Tr-De; LIDs for Tr-De; morphological features for Kpv-Ru and Tr-De).
As our baseline, we use Ma et al. (2018)'s reimplementation of the biaffine parser by Dozat and Manning (2017). We call this model BaseLSTM as it uses BiLSTMs for contextualized word vectors.
As a second baseline, we implement the traditional self-training approach (McClosky et al., 2006) in which the parser is first trained only on gold labeled data. Then, labels of unlabeled data are predicted by the trained parser. Finally the parser is re-trained on the combination of gold labeled data and pseudo-labeled data. We name this approach as Self-training.
For our experiments with XLM-R, we use Grunewald et al. (2021)'s implementation of the biaffine parser with XLM-R-based input representation. Input word embeddings are calculated as a weighted sum of all intermediate outputs of the transformer layers. Coefficients of the weighted sum are learned during the training phase. Apart from its multilingual transformer-based contextualized word representation model, it has the same biaffine parsing model in Dozat and Manning (2017). We call this version BaseXLMR.
Hyper-parameters of both parser models and sequence labelers can be found in Appendix A.1.
4.4 Semi-supervised Enhancement Models
We provide the list of enhancement models built on top of BaseLSTM and BaseXLMR where parser is combined with a sequence labeler trained on:
+NOC: Number of Children,
- +DTR: Distance to the Root,
-  $+\mathrm{RPE}$  : Relative POS Encoding,
+LIH: Language ID of Head,
+SMH: Simplified Morphology of Head,8
+PC: Punctuation Count.9
Available via a license agreement. https://www.ru.nl/clst/tools-demos/datasets/  
Available at https://github.com/ozlemcek/TuGeBiC
Note that only Kpv-Ru and Tr-De treebanks have morphological annotation. Hence,  $+\mathrm{SMH}$  is applied only to them. The  $+\mathrm{PC}$  model is not applied to Fy-NI since the treebank does not have punctuation.
Additionally, we perform experiments by ensembling more than one auxiliary task model with the base parser. We experiment with two configurations. First, we integrate Number of Children, Distance to the Root, and Relative POS Encoding models together (+NOC, +DTR, +RPE). This is also the ensemble configuration in Rotman and Reichart (2019). Since we have additional three tasks, we also make the combination of three best performing models for each treebank and name this ensemble version as +Best Combination. For combining encoders of more than one auxiliary task model with the parser encoder, we use Rotman and Reichart (2019)'s extension to the gating mechanism of Sato et al. (2017).
We perform three runs for each model and report the average scores. We measure the performance of all models using the CoNLL 2018 Shared Task evaluation script<sup>11</sup> and report the unlabeled and labeled attachment scores (UAS and LAS, respectively).