SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 2 shows the performance of all LSTM-based models and of the previous works on the test set of each treebank in terms of attachment scores. Significance testing is performed using the approximate randomization test (Noreen, 1989) on the model outputs with the number of shuffles set to 5,000.
Comparison to Baselines On all treebanks, the auxiliary task enhancement methods improve the scores when compared to BaseLSTM by 4.94 points in UAS and 3.86 points in LAS on average. The best performing enhancement model differs across treebanks. We observe the same pattern for the traditional self-training method. Self-training fails to surpass the proposed approach on any of the treebanks. Its parsing performance even falls below that of BaseLSTM on Kpv-Ru and Tr-De. It shows the highest improvement with respect to BaseLSTM on Fy-Nl. Yet, the best one of the auxiliary task enhancement methods significantly outperforms Self-training on each treebank.
New Individual Tasks The  $+\mathrm{L}\mathrm{I}\mathrm{H}$  model which employs LIDs performs best on Kpv-Ru, and second best on Hi-En. Its performance on Tr-De and Fy-Nl is comparable with the other models. It is also in the Best Combination ensemble for
all treebanks. This indicates the importance of language IDs in CS dependency parsing.
The  $+\mathrm{SMH}$  model which is only applied to KpvRu and Tr-De is the best performing one on Tr-De. However, all other tasks outperform  $+\mathrm{SMH}$  on KpvRu. This might be due to the quality difference in morphological taggers trained on these treebanks. The morphological tagger we trained on the CS training set of Tr-De has an accuracy of  $82\%$  on its test set. However, to train a tagger for Kpv-Ru we used monolingual Komi data only. Accuracy of this tagger on Kpv-Ru test set is  $66\%$ . It seems the Kpv-Ru parser suffers from error propagation.
The simplest enhancement model  $+\mathrm{PC}$  performs comparable to others, even outperforming  $+\mathrm{NOC}$  and  $+\mathrm{DTR}$  on Kpv-Ru and Tr-De. Since it only needs the root position in the sentence to perform, this model can be an alternative to other models when gold/predicted POS or morphological tags are hard to acquire. It can also be preferred when the error propagated to the auxiliary tasks from the base parser through predicted trees is high, damaging accuracy of the tasks that rely on these parses.
Individual Tasks vs Ensembles Ensembling multiple tasks improves UAS and LAS on Hi-En and Fy-Nl and LAS on Tr-De when compared with the best performing single task. The +Best Combination ensemble works better on Fy-Nl and Tr-De than the +NOC, +DTR, +RPE ensemble proposed by Rotman and Reichart (2019). Looking at the overall results, we observe that including +RPE and +LIH together has a favorable effect on improving CS parsing performance.
Who Benefits Most and Least? Fy-Nl is the most benefited treebank from the proposed model. The best performing enhancement model +Best Combination on Fy-Nl achieves almost 10/7 points increase in UAS/LAS when compared with BaseLSTM. The least benefited treebank is Kpv-Ru with 2.5/1.1 points increase in UAS/LAS. Having similar amount of unlabeled data and no CS training data, these treebanks differ in their training data amounts. The Dutch Alpino Treebank used to train Fy-Nl models has 13,603 sentences whereas the Komi-Zyrian Lattice Treebank for Kpv-Ru models includes 662 sentences. So, automatic parsing of unlabeled data of Kpv-Ru by a model trained on 662 sentences can be much noisier than that of Fy-Nl. In Appendix A.2, we show that the performance ranking of the systems does not change by the amount of gold training data.
<sup>10</sup>Due to high memory consumption of XLM-R-based models, this ensemble technique cannot be applied to our XLM-R-based parsing architecture. <sup>11</sup>https://universaldependencies.org/conll18/conll18.ud_eval.py
Kpv-Ru	Hi-En	Fy-NI	Tr-De
UAS	LAS	UAS	LAS	UAS	LAS	UAS	LAS
Baselines	\(Base_{LSTM}\)	62.24	45.10	80.10	71.29	64.97	49.56	67.50	57.88
Self-training	59.55	43.27	80.47	72.88	68.91	53.24	60.86	52.04
Semi-supervised Enhancement	+NOC	64.83*	46.53*	81.67	72.94	71.80*	53.35	70.86*	60.97*
+DTR	64.80*	45.53	81.94	72.96	71.48*	53.10	70.88*	60.63*
+RPE	64.95*	45.90	82.75*	73.84	72.98*	54.12	71.40*	61.46*
+LIH	65.70*	47.13*	82.24*	73.54	72.20*	51.98	71.39*	61.46*
+SMH	64.63*	45.31	-	-	-	-	71.41*	61.50*
+PC	64.67*	46.79*	81.40	72.76	-	-	71.25*	61.44*
Ensemble	+NOC, +DTR, +RPE	65.59*	46.86*	82.75*	74.09*	73.97*	56.10*	70.55*	60.95*
+Best Combination†	64.98*	46.22*	82.77*	74.02*	74.69*	56.39*	70.92*	61.65*
Previous Work	Bhat et al. (2018)	-	-	80.23	71.03	-	-	-	-
Braggaar and van der Goot (2021)	-	-	-	-	70.20	55.60	-	-
van der Goot et al. (2021)	-	22.20	-	65.50	-	54.00	-	60.90
Müller-Eberstein et al. (2021)	-	-	73.62	62.66	-	-	66.75	55.04
Table 2: Attachment scores of baselines, our models, and the previous works on all CS UD treebanks. +SMH is not applicable to Hi-En and Fy-Nl due to the lack of morphology in these treebanks. +PC cannot be applied to Fy-Nl since it has no punctuation. †Best combination for each treebank: +NOC, +LIH, +PC for Kpv-Ru, +DTR, +RPE, +LIH for Hi-En, +NOC, +RPE, +LIH for Fy-Nl, and +RPE, +LIH, +SMH for Tr-De. The best scores for each dataset are underlined and bold. Scores marked with * significantly outperform both BaseLSTM and Self-training.
Kpv-Ru	Hi-En	Fy-N1	Tr-De
UAS	LAS	UAS	LAS	UAS	LAS	UAS	LAS
BaseXLMR	57.90	43.12	81.42	71.54	65.75	50.27	75.93	66.30
+NOC	57.09	42.79	81.28	71.58	67.50*	51.64*	75.79	65.98
+DTR	56.65	42.37	82.15*	71.89	66.85*	50.45	75.56	65.73
+RPE	58.77*	43.84	81.79	71.84	67.35*	51.13*	75.49	65.77
+LIH	57.24	43.19	81.92	71.93	66.26	50.10	75.51	65.78
+SMH	56.98	43.25	-	-	-	-	75.53	65.66
+PC	56.81	41.97	81.46	71.89	-	-	75.14	65.45
Table 3: Performance of XLM-R-based parser and our XLM-R adaptation of auxiliary task enhancement models. The best scores for each dataset are underlined and bold. Scores marked with * significantly outperform  $Base_{XLMR}$ .
Comparison to Previous Work The best enhancement model always achieves better scores than previous state-of-the-art on each treebank. In this respect, the biggest improvement is observed on Kpv-Ru with more than 24 points increase in LAS. In addition, it should be noted that model architectures are not quite comparable as some of the previous work use a lot more resources than our models. For instance, Müller-Eberstein et al. (2021) perform data selection on whole UD datasets for training and utilize multilingual BERT.
Proposed Method and XLM-R Attachment scores of BaseXLMR and our XLM-R adaptation of auxiliary task enhancement models are given in Table 3. Our first observation is the limited performance of BaseXLMR in parsing CS treebanks. We see that the enhancement models do not have the same impact on BaseXLMR as they have on BaseLSTM. The only significant performance increase is on Fy-Nl where the best performing en
hancement model +NOC outperforms BaseXLMR by almost 2/1.5 points in UAS/LAS. For Kpv-Ru, the only model that surpasses the baseline is +RPE. The difference is found statistically significant only in UAS. For Hi-En, all enhancement models except +NOC perform better than BaseXLMR. Yet, the only significant improvement is achieved by +DTR in UAS. None of the enhancement models surpass BaseXLMR on Tr-De but the difference between the scores is not found to be significant. Another remarkable observation is our models built on top of BaseLSTM outperforming all XLM-R-based models with the exception of Tr-De. This answers our second research question: XLM-R is not always the best option. For powerful models like XLM-R, multilinguality can harm the performance when the target language is unknown to the model. Our results suggest that in such cases it is better to employ simpler models that are tailored for the exact task.
Comparison of the Proposed Approach with Baselines in terms of Computational Resources
Table 4 provides time and memory usage of BaseLSTM, BaseXLMR, and our proposed best model for each treebank. Labeled attachment scores (LAS) acquired by these models on each treebank are also given.
Kpv-Ru	Hi-En	Fy-NI	Tr-De
Training time
BaseLSTM	0h9m	0h15m	0h45m	0h20m
Our best model	0h25m	0h40m	2h30m	0h55m
BaseXLMR	3h40m	3h15m	11h0m	1h30m
Memory usage (GB)
BaseLSTM	3.6	3.6	3.8	3.5
Our best model	4.5	7.6	7.3	7.4
BaseXLMR	9.9	7.9	9.6	8.4
LAS
BaseLSTM	45.10	71.29	49.56	57.88
Our best model	47.13	74.09	56.39	61.65
BaseXLMR	43.12	71.54	50.27	66.30
Table 4: Comparison of baselines and the proposed approach according to training time, memory usage during training, and LAS. Our best model on Kpv-Ru is the  $+\mathrm{L}\mathrm{I}\mathrm{H}$  model. For all other treebanks, our best model is an ensemble that combines three task models.
From the table, we observe that there is a tradeoff between performance and resource consumption for the three models. The training time of the BaseLSTM model is the shortest. Yet, our best model improves the performance significantly at the expense of a slight increase in training time. BaseXLMR has the longest training time by a large margin.
In terms of memory usage, there is a similar pattern to that of training time.  $\mathrm{Base}_{\mathrm{LSTM}}$  needs approximately  $50\%$  less memory than our best model, yet there is on average 3.86 points gap between LAS of the two models.  $\mathrm{Base}_{\mathrm{XLMR}}$  is again the least preferable model here due to its highest memory consumption and low performance on parsing the treebanks with the exception of Tr-De. Only for Tr-De it outperforms the other two models and can be the model of choice for the parsing of Tr-De data.
Considering the long training time and high resource consumption of the XLM-R-based parser and the success of our LSTM-based enhancement models, we suggest LSTM-based auxiliary task enhancement for low-resource dependency parsing of CS data.
In this paper, we focus on CS dependency parsing. We present a semi-supervised auxiliary task enhancement to a graph-based neural parser and create novel sequence labeling tasks that are shown as useful in improving the parser's success. Experimental results show that our enhancement technique achieves SOTA performance on all CS UD treebanks and helps better utilization of unlabeled data for CS dependency parsing. We combine our enhancement models with XLM-R to see their performance on a multilingual transformer-based model. Results demonstrate that the powerful XLM-R shows limited performance and fails to surpass our semi-supervised auxiliary task enhancement models. Our implementation of the proposed sequence labeling tasks and the XLM-R-based enhancement are publicly available for research purposes at https://github.com/sb-b/ss-cs-depparser.
This work is funded by DFG via project CE 326/1-1 "Computational Structural Analysis of German-Turkish Code-Switching (SAGT)". The numerical calculations reported in this paper were performed at the computing servers of IMS at University of Stuttgart and at TUBITAK ULAKBIM (TRUBA resources). GEBIP Award of the Turkish Academy of Sciences (to A.Ö.) is gratefully acknowledged. We thank the anonymous reviewers for their valuable comments and feedbacks.
The UD datasets we use in this study are all from published works and licensed with CC BY-SA 4.0. All the used unlabeled corpora also have accompanying papers and were made publicly available by their providers. To utilize TuGeBiC in our experiments, we removed speaker IDs that were already anonymized.
Code-switching is very understudied computationally and by employing datasets from multiple language pairs, including minority languages, we minimize the potential risk of exposure. Our study asks foundational research questions on code-switching datasets. Our experiments show that even within our datasets results vary, hence we believe the paper avoids possible overgeneralizations.
XLM-R-based models employed in this study have higher memory consumption than the LSTM-
based models (see Table 4). They also require longer training time which means higher energy consumption. To reduce this negative side-effect, we avoid performing the same experiment many times and repeated each experiment only three times. Moreover, we chose not to utilize XLM-R-large model with 550M parameters but instead used XLM-R-base model with 270M parameters to reduce the training time and memory consumption. All the experiments with XLM-R models in this study were run on a single NVIDIA RTX A6000 GPU.
Gustavo Aguilar, Sudipta Kar, and Thamar Solorio. 2020. LinCE: A centralized benchmark for linguistic code-switching evaluation. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 1803-1813, Marseille, France. European Language Resources Association.
Gustavo Aguilar and Thamar Solorio. 2020. From English to code-switching: Transfer learning with strong morphological clues. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8033-8044, Online. Association for Computational Linguistics.
Timofey Arkhangelskiy. 2019. Corpora of social media in minority Uralic languages. In Proceedings of the Fifth International Workshop on Computational Linguistics for Uralic Languages, pages 125-140, Tartu, Estonia. Association for Computational Linguistics.
Peter Auer and Li Wei. 2007. Handbook of multilingualism and multilingual communication, volume 5. Walter de Gruyter.
Irshad Bhat, Riyadh A. Bhat, Manish Shrivastava, and Dipti Sharma. 2017. Joining hands: Exploiting monolingual treebanks for parsing of code-mixing data. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 324-330, Valencia, Spain. Association for Computational Linguistics.
Irshad Bhat, Riyadh A. Bhat, Manish Shrivastava, and Dipti Sharma. 2018. Universal Dependency parsing for Hindi-English code-switching. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 987–998, New Orleans, Louisiana. Association for Computational Linguistics.
David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. J. Mach. Learn. Res., 3:993-1022.
Anouck Braggaar and Rob van der Goot. 2021. Challenges in annotating and parsing spoken, codeswitched, Frisian-Dutch data. In Proceedings of the Second Workshop on Domain Adaptation for NLP, pages 50-58, Kyiv, Ukraine. Association for Computational Linguistics.
Özlem Çetinoglu and Çağrı Çoltekin. 2019. Challenges of annotating a code-switching treebank. In Proceedings of the 18th International Workshop on Treebanks and Linguistic Theories (TLT, SyntaxFest 2019), pages 82-90, Paris, France. Association for Computational Linguistics.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishray Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle-moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451, Online. Association for Computational Linguistics.
Amitava Das and Björn Gambäck. 2014. Identifying languages at the word level in code-mixed Indian social media text. In Proceedings of the 11th International Conference on Natural Language Processing, pages 378-387, Goa, India. NLP Association of India.
Mathieu Dehouck and Pascal Denis. 2018. A framework for understanding the role of morphology in Universal Dependency parsing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2864-2870, Brussels, Belgium. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Stefan Grünewald, Annemarie Friedrich, and Jonas Kuhn. 2021. Applying Occam's razor to transformer-based dependency parsing: What works, what
doesn't, and what is really necessary. In Proceedings of the 17th International Conference on Parsing Technologies and the IWPT 2021 Shared Task on Parsing into Enhanced Universal Dependencies (IWPT 2021), pages 131-144, Online. Association for Computational Linguistics.
Anupam Jamatia, Björn Gambäck, and Amitava Das. 2015. Part-of-speech tagging for code-mixed English-Hindi Twitter and Facebook chat messages. In Proceedings of the International Conference Recent Advances in Natural Language Processing, pages 239-248, Hissar, Bulgaria. INCOMA Ltd. Shoumen, BULGARIA.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Zhenghua Li, Wanxiang Che, and Ting Liu. 2010. Improving dependency parsing using punctuation. In 2010 International Conference on Asian Language Processing, pages 53-56.
Kyung Tae Lim and Thierry Poibeau. 2017. A system for multilingual dependency parsing based on bidirectional LSTM feature representations. In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 63-70, Vancouver, Canada. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Xuezhe Ma, Zeong Hu, Jingzhou Liu, Nanyun Peng, Graham Neubig, and Eduard Hovy. 2018. Stackpointer networks for dependency parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1403-1414, Melbourne, Australia. Association for Computational Linguistics.
Deepthi Mave, Suraj Maharjan, and Thamar Solorio. 2018. Language identification and analysis of codeswitched social media text. In Proceedings of the Third Workshop on Computational Approaches to Linguistic Code-Switching, pages 51-61, Melbourne, Australia. Association for Computational Linguistics.
David McClosky, Eugene Charniak, and Mark Johnson. 2006. Effective self-training for parsing. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, pages 152-159,
New York City, USA. Association for Computational Linguistics.
Max Müller-Eberstein, Rob van der Goot, and Barbara Plank. 2021. Genre as weak supervision for cross-lingual dependency parsing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4786-4802, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal Dependencies v1: A multilingual treebank collection. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 1659-1666, Portoroz, Slovenia. European Language Resources Association (ELRA).
Eric W Noreen. 1989. Computer-intensive methods for testing hypotheses. Wiley New York.
Şaziye Betül Özates and Özlem Çetinoğlu. 2021. A language-aware approach to code-switched morphological tagging. In Proceedings of the Fifth Workshop on Computational Approaches to Linguistic Code-Switching, pages 72-83, Online. Association for Computational Linguistics.
Niko Partanen, Rogier Blokland, KyungTae Lim, Thierry Poibeaun, and Michael RieBler. 2018a. The first Komi-Zyrian Universal Dependencies treebanks. In Proceedings of the Second Workshop on Universal Dependencies (UDW 2018), pages 126-132, Brussels, Belgium. Association for Computational Linguistics.
Niko Partanen, Kyungtae Lim, Michael Rießler, and Thierry Poibeau. 2018b. Dependency parsing of code-switching data with cross-lingual feature representations. In Proceedings of the Fourth International Workshop on Computational Linguistics of Uralic Languages, pages 1-17, Helsinki, Finland. Association for Computational Linguistics.
Guy Rotman and Roi Reichart. 2019. Deep contextualized self-training for low resource dependency parsing. Transactions of the Association for Computational Linguistics, 7(0):695-713.
Piotr Rybak and Alina Wróblewska. 2018. Semi-supervised neural system for tagging, parsing and lematization. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 45-54, Brussels, Belgium. Association for Computational Linguistics.
Jivnesh Sandhan, Amrith Krishna, Ashim Gupta, Laxmidhar Behera, and Pawan Goyal. 2021. A little pretraining goes a long way: A case study on dependency parsing task for low-resource morphologically rich languages. In Proceedings of the 16th
Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 111-120, Online. Association for Computational Linguistics.
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108.
Motoki Sato, Hitoshi Manabe, Hiroshi Noji, and Yuji Matsumoto. 2017. Adversarial training for cross-domain Universal Dependency parsing. In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 71-79, Vancouver, Canada. Association for Computational Linguistics.
Djame Seddah, Farah Essaidi, Amal Fethi, Matthieu Futeral, Benjamin Muller, Pedro Javier Ortiz Suárez, Benoit Sagot, and Abhishek Srivastava. 2020. Building a user-generated content North-African Arabizi treebank: Tackling hell. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1139-1150, Online. Association for Computational Linguistics.
Kushagra Singh, Indira Sen, and Ponnurangam Kumaraguru. 2018a. Language identification and named entity recognition in Hinglish code mixed tweets. In Proceedings of ACL 2018, Student Research Workshop, pages 52-58, Melbourne, Australia. Association for Computational Linguistics.
Kushagra Singh, Indira Sen, and Ponnurangam Kumaraguru. 2018b. A Twitter corpus for Hindi-English code mixed POS tagging. In Proceedings of the Sixth International Workshop on Natural Language Processing for Social Media, pages 12-17, Melbourne, Australia. Association for Computational Linguistics.
Valentin I. Spitkovsky, Hiyan Alshawi, and Daniel Jurafsky. 2011. Punctuation: Making a point in unsupervised dependency parsing. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning, pages 19-28, Portland, Oregon, USA. Association for Computational Linguistics.
Milan Straka and Jana Straková. 2017. Tokenizing, POS tagging, lemmatizing and parsing UD 2.0 with UDPipe. In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 88-99, Vancouver, Canada. Association for Computational Linguistics.