SlowGuess/ACL_2020_OCR huggingface dataset:
<sup>11</sup>Note that we performed a set of baseline experiments with UDPipe 2.0 (Straka and Straková, 2017) as well on a previous version of this data set. It reached only 73.7 of UPOS on the test set.
12https://github.com/VowpalWabbit/vowpal_wabbit/wiki
Without this endogenous lexicon extraction step, the tagger performed slightly worse, although the difference is small.
French GSD UD treebank and our Arabizi version of the Prague Arabic Dependency Treebank. We trained the UDPipe parser (Straka and Straková, 2017) on various treebanks obtained by combining different proportions of the French GSD and our PADT-based pseudo-Arabizi treebank. We ran these parsers with already annotated gold parts-of-speech. The best scores were obtained with a model trained on a mix  $30\%$  of pseudo-Arabizi and  $70\%$  of French, which we call the MIX treebank, totaling 5,955 training sentences. We split this treebank into training, development and test sets, called  $\mathrm{MIX}_{\mathrm{train / dev / test}}$ , following a  $80 / 10 / 10$  split. We used a very small manually annotated NArabizi development dataset of 200 NArabizi sentences, called Arabizi $_{\mathrm{dev}}$ , to evaluate our parser. As shown in Table 6 (line “Mix”), despite good results on MIX's development and training sets,  $\mathrm{MIX}_{\mathrm{dev}}$  and  $\mathrm{MIX}_{\mathrm{test}}$  respectively (see Table 6), this first parser did not performed very well when evaluated on Arabizi $_{\mathrm{dev}}$ . This performance level proved insufficient to speed up the annotation task. We therefore manually annotated 300 more NArabizi sentences (Arabizi $_{\mathrm{train300}}$ ), to be used as additional training data. When added to  $\mathrm{MIX}_{\mathrm{train}}$ , parsing performance did improve, yet not to a sufficient extent, especially in terms of Labeled Attachment Score (LAS). It turned out that training UDPipe on these 300 manually annotated NArabizi sentences only (Arabizi $_{\mathrm{train300}}$ ) produced better scores, resulting in a parser that we did use as a pre-annotation tool in a constant bootstrap process to speed up the annotation of the remaining sentences.
Training corpus	Dev	Test	Test Arabizi
LAS	UAS	LAS	UAS	LAS	UAS
MIX	87.67	89.42	87.69	89.44	39.28	51.52
MIX+Arabizi 300	87.42	89.20	87.44	89.22	55.54	65.36
Arabizi 300	39.11	49.62	39.14	49.65	63.03	71.21
Table 6: Results of UDPipe (trained 100 epochs) on the preliminary test set.
How interleaved are French and NArabizi? As stated before, NArabizi takes its root in Classical Arabic and in multiple sources of integration of French, MSA and Berber, the Amazigh language. As the NArabizi treebank contains more than  $36\%$  of French words, it is of interest to use recent methods of visualization to see how interleaved it is
(a) dim: 50
Figure 2: Two-dimensional representation of Fasttext word embeddings for 300 words (100 for transliterated MSA - blue -, French - yellow - and NArabizi - red -) after PCA analysis
(b) dim: 100
with some of its source languages. To this end, we extract words embeddings using fastText (Joulin et al., 2016) from a corpus made of the "transliterated" PADT described in Section 5, the French UD GSD and NArabizi original corpus (Cotterell et al., 2014). Two-dimensional representations of the resulting embeddings space for 300 selected words are shown in Figure 2 for embeddings of size 50 and 100.
We notice that the overall shapes of both representations are very similar, apart from a non significant  $x$ -axis reversal. On the first components, increasing the embedding size does not provide more information.
We also see that French and transliterated Arabic words are clearly separated into two clusters of low standard deviation, while NArabizi words are very spread out. Some fall within the French cluster, they correspond to French words present in this Algerian dialect. Others are in the middle of the Arabic cluster, these are the purely Arabic words of the dialect. Between the two, there are Amazigh words (rak, mech), arabized French words (tomobile  $<$  French automobile), Arabic words whose Berber pronunciation has resulted in an unexpected NArabizi rendering (nta instead of expected enta 'you', mchit instead of expected machayt 'to go-2SING').
What is the Impact of Code-Switching in POS-tagging performance? Given the large degree of interleaving between French and NArabizi, it is interesting to assess the impact of the French vocabulary on the performance of a POS-tags trained on French data only. For these experiments, we use the StanfordNLP neural tagger (Qi et al., 2019), which ranked 1st in POS tagging at the 2018 UD shared task, trained on the UD French
ParTUT treebank, using French fastText vectors (Mikolov et al., 2018). In order to perform a meaningful evaluation, we split the NArabizi training set into 4 buckets of approximately  $25\%$  of its size in tokens, with a increasing proportion of identified NArabizi tokens. Results in Table 7 show a clear drop of performance between the sentences that contain more code-switching  $(59.55\%)$  of UPOS accuracy) and those with none  $(16.84\%)$ . This suggests that low-resource languages with a high-level of code-switching such as NArabizi can benefit from NLP models trained on the secondary language. The level of performance to expect from these cross-language approaches is yet to be determined.
% of NArabizi per sent. bucket set size (sent.)	<60	60-78	78-100	100
322	286	283	276
StanfordNLP (French)	59.55	35.93	25.41	16.84
8 Treebanking Costs
Following Martínez Alonso et al. (2016), we provide here the cost figures of this annotation campaign. We do not include the salaries of the permanent staff, nor do we include the overhead. These figures are meant as an indication of the effort needed to create an annotated data set from scratch. It shall be noted that even though the interannotator agreement gave us early indications on the difficulty of the tasks, it also acted as a metric in terms of language variability among annotators. None of them come from the same part of North-Africa and none of them has the same familiarity with the topics discussed in the web-forums we annotated. We had to constantly re-announce sentences and update the guidelines every time new idiosyncrasies were encountered and most importantly accepted as such by the annotators. Compared to what was reported in (Martínez Alonso et al., 2016), the figures are here much higher (about 5 times higher), because unlike their work on French treebanks, we could not use preexisting guidelines for this language and because we could not keep the same team all along the project, so that new members had to be trained almost from scratch or to work on totally different layers.
Table 7: POS tagging Performance with regard to codemix proportion trained on UD French Partut treebank
Phase	1st	2nd	3rd	4th	5th	p.m	Costs (k€)
Annotators	8	2	2	3	15	45
Jr Researcher	2	5	7	21
Confirmed	6	6	21
total	8	4	7	3	6	28	87
Table 8: Treebanking costs. The annotation phases are (i) Morphology-tokenization, (ii) Translation, (iii) Pre-annotation Syntax, (iv) Correction, (v) Final Syntax. P.M stands for person.month
Research on Arabic dialects is quite extensive. Space is lacking to describe it exhaustively. In relation to our work regarding North-African dialect, we refer to the work of (Samih, 2017) who along his PhD covered an large range of topics regarding the dialect spoken specifically in Morocco and generally regarding language identification (Samih et al., 2016) in code-switching scenario for various Arabic dialects (Attia et al., 2019).
Unlike NArabizi dialects, the resource situation for Arabic dialects in canonical written form can hardly be qualified as scarce given the amount of resources produced by the Linguistic Data Consortium regarding these languages, see (Diab et al., 2013) for details on those corpora. These data have been extensively covered in various NLP aspects by the former members of the Columbia Arabic NLP team, among which Mona Diab, Nizar Habash, and Owen Rambow, in their respective subsequent lines of works. Many small to medium scale linguistics resources, such as morphological lexicons or bilingual dictionaries have been produced (Shoufan and Alameri, 2015). Recently, in addition to the release of a small-range parallel corpus for some Arabic dialects (Bouamor et al., 2014), a larger corpus collection was released, covering 25 city dialects in the travel domain (Bouamor et al., 2018).
Regarding the specific NLP modeling challenges of processing Arabic-based languages, as part of the morphologically-rich languages, recent advances in joint models have been addressed by Zalmout and Habash (2019) that recently efficiently adapted a neural architecture to perform joint word segmentation, lemmatization, morphological analysis and POS tagging on an Arabic dialect. Recent works on cross-language learning using the whole massively multilingual pre-trained language models artillery have started to emerge
(Srivastava et al., 2019). If successful, such models could help to alleviate the resource scarcity issue that plagues low-resources languages in the more-than-ever data hungry modern NLP.
We introduced the first treebank for an Arabic dialect spoken in North-Africa and written in Romanized form, NArabizi. More over, being made of user-generated content, this treebank covers a large variety of language variation among native speakers and displays a high level of code-switching. Annotated with 4 standard morphosyntactic layers, two of them following the Universal Dependency annotation scheme, and provided with translation to French as well as glosses and word language identification, we believe that this corpus will be useful for the community at large, both for linguistic purposes and as training data for resource-scarce NLP in a high-variability scenario. In addition to the annotated data, we provide around 1 million tokens (over 46k sentences) of unlabeled NArabizi content, resulting in the largest dataset available for this dialect. Our corpora are freely available<sup>14</sup> under the CC-BY-SA license and the NArabizi treebank is also released as part of the Universal Dependencies project.
The work was partially funded by the French Research Agency projects ParSiTi (ANR-16-CE33-0021), SoSweet (ANR15-CE38-0011-01) and by the French Ministry of Industry and Ministry of Foreign Affairs via the PHC Maimonide France-Israel cooperation programme, as well as by the Sagot's chair in the PRAIRIE institute funded by the French national agency ANR as part of the "Investissements d'avvenir" programme under the reference ANR-19-P3IA-0001.
Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. Farasa: A fast and furious segmenter for arabic. In HLT-NAACL Demos.
Mohammed Attia, Younes Samih, Ali Elkahky, Hamdy Mubarak, Ahmed Abdelali, and Kareem Darwish. 2019. POS tagging for improving code-switching identification in Arabic. In
Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 18-29, Florence, Italy. Association for Computational Linguistics.
Yonatan Belinkov and James Glass. 2015. Arabic diacritization with recurrent neural networks. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2281-2285, Lisbonne, Portugal. Association for Computational Linguistics.
Houda Bouamor, Nizar Habash, and Kemal Oflazer. 2014. A multidialectal parallel corpus of Arabic. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), pages 1240-1245, Reykjavik, Iceland. European Language Resources Association (ELRA).
Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer. 2018. The MADAR Arabic dialect corpus and lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Ryan Cotterell, Adithya Renduchintala, Naomi Saphra, and Chris Callison-Burch. 2014. An Algerian Arabic-French code-switched corpus. In Workshop on Free/Open-Source Arabic Corpora and Corpora Processing Tools Workshop Programme, page 34.
Mona Diab, Nizar Habash, Owen Rambow, and Ryan Roth. 2013. LDC Arabic treebanks and associated corpora: Data divisions manual. Technical Report CCLS-13-02, Center for Computational Learning Systems, Columbia University.
Nizar Habash. 2010. Introduction to Arabic Natural Language Processing. Morgan and Claypool.
Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Herve Jegou, and Tomas Mikolov. 2016. Fasttext. zip: Compressing text classification models. arXiv preprint arXiv:1612.03651.
<sup>14</sup>http://almanach-treebanks.fr/NArabizi
Éric de La Clergerie, Benoît Sagot, and Djamé Sed-dah. 2017. The ParisNLP entry at the ConLL UD shared task 2017: A tale of a #Parsing-Tragedy. In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 243-252, Vancouver, Canada. Association for Computational Linguistics.
Marco Lui and Timothy Baldwin. 2012. langid.py: An off-the-shelf language identification tool. In *The 50th Annual Meeting of the Association for Computational Linguistics*, Proceedings of the System Demonstrations, July 10, 2012, Jeju Island, Korea, pages 25-30. The Association for Computer Linguistics.
Teresa Lynn and Kevin Scannell. 2019. Code-switching in irish tweets: A preliminary analysis. In Proceedings of the Celtic Language Technology Workshop, pages 32-40.
Héctor Martínez Alonso, Djamé Seddah, and Benoit Sagot. 2016. From Noisy Questions to Minecraft Texts: Annotation Challenges in Extreme Syntax Scenarios. In The 2nd Workshop on Noisy User-generated Text (W-NUT).
Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in pre-training distributed word representations. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018).
Robert Munro. 2010. Crowdsourced translation for emergency response in haiti: the global collaboration of local knowledge. In AMTA Workshop on Collaborative Crowdsourcing for Translation, pages 1-4.
Carol Myers-Scotton. 1993. Common and uncommon ground: Social and structural factors in codeswitching. Language in Society, 22(4):475-503.
Joakim Nivre, Mitchell Abrams, Željko Agić, Lars Ahrenberg, Lene Antonsen, Maria Jesus Aranzabe, Gashaw Arutie, Masayuki Asahara, Luma Ateyah, Mohammed Attia, Aitziber Atutxa, Liesbeth Augustinus, Elena Badmaeva, Miguel Ballesteros, Esha Banerjee, Sebastian Bank, Verginica Barbu Mititelu, John Bauer, Sandra Bellato, Kepa Bengoetxea, Riyaz Ahmad Bhat,
Erica Biaggi, Eckhard Bick, Rogier Blokland, Victoria Bobicev, Carl Borstell, Cristina Bosco, Gosse Bouma, Sam Bowman, Adriane Boyd, Aljoscha Burchardt, Marie Candito, Bernard Caron, Gauthier Caron, Gulsen Cebiroglu Eryigit, Giuseppe G. A. Celano, Savas Cetin, Fabricio Chalub, Jinho Choi, Yongseok Cho, Jayeol Chun, Silvie Cinkova, Aurélie Collomb, Căgni Žoltekin, Miriam Connor, Marine Courtin, Elizabeth Davidson, Marie-Catherine de Marneffe, Valeria de Paiva, Arantza Diaz de Ilarraza, Carly Dickerson, Peter Dirix, Kaja Dobrovoljc, Timothy Dozat, Kira Droganova, Puneet Dwivedi, Marhaba Eli, Ali Elkahky, Binyam Ephrem, Tomaz Erjavec, Aline Etienne, Richard Farkas, Hector Fernandez Alcalde, Jennifer Foster, Cláudia Freitas, Katarina Gajdosová, Daniel Galbraith, Marcos Garcia, Moa Gartenfors, Kim Gerdes, Filip Ginter, lakes Goenaga, Koldo Gojenola, Memduh Gökirmak, Yoav Goldberg, Xavier Gómez Guinovart, Berta Gonzáles Saavedra, Matias Grioni, Normunds Gruzītis, Bruno Guillaume, Céline Guillot-Barbance, Nizar Habash, Jan Hajic, Jan Hajic jr., Linh Hà My, Na-Rae Han, Kim Harris, Dag Haug, Barbora Hladka, Jaroslava Hlaváčová, Florinel Hociung, Petter Hohle, Jena Hwang, Radu Ion, Elena Irimia, Tomás Jelinek, Anders Johannsen, Fredrik Jorgensen, Hüner Kaşikara, Sylvain Kahane, Hiroshi Kanayama, Jenna Kanerva, Tolga Kayadelen, Václava Kettnerová, Jesse Kirchner, Natalia Kotsyba, Simon Krek, Sookyoung Kwak, Veronika Laippala, Lorenzo Lambertino, Tatiana Lando, Septina Dian Larasati, Alexei Lavrentiev, John Lee, Phuong Lê Hong, Alessandro Lenci, Saran Lertpradit, Herman Leung, Cheuk Ying Li, Josie Li, Keying Li, KyungTae Lim, Nikola Ljubesić Olga Loginova, Olga Lyashevskaya, Teresa Lynn, Vivien Macketanz, Aibek Makazhanov, Michael Mandl, Christopher Manning, Ruli Manurung, Catalina Máranduc, David Mareček, Katrin Marheinecke, Héctor Martínez Alonso Andre Martins, Jan Mašek, Yuji Matsumoto Ryan McDonald Gustavo Mendonça Niko Miekka, Anna Missilä,Catalin Mititelu,Yusuke Miyao,Simonetta Montemagni,Amir More Laura Moreno Romero, Shinsuke Mori,Bjartur Mortensen Bohdan Moskalevskyi Kadri Muischnek Yugo Murawaki,Kaili Mürirsep Pinkey Nainwani Juan Ignacio Navarro Horniacek
Anna Nedoluzhko, Gunta Nespore-Berzkalne, Luong Nguyen Thi, Huyen Nguyen Thi Minh, Vitaly Nikolaev, Rattima Nitisaroj, Hanna Nurmi, Stina Ojala, Robert Oluokun, Adedayoostling, Lilja Øvrelid, Niko Partanen, Elena Pascual, Marco Passarotti, Agnieszka Patejuk, Siyao Peng, Cenel-Augusto Perez, Guy Perrier, Slav Petrov, Jussi Piitulainen, Emily Pitler, Barbara Plank, Thierry Poibeaumartin Popel, Lauma Pretkalinna, Sophie Prevost, Prokopis Prokopidis, Adam Przepiorkowski, Tiina Puolakainen, Sampo Pyysalo, Andriela Raabis, Alexandre Rademaker, Loganathan Ramasamy, Taraka Rama, Carlos Ramisch, Vinit Ravishankar, Livy Real, Siva Reddy, Georg Rehm, Michael Rießler, Larissa Rinaldi, Laura Rituma, Luisa Rocha, Mykhailo Romanenko, Rudolf Rosa, Davide Rovati, Valentin Rosca, Olga Rudina, Shoval Sadde, Shadi Saleh, Tanja Samardžić, Stephanie Samson, Manuela Sanguinetti, Baiba Saulite, Yanin Sawanakunanon, Nathan Schneider, Sebastian Schuster, Djame Seddah, Wolfgang Seeker, Mojgan Seraji, Mo Shen, Atsuko Shimada, Muh Shohibussirri, Dmitry Sichinava, Natalia Silveira, Maria Simi, Radu Simionescu, Katalin Simko, María Šimkova, Kiril Simov, Aaron Smith, Isabela Soares-Bastos, Antonio Stella, Milan Straka, Jana Strnadová, Alane Suhr, Umut Sulubacak, Zsolt Szántó, Dima Taji, Yuta Takahashi, Takaaki Tanaka, Isabelle Tellier, Trond Trosterud, Anna Trukhina, Reut Tsarfaty, Francis Tyers, Sumire Uematsu, Zdenka Uresová, Larraitz Uria, Hans Uszkoreit, Sowmya Vajjala, Daniel van Niekerk, Gertjan van Noord, Viktor Varga, Veronika Vincze, Lars Wallin, Jonathan North Washington, Seyi Williams, Mats Wiren, Tsegay Woldemariam, Tak-sum Wong, Chunxiao Yan, Marat M. Yavrumyan, Zhuoran Yu, Zdeněk Žabokrtský, Amir Zeldes, Daniel Zeman, Manying Zhang, and Hanzhi Zhu. 2018. Universal dependencies 2.2. LINDAT/CLARIN digital library at the Institute of Formal and Applied Linguistics (UFAL), Faculty of Mathematics and Physics, Charles University.
Adam Nossiter. 2019. Algeria protests grow against president bouteflika, ailing and out of sight. In New York Times (March 01, 2019).
Pedro Javier Ortiz Suárez, Benoit Sagot, and Lau
rent Romary. 2019. Asynchronous Pipeline for Processing Huge Corpora on Medium to Low Resource Infrastructures. In 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7), Cardiff, United Kingdom. Leibniz-Institut für Deutsche Sprache.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318.
Slav Petrov, Dipanjan Das, and Ryan McDonald. 2011. A universal part-of-speech tagset. arXiv preprint arXiv:1104.2086.
Peng Qi, Timothy Dozat, Yuhao Zhang, and Christopher D Manning. 2019. Universal dependency parsing from scratch. arXiv preprint arXiv:1901.10457.
Houda Saadane and Nizar Habash. 2015. A conventional orthography for Algerian arabic. In Proceedings of the Second Workshop on Arabic Natural Language Processing, pages 69-79.
Younes Samih. 2017. *Dialectal Arabic processing Using Deep Learning*. Ph.D. thesis, Düsseldorf, Germany.
Younes Samih, Suraj Maharjan, Mohammed Attia, Laura Kallmeyer, and Thamar Solorio. 2016. Multilingual code-switching identification via LSTM recurrent neural networks. In Proceedings of the Second Workshop on Computational Approaches to Code Switching, pages 50-59, Austin, Texas. Association for Computational Linguistics.
Djamé Seddah, Benoit Sagot, Marie Candito, Virginie Mouilleron, and Vanessa Combet. 2012. The French Social Media Bank: a Treebank of Noisy User Generated Content. In CoLing, Mumbai, India.
Abdulhadi Shoufan and Sumaya Alameri. 2015. Natural language processing for dialectical Arabic: A survey. In Proceedings of the Second Workshop on Arabic Natural Language Processing, pages 36-48, Beijing, China. Association for Computational Linguistics.
Abhishek Srivastava, Benjamin Muller, and Djamé Seddah. 2019. Unsupervised Learning for Handling Code-Mixed Data: A Case Study on POS Tagging of North-African Arabizi Dialect. EurNLP - First annual EurNLP. Poster.
Milan Straka and Jana Straková. 2017. Tokenizing, pos tagging, lemmatizing and parsing ud 2.0 with upipe. In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 88-99, Vancouver, Canada. Association for Computational Linguistics.
Reut Tsarfaty, Djamé Seddah, Yoav Goldberg, Sandra Kübler, Marie Candito, Jennifer Foster, Yannick Versley, Ines Rehbein, and Lamia Tounsi. 2010. Statistical parsing of morphologically rich languages (spmrl): what, how and whither. In Proceedings of the NAACL HLT 2010 First Workshop on Statistical Parsing of Morphologically-Rich Languages, pages 1-12. Association for Computational Linguistics.
Nasser Zalmout and Nizar Habash. 2019. Joint disacritization, lemmatization, normalization, and fine-grained morphological tagging. arXiv preprint arXiv:1910.02267.
Daniel Zeman, Martin Popel, Milan Straka, Jan Hajič, Joakim Nivre, Filip Ginter, Juhani Luotolahti, Sampo Pyysalo, Slav Petrov, Martin Potthast, Francis Tyers, Elena Badmaeva, Memduh Gokirmak, Anna Nedoluzhko, Silvie Cinková, Jan Hajič jr., Jaroslava Hlaváčová, Václava Kettnerová, Zdenka Urešová, Jenna Kanerva, Stina Ojala, Anna Missilä, Christopher D. Manning, Sebastian Schuster, Siva Reddy, Dima Taji, Nizar Habash, Herman Leung, Marie-Catherine de Marneffe, Manuela Sanguinetti, Maria Simi, Hiroshi Kanayama, Valeria de Paiva, Kira Droganova, Héctor Martínez Alonso, Căgni Çöltekin, Umut Sulubacak, Hans Uszkoreit, Vivien Macketanz, Aljoscha Burchardt, Kim Harris, Katrin Marheinecke, Georg Rehm, Tolga Kayadelen, Mohammed Attia, Ali Elkahky, Zhuoran Yu, Emily Pitler, Saran Lertpradit, Michael Mandl, Jesse Kirchner, Hector Fernandez Alcalde, Jana Strnadová, Esha Banerjee, Ruli Manurung, Antonio Stella, Atsuko Shimada, Sookyoung Kwak, Gustavo Mendonça, Tatiana Lando, Rattima Ni
tisaroj, and Josie Li. 2017. CoNLL 2017 shared task: Multilingual parsing from raw text to universal dependencies. In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 1-19, Vancouver, Canada. Association for Computational Linguistics.
A.1 Class-noise for our language classifier
Given the large number of noisy data in Common Crawl, a class noise is added to the classification model we presented section 4.2 and is built according to the following empirical rules:
- If the word "url" appears more than two times, the sentence is added to class noise.
- If the sentence has more than four ” [ ” , the sentence is added to class noise.
- The same rule as above works for
  - ” , ” , ” {” or “} ” symbols.
- If the word "http" appears more than two times, the sentence is added to class noise.
- If more than two "@” character, the sentence is added to class noise in order to capture sentences with email address or tweets with only mentioned people.
- If the phrase "WARC-Refers-To" appears, the sentence is added to class noise.
A.2 Post-processing steps