# Enriching the NArabizi Treebank: A Multifaceted Approach to Supporting an Under-Resourced Language

Arij Riabi, Menel Mahamdi, Djamé Seddah

► To cite this version:

Arij Riabi, Menel Mahamdi, Djamé Seddah. Enriching the NArabizi Treebank: A Multifaceted Approach to Supporting an Under-Resourced Language. Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII), Jul 2023, Toronto, Canada. pp.266-278, 10.18653/v1/2023.law-1.26. hal-04243832

HAL Id: hal-04243832

https://inria.hal.science/hal-04243832v1

Submitted on 16 Oct 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Enriching the NArabizi Treebank: A Multifaceted Approach to Supporting an Under-Resourced Language

Arij Riabi Menel Mahamdi Djamé Seddah

Inria, Paris

{firstname, lastname} @inria.fr

## Abstract

In this paper we address the scarcity of annotated data for NArabizi, a Romanized form of North African Arabic used mostly on social media, which poses challenges for Natural Language Processing (NLP). We introduce an enriched version of NArabizi Treebank (Seddah et al., 2020) with three main contributions: the addition of two novel annotation layers (named entity recognition and offensive language detection) and a re-annotation of the tokenization, morpho-syntactic and syntactic layers that ensure annotation consistency. Our experimental results, using different tokenization schemes, showcase the value of our contributions and highlight the impact of working with non-gold tokenization for NER and dependency parsing. To facilitate future research, we make these annotations publicly available. Our enhanced NArabizi Treebank paves the way for creating sophisticated language models and NLP tools for this under-represented language.

## 1 Introduction

Despite the abundance of rich and diverse dialects worldwide, each possessing distinctive features and characteristics, many of these dialects still lack the necessary resources and support to enable their speakers to access modern technologies in their own language (Joshi et al., 2020). Therefore, it is imperative to undertake endeavors aimed at creating annotated corpora, developing language models, and establishing dictionaries and grammars for low-resource dialects. These efforts are crucial for the preservation and advancement of these dynamic languages, which encapsulate unique cultures, histories, and experiences within their respective communities.

One notable example of such an effort is the Masakhane community, which is dedicated to enhancing natural language processing (NLP) research for African languages through significant initiatives such as MasakhaNER (Adelani et al.,

2021). Similar efforts are ongoing for Indonesian languages (Cahyawijaya et al., 2022).

In addition, a long-standing and somewhat unrelated initiative known as the Universal Dependencies project (Nivre et al., 2020) originally aimed to provide a standardized set of syntactic guidelines for a limited number of languages turned out to become the recipient of numerous treebank initiatives for low-resource languages. These initiatives not only adopted the initial guidelines but also expanded upon them to accommodate the unique idiosyncrasies of each language.

In this work, we aim to enhance a pre-existing multi-view treebank devoted to a very low-resource language, namely the North-African Arabic dialect written in Latin script, collected from Algerian sources and denoted as the Narabizi treebank, the first available for this dialect, where Arabizi refers to both the practice of writing Arabic using the Latin alphabet and N for the North African dialect (Seddah et al., 2020). Made of noisy user-generated content that exhibits a high level of language variability, its annotations faced many challenges as described by the authors and contained remaining errors (Touileb and Barnes, 2021).

Our work builds on previous efforts to annotate and standardize treebank annotations for low-resource languages to enhance the quality and consistency of linguistic resources (Schluter and van Genabith, 2007; Sade et al., 2018; Türk et al., 2019; Zariquiey et al., 2022).

Following previous research, we consider the impact of refining annotation schemes on downstream tasks. Mille et al. (2012) examine how much a treebank's performance relies on its annotation scheme and whether employing a more linguistically rich scheme would decrease performance. Their findings indicate that using a fine-grained annotation for training a parser does not necessarily improve performance when parsing with a coarse-grained tagset. This observation is relevant to our study as
we expect refining the treebank could enhance the parsing performance even though the inherent variability of this language, which, tied to its small size treebank, could bring a negative impact on such enhancements.

On the other hand, the experiments conducted by Schluter and van Genabith (2007) demonstrate that using a cleaner and more coherent treebank yields superior results compared to a treebank with a training set five times larger. This observation highlights the significance of high-quality dataset annotations, particularly for smaller datasets. This understanding primarily drives the goal of improving the NArabizi treebank's annotations.

In this context, we propose a heavily revised version of NArabizi treebank (Seddah et al., 2020) that includes two novel annotation layers for Named Entity Recognition (NER) and offensive language detection. One of the goals of this work is also to study the impact of non-gold tokenization on NER, a scenario almost never investigated by the community (Bareket and Tsarfaty, 2021). Our primary contributions are as follows:

* Using error mining tools, we release a new corrected version of the treebank, which leads to improved downstream task performance.

* We show that corrections made to a small size treebank of a highly variable language favorably impacts the performance of NLP models trained on it.

* We augment the treebank by adding NER annotations and offensive language detection, expanding its applicability in various NLP tasks.

* We homogenize tokenization across the dataset, analyze the impact of proper tokenization on UD tasks and NER and conduct a realistic evaluation on predicted tokenization, including NER evaluation.

The enhanced version of the Narabizi Treebank is freely available.<fnref n="1" />

## 2 Related work

**NArabizi** The Arabic language exhibits diglossia, where Modern Standard Arabic (MSA) is employed in formal contexts, while dialectal forms are used informally (Habash, 2010). Dialectal forms, which display significant variability across regions and predominantly exist in spoken form, lack standardized spelling when written. Many Ara-

bic speakers employ the Latin script for transcribing their dialects online, using digits and symbols for phonemes not easily mapped to Latin letters (Seddah et al., 2020). This written form, known as Arabizi and its North African variant, NArabizi, often showcases code-switching with French and Amazigh (Amazouz et al., 2017). Textual resources for Arabizi primarily consist of noisy, user-generated content (Foster, 2010; Seddah et al., 2012; Eisenstein, 2013), complicating the creation of supervised models or collection of extensive pre-training datasets. The original NArabizi treebank (Seddah et al., 2020), contains about 1500 sentences. The sentences are randomly sampled from the romanized Algerian dialectal Arabic corpus of Cotterell et al. (2014) and from a small corpus of lyrics from Algerian dialectal Arabic songs popular among the younger generation. This treebank is manually annotated with morpho-syntactic information (parts-of-speech and morphological features), together with glosses and code-switching labels at the word level, as well as sentence-level translations to French. Moreover, this treebank also contains 36% of French tokens. Since its creation, this treebank spawned two derived versions that first added a transliteration to the Arabic script at the word level and sentiment and topic annotation at the sentence level (Touileb and Barnes, 2021). In parallel to our own corrections and annotation work<fnref n="2" />, Touileb (2022) extended this work to include a named-entity annotation layer.

**Treebanking for User-generated Content** Treebanks and annotated corpora have greatly impacted NLP tools, applications, and research in general. Despite the challenges of constructing large and structurally consistent corpora, which requires considerable effort and time, many in the field considered this pursuit valuable and necessary (de Marneffe et al., 2021). However, constructing treebanks for user-generated content is more challenging due to the extensive variation in language usage and style, the prevalence of non-standard spellings and grammar, and the necessity for domain-specific annotations (Sanguinetti et al., 2022). Interest in treebanking user-generated content, such as social media posts and online forum discussions, has risen, and numerous efforts have been undertaken to create treebanks for user-generated content (Foster et al., 2011; Seddah et al., 2012; Sanguinetti

<footnote n="1">¹https://gitlab.inria.fr/ariabi/release-narabizi-treebank</footnote>

<footnote n="2">²Released on November 26th, 2022, the same day as the publication of (Touileb, 2022).</footnote>
et al., 2018; Rehbein et al., 2019; Sanguinetti et al., 2020).

## NER for Dialects and User-generated Content

NER is an information extraction task that identifies and categorizes entities at the token level. It is an extensively investigated NLP task with numerous datasets and models for various languages. However, datasets for low-resource languages are rare, and NER datasets for social media platforms such as Twitter predominantly exist for English (Ritter et al., 2011; Derczynski et al., 2016, 2017).

A prominent NER dataset for lower-than-English resource languages is the CoNLL 2002 Shared Task dataset (Tjong Kim Sang, 2002), which provides NER annotations for four languages: Dutch, Spanish, Chinese, and Czech. Additionally, the WikiAnn dataset (Pan et al., 2017) includes NER annotations for several low-resource languages. Nevertheless, it is derived from Wikipedia content which is not well-suited for NER tasks involving user-generated content. As mentioned above, Touileb (2022) added a NER annotation for the first version of the NArabizi treebank. However, they did not address the tokenization issues inherent in the dataset and used a different annotation scheme. The following sections delve deeper into the tokenization challenges and the differences between the two datasets.

## 3 Extending a Low-resource Language treebank

In this section, we outline our methodology for expanding and enhancing the NArabizi treebank. We start by re-annotating tokenization, morphosyntactic, and syntactic layers to ensure consistency, followed by detailing the annotation guidelines and procedures for NER and Offensive Language detection. We refer to the initial treebank introduced by Seddah et al. (2020) as NArabiziV1 and our extended version as NArabiziV2.

### 3.1 Maintaining Consistency in Treebank Annotations

We start with an extended clean-up of the NArabiziV1 formatting, which involves reinstating missing part-of-speech tags and rectifying Conllu formatting discrepancies. Then, we embark on general error mining in the lexical and syntactical annotation and correction phase. We implement this stage using semi-automated methods. We do not change the UD tagsets used in the original treebank.

**Error Mining** We use the UD validator Vr2.11 <fnref n="3" />, a tool designed to assess the annotation of treebanks in UD and ensure compliance with the UD specifications. The validator is specifically employed to detect common errors, such as invalid dependency relations, incorrect part-of-speech tags, and inconsistent usage of features like tense and aspect. By leveraging the UD validator, we guarantee that our dataset is syntactically consistent and conforms to the standards established by the UD project. These changes encompass correcting cycle and projectivity issues and removing duplicates.

We also use Errator (Wisniewski, 2018), a data mining tool, to pinpoint inconsistencies in our dataset. It implements the annotation principle presented by Boyd et al. (2008), which suggests that if two identical word sequences have different annotations, one is likely erroneous.

We remove the duplicated sentences when the text field is an exact match and fix duplicated sentence identification for different sentences. We also fixed some problems with the original text, such as Arabic characters encoding and sentence boundaries.

**Tokenization** We address tokenization concerns to uphold consistency in the NArabizi Treebank annotations. Furthermore, we introduce targeted adjustments to resolve issues related to segmenting specific word classes, including conjunctions, interjections (e.g., “ya”), determiners, and prepositions, especially when adjacent to noun phrases. For example, we segment determiners at the initial vowel (“a” or “e”), as demonstrated in the examples “e ssalam” (“the peace”) and “e dounoub” (“the sins”). The lemma field for these terms is aligned with the French translation for the splitting (e.g., “e ssalam” ⇒ “la paix” (“the peace”)). For prepositions, we perform splitting at the first letter followed by “i” when possible, as seen in “brabi” ⇒ “b rabi” (“with my god”). We also establish rules for segmenting determiners and proper nouns. When possible, we separate prepositions at the initial letter and “i” and instituted guidelines for segmenting determiners and proper nouns. We implement these alterations for splitting using the Grew graph rewriting tool for NLP (Guillaume, 2021) to improve the consistency and quality of the treebank annotations. Additionally, we fix all the problems mentioned by Touileb and Barnes (2021) regarding the incoherence of the

<footnote n="3">³https://github.com/UniversalDependencies/too ls/releases/tag/r2.11</footnote>
<figure><img src="image_2.png" /><figcaption>Figure 1: Illustration of an example from the NArabizi treebank before and after the modifications.</figcaption></figure>

tokenization, wrong translations, and incoherent
annotations.

**Translation** The translation quality is also enhanced; previously, translations were not consistently carried out by Algerian speakers, resulting in local expressions and phrases being frequently misinterpreted, either in a literal manner or, at times, entirely inaccurately. This had implications for lexical and syntactical annotation. For instance, the term “skara” was initially annotated as “on purpose” but was later revised to “taunting”. Recognizing that “skara fi” represents a local expression facilitates annotation and promotes corpus harmonization.

**Example** In Figure 1, we illustrate a parse tree
before and after applying several corrections. Tok-
enization errors in French were rectified (“jetaime”
⇒ “je t aime”), and Arabic prepositions, articles,
and conjunctions were separated from the nouns
or adverbs they were attached to (“fal3ali” ⇒ “f
al 3ali”, “wdima” ⇒ “w dima”). We also correct
some dependency relations: the previous “obj” rela-
tion between the verb “aimer” and the proper noun
“madjid” was altered to “vocative”.

**Interesting Properties** The corpus displays
several interesting linguistic features, including
parataxis, goeswith, and dislocated structures, char-
acteristic of oral productions and user-generated
content. A deeper examination of the root/parataxis
ratio and the average parataxis per tree in the cor-

pus, which contains 2066 parataxis for 1287 sen-
tences, shows that the corpus exhibits a high level
of juxtaposed clauses resulting from the absence
of punctuation. Given the initial data sources (web
forums), it is likely that these end of sentences
markers were initially present as carriage returns.

As pointed out by Seddah et al. (2020) the corpus also exhibits a high level of spelling variation, reflecting the speakers’ diversity in terms of geography and accents. Furthermore, analyzing the number of sentences without a verb and the average number of verbs per sentence shows that NArabizi speakers tend to favor nominalization, as seen in the abundance of ellipses (e.g., “rabbi m3ak” which translates in English to “God bless you”).

## 3.2 Annotation Methodology for NER and Offensive Language Detection

**Named Entity Recognition** Our NER annotation guidelines are based on the revised tokenization of the NArabizi treebank, which ensures consistency between token-level annotations, an essential aspect of multi-task learning. We use the Inception tool (Klie et al., 2018) for our manual annotation by two native speakers, adhering to the IOB2 Scheme (Tjong Kim Sang and Veenstra, 1999). Each word is labeled with a tag indicating whether it is at the beginning, inside, or outside of a named entity. In case of disagreement between annotators, the multiple annotations were subsequently discussed until agreement was reached, and one annotation was selected to be retained. We extend the FTB NER
(Sagot et al., 2012) French treebank annotations. Our annotation contains the following NE types: PER for real or fictional persons, ORG for organizations, LOC for locations, COMP for companies, and OTH for brands, events, and products.

In cases of ambiguity between products and companies, we adhere to the decision made in the FTB dataset. For person names, we exclude grammatical or contextual words from the mention. We annotate football teams as organizations, and we annotate mentions of "Allah" or "Rabi" as PERderivA. The PERderiv annotation is applied to groups of individuals who originate from or share the same location. Country names are consistently labeled as locations, irrespective of the context. TV channels and ambiguous brand names are annotated as companies, while religious groups are not designated entities. The names of football stadiums are classified under OTH, whereas journal names are identified as organizations.

Table 1 presents the distribution of entities, with a similar distribution observed across both the development and test splits. The most frequent entity type is PERderivA, while the least frequent is COMP.

<table><caption>Table 1: Named entity type distribution across train, dev, and test splits.</caption><thead><tr><th>Type</th><th>train</th><th>dev</th><th>test</th><th>Total</th></tr></thead><tbody><tr><td>PER</td><td>371</td><td>61</td><td>47</td><td>479</td></tr><tr><td>LOC</td><td>358</td><td>58</td><td>50</td><td>466</td></tr><tr><td>ORG</td><td>200</td><td>23</td><td>28</td><td>251</td></tr><tr><td>COMP</td><td>6</td><td>5</td><td>3</td><td>14</td></tr><tr><td>OTH</td><td>44</td><td>6</td><td>7</td><td>57</td></tr><tr><td>PERderiv</td><td>96</td><td>14</td><td>13</td><td>123</td></tr><tr><td>PERderivA</td><td>386</td><td>57</td><td>66</td><td>509</td></tr><tr><td>Total</td><td>1461</td><td>224</td><td>214</td><td>1899</td></tr></tbody></table>

<table><caption>Table 2: Statistics of the deduplicated corpus across train, dev, and test splits. The train-dev intersection contains 549 tokens, the train-test intersection contains 551 tokens, and the dev-test intersection contains 266 tokens.</caption><thead><tr><th>Type</th><th>train</th><th>dev</th><th>test</th></tr></thead><tbody><tr><td>nb sentences</td><td>1003</td><td>139</td><td>145</td></tr><tr><td>nb tokens</td><td>15522</td><td>2124</td><td>2118</td></tr><tr><td>nb unique tokens</td><td>6652</td><td>1284</td><td>1327</td></tr></tbody></table>

Table 2 displays the number of unique words which can provide information about the language used in the corpus. The fact that the count of unique tokens constitutes nearly half of the total tokens suggests that the language used in the corpus is

complex and diverse, with a wide range of vocabulary and expressions. This can make it more challenging for NER algorithms to accurately identify and classify named entities in the corpus.

Touileb (2022) recently introduced NERDz, a version of the NArabizi treebank annotated for NER. As our dataset's annotation labels differ from theirs, we establish a mapping between the two annotation schemes to enable comparisons (cf. see Table 10 in the appendix A). Our schemes also differ in named entities' scope, as we split contracted forms, ours only cover the nominal phrase parts. Regarding nouns, such as "bled", which means country, some are annotated as entity GPE in NERDz, which is not the case in our dataset. Also, the names of stadiums are annotated as LOC in NERDz while they are considered OTH in our dataset. Similarly, for "equipe nationale", which means national team is annotated ORG in NERDz, while we do not consider it as an entity, following the FTB NER's guidelines. Added to annotator divergences, this may explain the differences in the count of the entities.

**Offensive Language Classification** The annotation process for offensive language classification was conducted manually by three annotators with diverse backgrounds. The annotators consisted of two females and one male, each bringing unique expertise to the task. One female annotator is a Ph.D. student in NLP, the other is a Ph.D. student in political sciences, and the male annotator is an engineer with in-depth knowledge of North African football, a prominent topic in the dataset.

The annotators were asked to annotate every sentence as offensive (OFF) or non-offensive (NOT-OFF). Offensive posts included any form of unacceptable language, targeted offense (veiled or direct), insults, threats, profane language, and swear words. To maintain objectivity and minimize potential bias, the annotators were not granted access to the other annotators' work and were not allowed to discuss their annotations with one another. This approach ensured the independence of their judgments, allowing for a more reliable evaluation of the offensive language classification process. For the offensive annotation, the two female annotators did not usually agree with the male annotator as they have different backgrounds and hence different opinions about football-related sentences. The final label is determined through a majority voting process. Additionally, we calculate the average
pair-wise Cohen's κ (Cohen, 1960) to highlight how hard this task was. The average κ value is 0.54, indicating a moderate agreement between annotators, common in sentence level annotation for annotators with different backgrounds and topic familiarity (Bobicev and Sokolova, 2017). This disagreement likely stems from the interpretation of terms that can be considered offensive or non-offensive depending on either the dialect or context.

Table 3 presents the distribution of non-offensive and offensive language instances. The dataset features an imbalance between non-offensive and offensive classes, with non-offensive samples being considerably more frequent in each split.

<table><caption>Table 3: Offensive language detection distributions across train, dev, and test splits.</caption><thead><tr><th>Split</th><th>Non-Offensive</th><th>Offensive</th></tr></thead><tbody><tr><td>Train</td><td>804</td><td>199</td></tr><tr><td>Dev</td><td>86</td><td>53</td></tr><tr><td>Test</td><td>118</td><td>27</td></tr></tbody></table>

# 4 Dataset Evaluation

We evaluate the NarabiziV2 dataset on UD parsing tasks and NER using standard transfer learning architectures on which we vary the pre-trained language model and the tokenization scenario.

**New NArabizi CharacterBERT Model** Following Riabi et al. (2021), we train a CharacterBERT (El Boukkouri et al., 2020) model, a character-based BERT variant, on a NArabizi new filtered corpus. The authors demonstrate that CharacterBERT achieves significant results when dealing with noisy data while being extremely data efficient.

We improve the initial pre-training dataset used by Riabi et al. (2021) by more stringently filtering non-NArabizi examples from the 99k instances provided by Seddah et al. (2020), as well as incorporating new samples from the CTAB corpus (Amara et al., 2021) and 12k comments extracted from various Facebook and forum posts, mostly in the Tunisian dialect taken from different datasets listed by Younes et al. (2020). This results in a 111k sentence corpus. To exclude non-NArabizi content, we first use a language detection tool (Nakatani, 2010) with a 0.9 confidence threshold to eliminate text in French, English, Hindi, Indonesian, and Russian, which are commonly found in mixed Arabizi data. Following the filtering process, a bootstrap

sampling method is adopted to randomly select a subset of the remaining text for manual annotation. This annotated text is then used to train an SVM classifier for NArabizi detection. The final dataset, containing 91k annotated text instances after deduplication, focuses on North African Arabizi text. We make this corpus publicly available.

**Sub-word Models** We also evaluate the performance of subword-based language models, monolingual and multilingual. For the multilingual subword-based language model, we use mBERT, the multilingual version of BERT (Devlin et al., 2018). It is trained on data from Wikipedia in 104 different languages, including French and Arabic. Muller et al. (2020) demonstrated that such a model could be transferred to NArabizi to some degree. Finally, our monolingual model is DziriBERT (Abdaoui et al., 2021), a monolingual BERT model trained on 1.2M tweets from major and highly-populated Algerian cities scrapped using a set of popular keywords in the Algerian spoken dialect in both Arabic and Latin scripts.

# 5 Results

## 5.1 New Results for UD

For our updated version of the treebank, we present results for models trained and tested on NArabiziV2, as shown in Table 4 and highlighted by a red box. These results represent the new state-of-the-art performance for the treebank, and we report findings for three previously used models. The DziriBERT model exhibits the best performance; however, CharacterBERT delivers competitive results while being trained on a mere 7.5% of the data used for training DziriBERT. This observation is consistent with the conclusions drawn by Riabi et al. (2021).

In order to assess the influence of the implemented corrections, we use NArabiziV1 and eliminate duplicate sentences<fnref n="4" />. For this comparison, we focused on the DziriBERT model's performance when trained on either NArabiziV1 or NArabiziV2 and tested on NArabiziV2, as denoted by the blue highlights in Table 4. Training on NArabiziV2 enhances the average scores for UPOS, UAS, and LAS by 3.5 points, illustrating the favorable outcomes of the refinements introduced in the NArabiziV2 dataset. This observation is further substantiated<fnref n="4" />.

<footnote n="4">⁴To use the prior version with an equivalent number of sentences, format errors must be rectified (earlier experiments with these sentences excluded them).</footnote>
<table><caption>Table 4: Results for UD on test set, DEV set is used for validation (with gold tokenization) (We report average of F1 scores over 5 seeds with the standard deviation)</caption><thead><tr><th rowspan="2">Model</th><th rowspan="2">Test Train</th><th colspan="3">NArabiziV1</th><th colspan="3">NArabiziV2</th></tr><tr><th>UPOS</th><th>UAS</th><th>LAS</th><th>UPOS</th><th>UAS</th><th>LAS</th></tr></thead><tbody><tr><td>mBERT</td><td rowspan="3">NArabiziV1</td><td>77.42 ± 1.52</td><td>68.91 ± 0.65</td><td>56.19 ± 0.86</td><td>74.59 ± 1.42</td><td>66.01 ± 0.47</td><td>53.19 ± 0.87</td></tr><tr><td>DziriBERT</td><td>83.57 ± 0.92</td><td>73.97 ± 0.72</td><td>62.04 ± 0.54</td><td>80.19 ± 0.82</td><td>70.28 ± 0.83</td><td>58.63 ± 0.78</td></tr><tr><td>CharacterBERT</td><td>76.19 ± 2.48</td><td>68.78 ± 0.36</td><td>55.14 ± 0.38</td><td>73.01 ± 2.05</td><td>66.10 ± 0.48</td><td>52.41 ± 0.50</td></tr><tr><td>mBERT</td><td rowspan="3">NArabiziV2</td><td>74.48 ± 0.95</td><td>66.03 ± 0.35</td><td>52.82 ± 0.66</td><td>79.65 ± 0.90</td><td>70.56 ± 0.32</td><td>58.08 ± 0.76</td></tr><tr><td>DziriBERT</td><td>78.75 ± 1.29</td><td>70.51 ± 0.43</td><td>57.51 ± 0.67</td><td>83.10 ± 1.60</td><td>74.26 ± 0.27</td><td>62.66 ± 0.52</td></tr><tr><td>CharacterBERT</td><td>72.24 ± 2.62</td><td>65.74 ± 0.24</td><td>51.86 ± 0.51</td><td>76.34 ± 2.68</td><td>69.84 ± 0.27</td><td>56.27 ± 0.54</td></tr></tbody></table>

tiated by examining the performance of Character-BERT and mBERT, reinforcing the validity of the noted improvements.

A comparative analysis of the results for models trained and tested on NArabiziV1, denoted by the blue box, and those for models trained and tested on NArabiziV2, denoted by the red box, reveals that NArabiziV2 generally yields superior evaluation scores. This observation underlines the impact of the treebank's consistency on the overall performance of the models. When we test on NArabiziV1, the model trained on NArabiziV1 gets better results than the model trained on NArabiziV2. The modifications in tokenization can explain this drop in performance.

## 5.2 Results for NER and Offensive Language Detection

**NER** Table 5 presents the results for NER<fnref n="6" />. The CharacterBERT model achieves the highest F1 scores for LOC and OTH categories, as well as the best performance for PERderiv and PERderivA. On the other hand, the DziriBERT model outperforms the other models in the ORG and PER categories. It is important to note that the performance varies significantly across the different categories, reflecting the diverse challenges posed by each entity type. For instance, some categories contain named entities with variations of the same word, such as “Allah”/“Alah”/“Elah”, which translates into God for PERderivA. Since CharacterBERT uses character-level information, it is more robust to noise, which explains the high performances for those entities.

**Offensive Language Detection** The imbalance between non-offensive and offensive instances is challenging during the models' training and eval-

uation. For example, we fail to train mBERT as it only predicts non-offensive labels corresponding to the majority class. This can also be explained by how hard the distinction between offensive and non-offensive content is without context and external knowledge, as explained before. This also raises the question of how relevant is the backgrounds of the annotators for the offensive detection dataset (Basile et al., 2020; Uma et al., 2021; Almanea and Poesio, 2022).

# 6 Discussion

## 6.1 Impact of the Pre-training Corpus

In Appendix A, we present the results of all our experiments using the CharacterBERT model trained by Riabi et al. (2021). We observe a heterogeneous improvement in performance, with predominantly better outcomes for our CharacterBERT. We hypothesize that the impact of filtering the training data may not be overly beneficial, possibly due to some smoothing during the training process. Both models' final training data sizes are comparable: 99k for CharacterBERT (Riabi et al., 2021) and 91k for our CharacterBERT. Nevertheless, we believe this new corpus can be a valuable resource for this language.

## 6.2 Impact of Tokenization

In this section, we investigate the tokenization influence on the enhanced NArabizi Treebank, with a particular emphasis on the homogenization of the tokenization <fnref n="7" /> and its subsequent impact on our tasks. We also evaluate the models in a realistic scenario where gold tokenization is unavailable. We use the UDPipe tokenizer (Straka et al., 2016) that employs a Gated Linear Units (GRUs) (Cho

<footnote n="6">⁶We use Seqeval (Nakayama, 2018) classification report.</footnote>

<footnote n="7">⁷We follow the terminology of UD where a tokenizer performs token segmentation (i.e. source tokens).</footnote>
<table><caption>Table 5: NER average of F1 scores over 5 seeds with the standard deviation with gold tokenization⁵.</caption><thead><tr><th>Model</th><th>LOC</th><th>ORG</th><th>PER</th><th>OTH</th><th>PERderiv</th><th>PERderivA</th><th>macro avg</th></tr></thead><tbody><tr><td>mBERT</td><td>82.93 ± 4.02</td><td>66.17 ± 6.61</td><td>61.84 ± 3.56</td><td>25.56 ± 14.64</td><td>57.98 ± 11.30</td><td>95.62 ± 1.24</td><td>65.02 ± 1.24</td></tr><tr><td>DziriBERT</td><td>85.84 ± 3.43</td><td><b>73.67</b> ± 4.03</td><td><b>73.42</b> ± 3.52</td><td>26.27 ± 4.23</td><td>57.47 ± 6.62</td><td>94.98 ± 1.39</td><td>68.61 ± 1.39</td></tr><tr><td>CharacterBERT</td><td><b>87.98</b> ± 1.77</td><td>70.16 ± 3.63</td><td>69.35 ± 3.01</td><td><b>31.27</b> ± 9.30</td><td><b>64.19</b> ± 7.03</td><td><b>96.13</b> ± 0.70</td><td><b>69.85</b> ± 0.70</td></tr></tbody></table>

<table><caption>Table 6: Offensive language detection F1 scores, off for offensive and Non-Off for non offensive</caption><thead><tr><th>Model</th><th>Off</th><th>Non-Off</th><th>macro avg</th></tr></thead><tbody><tr><td>mBERT</td><td>0.00 ± 0.00</td><td><b>89.73</b> ± 0.00</td><td>44.87 ± 0.00</td></tr><tr><td>DziriBERT</td><td><b>36.77</b> ± 10.88</td><td><b>84.78</b> ± 2.58</td><td>60.78 ± 6.21</td></tr><tr><td>CharacterBERT</td><td>24.58 ± 7.44</td><td>80.21 ± 3.66</td><td>52.39 ± 3.18</td></tr></tbody></table>

et al., 2014) artificial neural network for the identification of token and sentence boundaries in plain text. It processes fixed-length segments of Unicode characters and assigns each character to one of three classes: token boundary follows, sentence boundary follows, or no boundary. The tokenizer is trained using the Adam stochastic optimization method, employing randomly shuffled input sentences to ensure effective tokenization across various NLP tasks.

<table><caption>Table 7: Tokenization evaluation average scores over 5 folds</caption><thead><tr><th>Tokenizer</th><th>Prec</th><th>Recall</th><th>F1</th></tr></thead><tbody><tr><td>Tokens</td><td>97.10 ± 0.35</td><td>95.49 ± 0.45</td><td>96.29 ± 0.39</td></tr><tr><td>Multiwords</td><td>79.74 ± 4.30</td><td>33.81 ± 2.87</td><td>47.35 ± 2.59</td></tr><tr><td>Words</td><td>92.92 ± 0.65</td><td>88.06 ± 0.96</td><td>90.42 ± 0.80</td></tr></tbody></table>

We conduct a 5-fold evaluation using the UD-Pipe tokenizer and assess its performance based on the token-level, multiword, and word-level scores. The results in Table 7 show high scores for the tokens and words F1 scores demonstrate the tokenizer's efficacy in handling various tokens and words, which shows that the tokenization for NArabizi is learnable. We also notice sub-optimal performance regarding multi-words, due to their random occurrence nature.<fnref n="8" />.

For our following experiments, we train a tokenizer using the train and dev as held-out and tokenize the test set for evaluation. We do not predict the boundaries of the sentence.

**Post-tagging and Dependency Parsing** Table 8 presents the results for models trained on the NArabiziV2 training set and tested on both the predicted tokenization and the previous version of tokenization with gold annotations from NArabiziV2. The outcomes for the predicted tokenization indicate that despite having a well-performing tokenizer, as demonstrated in Table 7, there is still a substantial loss in performance when compared to the gold tokenization results, highlighted by the red box in Table 4. Similarly, using the tokenization from NArabiziV1 and gold annotations from NArabiziV2 also exhibits a significant drop in performance. This observation first highlights the impact of the corrections brought to standardize the treebank tokenization and then, given the difference of performance between predicted and gold tokens, calls for the development of morphological-analysers, crucial for Arabic-based dialects, as UD tokenization is indeed a morpho-syntactic process.

**Named Entity Recognition Evaluation on Non-Gold Tokenization** The conventional evaluation methodology for NER typically assigns entities to distinct token positions. Nevertheless, this method proves inadequate when the token count for evaluation differs from the number of gold tokens, which is almost always the case when processing user-generated content.

As a result, we adopt the evaluation strategy devised by Bareket and Tsarfaty (2021), which associates entities with their forms instead of their indices. This approach yields F1 scores based on strict, exact matches of surface forms for entities, irrespective of the category distinctions, thereby offering a more accurate and reliable evaluation in scenarios with varying token counts. In other words, the gold and predicted NE spans must exhibit an exact match regarding their form, boundaries, and associated entity type.

Table 9 presents the NER scores, considering our three main NE categories: PER, LOC, and ORG. As expected, we observe a decline in performance when evaluating the models using predicted tokenization. The CharacterBERT model exhibits the

<footnote n="8">⁸It is important to note that tokens refer to surface tokens (e.g., French “au” counts as one token), while words represent syntactic words (“au” is split into two words, “à” and “le”).</footnote>
<table><caption>Table 8: UD results for models trained on NArabiziV2 treebank and tested on test set with predicted tokenization and old tokenization from NArabiziV1</caption><thead><tr><th rowspan="2">Model</th><th colspan="3">Predicted tokenization</th><th colspan="3">NArabiziV1 tokenization</th></tr><tr><th>UPOS</th><th>UAS</th><th>LAS</th><th>UPOS</th><th>UAS</th><th>LAS</th></tr></thead><tbody><tr><td>mBERT</td><td>72.44 ± 0.87</td><td>61.40 ± 0.29</td><td>50.39 ± 0.64</td><td>75.84 ± 0.92</td><td>65.77 ± 0.40</td><td>54.15 ± 0.68</td></tr><tr><td>DziriBERT</td><td>76.27 ± 1.46</td><td>65.35 ± 0.39</td><td>55.04 ± 0.65</td><td>79.49 ± 1.63</td><td>70.04 ± 0.48</td><td>59.19 ± 0.70</td></tr><tr><td>CharacterBERT</td><td>70.03 ± 2.10</td><td>61.08 ± 0.18</td><td>49.13 ± 0.42</td><td>73.10 ± 2.33</td><td>65.37 ± 0.22</td><td>52.99 ± 0.50</td></tr></tbody></table>

<table><caption>Table 9: Comparison of NER scores for PER/ LOC/ ORG entities F1 micro average on predicted tokenization and gold tokenization averaged across five seeds.</caption><thead><tr><th>Model</th><th>Gold</th><th>Predicted</th></tr></thead><tbody><tr><td>mBERT</td><td>71.79 ± 2.30</td><td>66.76 ± 1.52</td></tr><tr><td>DziriBERT</td><td>75.56 ± 2.13</td><td>68.89 ± 2.64</td></tr><tr><td>CharacterBERT</td><td><strong>76.30</strong> ± 1.29</td><td><strong>70.54</strong> ± 2.00</td></tr></tbody></table>

best performance on gold and predicted tokenization. Moreover, when evaluated using predicted tokenization, all models demonstrate a similar performance drop. This demonstrates that there is an important gap when evaluating using gold tokenization, which raises the question of how much the current evaluation of NER models reflects the actual model performance in a realistic setting for noisy UGC.

## 7 Conclusion

In this paper, we present a comprehensive study on the development and refinement of the NArabizi Treebank (Seddah et al., 2020) by improving its annotations, consistency, and tokenization, as well as providing new annotations for NER and offensive language. Our work contributes to the enhancement of the NArabizi Treebank, making it a valuable resource for research on low-resource languages and user-generated content with high variability. We explore the impact of tokenization on the refined NArabizi treebank, employing the UDPipe tokenizer for our evaluation. The results demonstrate the tokenizer's effectiveness in handling various tokens and multiword expressions. Our experiments show that training and testing on the NArabiziv2 improve the UD tasks performances. Furthermore, we show the impact of the tokenization for NER and UD tasks, and we report results using predicted tokenization for evaluation to estimate the models' performance on raw data.

Future research could emphasize expanding the NArabizi Treebank towards other dialects and ex-

amining the treebank's potential applications in various NLP tasks. Our dataset is made freely available as part of the new version of the Narabizi Treebank<fnref n="9" />. The next release will additionally contain a set of other sentence translations prepared by a Tunisian speaker. These translations will be interesting for cross-dialect studies, given that the Narabizi corpus is predominantly made of Algerian dialect.

## Acknowledgements

We warmly thank the reviewers for their very valuable feedback. This work received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No. 101021607. We are grateful to Roman Castagné for his valuable feedback and proofreading and wish to gratefully acknowledge the OPAL infrastructure from Université Côte d'Azur for providing resources and support.

## References

Amine Abdaoui, Mohamed Berrimi, Mourad Oussalah, and Abdelouahab Moussaoui. 2021. Dziribert: a pre-trained language model for the algerian dialect. *arXiv preprint arXiv:2109.12346*.

David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D'souza, Julia Kreutzer, Constantine Liginos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, Stephen Mayhew, Israel Abebe Azime, Shamsuddeen H. Muhammad, Chris Chinenye Emezue, Joyce Nakatumba-Nabende, Perez Ogayo, Aremu Anuoluwapo, Catherine Gitau, Derguene Mbaye, Jesujoba Alabi, Seid Muhie Yimam, Tajuddeen Rabiu Gwadabe, Ignatius Ezeani, Rubungo Andre Niyongabo, Jonathan Mukiibi, Verah Otiende, Iroro Orife, Davis David, Samba Ngom, Tosin Adewumi, Paul Rayson, Mofetoluwa Adeyemi, Gerald Muriuki, Emmanuel Anebi, Chiamaka Chukwuneke, Nkiruka Odu, Eric Peter Wairagala, Samuel Oyerinde, Clemencia Siro, Tobius Saul Bateesa,

<footnote n="9">⁹https://gitlab.inria.fr/ariabi/release-narabizi-treebank</footnote>
Temilola Oloyede, Yvonne Wambui, Victor Akin-
ode, Deborah Nabagereka, Maurice Katusiime, Ayo-
dele Awokoya, Mouhamadane MBOUP, Dibora Ge-
breyohannes, Henok Tilaye, Kelechi Nwaike, De-
gaga Wolde, Abdoulaye Faye, Blessing Sibanda, Ore-
vaoghene Ahia, Bonaventure F. P. Dossou, Kelechi
Ogueji, Thierno Ibrahima DIOP, Abdoulaye Diallo,
Adewale Akinfaderin, Tendai Marengereke, and Sa-
lomey Osei. 2021. MasakhaNER: Named entity
recognition for African languages. *Transactions
of the Association for Computational Linguistics*,
9:1116–1131.

Dina Almanea and Massimo Poesio. 2022. ArMIS - the Arabic misogyny and sexism corpus with annotator subjective disagreements. In *Proceedings of the Thirteenth Language Resources and Evaluation Conference*, pages 2282–2291, Marseille, France. European Language Resources Association.

Amina Amara, Houcemeddine Turki, Mohamed Ali Hadj Taieb, Mohamed Ben Aouicha, and Kawthar Ellouze. 2021. Ctab: Corpus of tunisian arabizi. This corpus has been developed by the Data Engineering and Semantics Research Unit (DES-Unit), University of Sfax, Tunisia. It has been developed to increase the coverage of Latin Script in the NLP resources for Tunisian. It is included as a part of the Tunisian Arabic Corpus (http://www.tunisiya.org/).

Djegdjiga Amazouz, Martine Adda-Decker, and Lori Lamel. 2017. Addressing code-switching in french/algerian arabic speech. In *Interspeech 2017*, pages 62–66.

Dan Bareket and Reut Tsarfaty. 2021. Neural model-
ing for named entities and morphology (NEMO2).
*Transactions of the Association for Computational
Linguistics*, 9:909–928.

Valerio Basile et al. 2020. It's the end of the gold stan-
dard as we know it. on the impact of pre-aggregation
on the evaluation of highly subjective tasks. In *CEUR
WORKSHOP PROCEEDINGS*, volume 2776, pages
31–40. CEUR-WS.

Victoria Bobicev and Marina Sokolova. 2017. Inter-
annotator agreement in sentiment analysis: Machine
learning perspective. In *Proceedings of the Inter-
national Conference Recent Advances in Natural
Language Processing, RANLP 2017*, pages 97–102,
Varna, Bulgaria. INCOMA Ltd.

Adriane Boyd, Markus Dickinson, and W Detmar Meur-
ers. 2008. On detecting errors in dependency tree-
banks. *Research on Language and Computation*,
6(2):113–137.

Samuel Cahyawijaya, Holy Lovenia, Alham Fikri Aji, Genta Indra Winata, Bryan Wilie, Rahmad Mahendra, Christian Wibisono, Ade Romadhony, Karissa Vincentio, Fajri Koto, Jennifer Santoso, David Moeljadi, Cahya Wirawan, Frederikus Hudi, Ivan Halim Parmonangan, Ika Alfina, Muhammad Satrio Wicaksono, Ilham Firdausi Putra, Samsul Rahmadani, Yulianti Oenang, Ali Akbar Septiandri, James Jaya, Kaustubh D.

Dhole, Arie Ardiyanti Suryani, Rifki Afina Putri, Dan Su, Keith Stevens, Made Nindyatama Nityasya, Muhammad Farid Adilazuarda, Ryan Ignatius, Ryan-dito Diandaru, Tiezheng Yu, Vito Ghifari, Wenliang Dai, Yan Xu, Dyah Damapuspita, Cuk Tho, Ichwanul Muslim Karo Karo, Tirana Noor Fatyanosa, Ziwei Ji, Pascale Fung, Graham Neubig, Timothy Baldwin, Sebastian Ruder, Herry Sujaini, Sakriani Sakti, and Ayu Purwarianti. 2022. Nusacrowd: Open source initiative for Indonesian nlp resources.

Kyunghyun Cho, Bart van Merriënboer, Caglar Gul-
cehre, Dzmitry Bahdanau, Fethi Bougares, Holger
Schwenk, and Yoshua Bengio. 2014. Learning
phrase representations using RNN encoder–decoder
for statistical machine translation. In *Proceedings
of the 2014 Conference on Empirical Methods in
Natural Language Processing (EMNLP)*, pages 1724–
1734, Doha, Qatar. Association for Computational
Linguistics.

Jacob Cohen. 1960. A coefficient of agreement for nominal scales. *Educational and Psychological Measurement*, 20(1):37–46.

Ryan Cotterell, Adithya Renduchintala, Naomi Saphra, and Chris Callison-Burch. 2014. An Algerian Arabic-French code-switched corpus. In *Workshop on Free/Open-Source Arabic Corpora and Corpora Processing Tools Workshop Programme*, page 34.

Marie-Catherine de Marneffe, Christopher D. Man-
ning, Joakim Nivre, and Daniel Zeman. 2021. Uni-
versal Dependencies. *Computational Linguistics*,
47(2):255–308.

Leon Derczynski, Kalina Bontcheva, and Ian Roberts. 2016. Broad Twitter corpus: A diverse named entity recognition resource. In *Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers*, pages 1169–1179, Osaka, Japan. The COLING 2016 Organizing Committee.

Leon Derczynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017. Results of the WNUT2017 shared task on novel and emerging entity recognition. In *Proceedings of the 3rd Workshop on Noisy User-generated Text*, pages 140–147, Copenhagen, Denmark. Association for Computational Linguistics.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. *arXiv preprint arXiv:1810.04805*.

Jacob Eisenstein. 2013. What to do about bad language on the internet. In *Proceedings of the 2013 conference of the North American Chapter of the association for computational linguistics: Human language technologies*, pages 359–369.

Hicham El Boukkouri, Olivier Ferret, Thomas Lavergne, Hiroshi Noji, Pierre Zweigenbaum, and Jun'ichi Tsujii. 2020. CharacterBERT: Reconciling ELMo and
BERT for word-level open-vocabulary representations from characters. In *Proceedings of the 28th International Conference on Computational Linguistics*, pages 6903–6915, Barcelona, Spain (Online). International Committee on Computational Linguistics.

Jennifer Foster. 2010. “cba to check the spelling”: Investigating parser performance on discussion forum posts. In *Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics*, pages 381–384, Los Angeles, California. Association for Computational Linguistics.

Jennifer Foster, Özlem Çetinoğlu, Joachim Wagner, Joseph Le Roux, Joakim Nivre, Deirdre Hogan, and Josef van Genabith. 2011. From news to comment: Resources and benchmarks for parsing the language of web 2.0. In *Proceedings of 5th International Joint Conference on Natural Language Processing*, pages 893–901, Chiang Mai, Thailand. Asian Federation of Natural Language Processing.

Bruno Guillaume. 2021. Graph matching and graph rewriting: Grew tools for corpus exploration, maintenance and conversion. In *Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations*, pages 168–175.

Nizar Habash. 2010. *Introduction to Arabic Natural Language Processing*. Morgan and Claypool.

Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 6282–6293, Online. Association for Computational Linguistics.

Jan-Christoph Klie, Michael Bugert, Beto Boullosa, Richard Eckart de Castilho, and Iryna Gurevych. 2018. The inception platform: Machine-assisted and knowledge-oriented interactive annotation. In *proceedings of the 27th international conference on computational linguistics: system demonstrations*, pages 5–9.

Simon Mille, Alicia Burga, Gabriela Ferraro, and Leo Wanner. 2012. How does the granularity of an annotation scheme influence dependency parsing performance? In *Proceedings of COLING 2012: Posters*, pages 839–852, Mumbai, India. The COLING 2012 Organizing Committee.

Benjamin Muller, Benoît Sagot, and Djamé Seddah. 2020. Can multilingual language models transfer to an unseen dialect? a case study on north african arabizi. *arXiv preprint arXiv:2005.00318*.

Shuyo Nakatani. 2010. Language detection library for java.

Hiroki Nakayama. 2018. seqeval: A python framework for sequence labeling evaluation. Software available from https://github.com/chakki-works/seqeval.

Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginster, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal Dependencies v2: An evergrowing multilingual treebank collection. In *Proceedings of the Twelfth Language Resources and Evaluation Conference*, pages 4034–4043, Marseille, France. European Language Resources Association.

Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Cross-lingual name tagging and linking for 282 languages. In *Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 1946–1958, Vancouver, Canada. Association for Computational Linguistics.

Ines Rehbein, Josef Ruppenhofer, and Bich-Ngoc Do. 2019. tweeDe – a Universal Dependencies treebank for German tweets. In *Proceedings of the 18th International Workshop on Treebanks and Linguistic Theories (TLT, SyntaxFest 2019)*, pages 100–108, Paris, France. Association for Computational Linguistics.

Arij Riabi, Benoît Sagot, and Djamé Seddah. 2021. Can character-based language models improve downstream task performances in low-resource and noisy language scenarios? In *Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021)*, pages 423–436, Online. Association for Computational Linguistics.

Alan Ritter, Sam Clark, Mausam, and Oren Etzioni. 2011. Named entity recognition in tweets: An experimental study. In *Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing*, pages 1524–1534, Edinburgh, Scotland, UK. Association for Computational Linguistics.

Shoval Sade, Amit Seker, and Reut Tsarfaty. 2018. The Hebrew Universal Dependency treebank: Past present and future. In *Proceedings of the Second Workshop on Universal Dependencies (UDW 2018)*, pages 133–143, Brussels, Belgium. Association for Computational Linguistics.

Benoît Sagot, Marion Richard, and Rosa Stern. 2012. Annotation référentielle du corpus arboré de paris 7 en entités nommées. In *Traitement Automatique des Langues Naturelles (TALN)*, volume 2.

Manuela Sanguinetti, Cristina Bosco, Lauren Cas sidy, Özlem Çetinoğlu, Alessandra Teresa Cignarella, Teresa Lynn, Ines Rehbein, Josef Ruppenhofer, Djamé Seddah, and Amir Zeldes. 2020. Treebanking user-generated content: A proposal for a unified representation in Universal Dependencies. In *Proceedings of the Twelfth Language Resources and Evaluation Conference*, pages 5240–5250, Marseille, France. European Language Resources Association.
Manuela Sanguinetti, Cristina Bosco, Lauren Cas-
sidy, Özlem Çetinoğlu, Alessandra Teresa Cignarella,
Teresa Lynn, Ines Rehbein, Josef Ruppenhofer,
Djamé Seddah, and Amir Zeldes. 2022. Treebank-
ing user-generated content: a ud based overview of
guidelines, corpora and unified recommendations.
*Language Resources and Evaluation*, pages 1–52.

Manuela Sanguinetti, Cristina Bosco, Alberto Lavelli, Alessandro Mazzei, Oronzo Antonelli, and Fabio Tamburini. 2018. PoSTWITA-UD: an Italian Twitter treebank in Universal Dependencies. In *Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)*, Miyazaki, Japan. European Language Resources Association (ELRA).

Natalie Schluter and Josef van Genabith. 2007. Prepar-
ing, restructuring, and augmenting a french treebank:
Lexicalised parsers or coherent treebanks?

Djamé Seddah, Farah Essaidi, Amal Fethi, Matthieu Futeral, Benjamin Muller, Pedro Javier Ortiz Suárez, Benoît Sagot, and Abhishek Srivastava. 2020. Building a user-generated content North-African Arabizi treebank: Tackling hell. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 1139–1150, Online. Association for Computational Linguistics.

Djamé Seddah, Benoit Sagot, Marie Candito, Virginie Mouilleron, and Vanessa Combet. 2012. The French Social Media Bank: a treebank of noisy user generated content. In *Proceedings of COLING 2012*, pages 2441–2458, Mumbai, India. The COLING 2012 Organizing Committee.

Milan Straka, Jan Hajič, and Jana Straková. 2016. UD-Pipe: Trainable pipeline for processing CoNLL-U files performing tokenization, morphological analysis, POS tagging and parsing. In *Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)*, pages 4290–4297, Portorož, Slovenia. European Language Resources Association (ELRA).

Erik F. Tjong Kim Sang. 2002. Introduction to the CoNLL-2002 shared task: Language-independent named entity recognition. In *COLING-02: The 6th Conference on Natural Language Learning 2002 (CoNLL-2002)*.

Erik F. Tjong Kim Sang and Jorn Veenstra. 1999. Representing text chunks. In *Ninth Conference of the European Chapter of the Association for Computational Linguistics*, pages 173–179, Bergen, Norway. Association for Computational Linguistics.

Samia Touileb. 2022. Nerdz: A preliminary dataset of named entities for algerian. In *Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing*, pages 95–101.

Samia Touileb and Jeremy Barnes. 2021. The interplay between language similarity and script on a novel multi-layer Algerian dialect corpus. In *Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021*, pages 3700–3712, Online. Association for Computational Linguistics.

Utku Türk, Furkan Atmaca, Şaziye Betül Özateş, Abdul-
latif Köksal, Balkiz Ozturk Basaran, Tunga Gungor,
and Arzucan Özgür. 2019. Turkish treebanking: Uni-
fying and constructing efforts. In *Proceedings of
the 13th Linguistic Annotation Workshop*, pages 166–
177, Florence, Italy. Association for Computational
Linguistics.

Alexandra N Uma, Tommaso Fornaciari, Dirk Hovy, Sil-
viu Paun, Barbara Plank, and Massimo Poesio. 2021.
Learning from disagreement: A survey. *Journal of
Artificial Intelligence Research*, 72:1385–1470.

Guillaume Wisniewski. 2018. Errator: a tool to help detect annotation errors in the universal dependencies project. In *Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)*.

Jihene Younes, Emna Souissi, Hadhemi Achour, and Ahmed Ferchichi. 2020. Language resources for maghrebi arabic dialects’ nlp: a survey. *Language Resources and Evaluation*, 54(4):1079–1142.

Roberto Zariquiey, Claudia Alvarado, Ximena Echevarría, Luisa Gomez, Rosa Gonzales, Mariana Illescas, Sabina Oporto, Frederic Blum, Arturo Oncevay, and Javier Vera. 2022. Building an endangered language resource in the classroom: Universal Dependencies for kakataibo. In *Proceedings of the Thirteenth Language Resources and Evaluation Conference*, pages 3840–3851, Marseille, France. European Language Resources Association.

A Appendix

A.1 Datasets

<table><caption>Table 10: Mapping of NER labels in our dataset to the
Published NERDz dataset (Touileb, 2022).</caption>
  <thead>
    <tr>
      <th colspan="2">NERDz</th>
      <th colspan="2">Our dataset</th>
    </tr>
    <tr>
      <th>Entities</th>
      <th>Count</th>
      <th>Entities</th>
      <th>Count</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>PER</td>
      <td>467</td>
      <td>PER</td>
      <td>479</td>
    </tr>
    <tr>
      <td>GPE/LOC</td>
      <td>479</td>
      <td>LOC</td>
      <td>466</td>
    </tr>
    <tr>
      <td>ORG</td>
      <td>290</td>
      <td>ORG/COMP</td>
      <td>265</td>
    </tr>
  </tbody>
</table>

A.2 Results with CharacterBERT from (Riabi et al., 2021)
<table><caption>Table 11: Results for UD on test set, DEV set is used for validation (with gold tokenization) (We report average of F1 scores over 5 seeds with the standard deviation)</caption><thead><tr><th rowspan="2">Model</th><th rowspan="2">Test Train</th><th colspan="3">NArabiziV1</th><th colspan="3">NArabiziV2</th></tr><tr><th>UPOS</th><th>UAS</th><th>LAS</th><th>UPOS</th><th>UAS</th><th>LAS</th></tr></thead><tbody><tr><td>CharacterBERT (Riabi et al., 2021)</td><td rowspan="2">NArabiziV1</td><td>75.33 ± 2.77</td><td>67.86 ± 0.95</td><td>54.40 ± 0.81</td><td>72.33 ± 2.60</td><td>65.17 ± 0.79</td><td>51.51 ± 1.05</td></tr><tr><td>CharacterBERT (Ours)</td><td><b>76.19</b> ± 2.48</td><td><b>68.78</b> ± 0.36</td><td><b>55.14</b> ± 0.38</td><td><b>73.01</b> ± 2.05</td><td><b>66.10</b> ± 0.48</td><td><b>52.41</b> ± 0.50</td></tr><tr><td>CharacterBERT (Riabi et al., 2021)</td><td rowspan="2">NArabiziV2</td><td><b>72.46</b> ± 3.19</td><td>65.30 ± 0.50</td><td>51.84 ± 0.68</td><td><b>79.65</b> ± 0.90</td><td><b>70.56</b> ± 0.32</td><td><b>58.08</b> ± 0.76</td></tr><tr><td>CharacterBERT (Ours)</td><td>72.24 ± 2.62</td><td><b>65.74</b> ± 0.24</td><td><b>51.86</b> ± 0.51</td><td>76.34 ± 2.68</td><td>69.84 ± 0.27</td><td>56.27 ± 0.54</td></tr></tbody></table>

<table><caption>Table 12: NER average of F1 scores over 5 seeds with the standard deviation with gold tokenization¹⁰.</caption><thead><tr><th>Model</th><th>LOC</th><th>ORG</th><th>PER</th><th>OTH</th><th>PERderiv</th><th>PERderivA</th></tr></thead><tbody><tr><td>CharacterBERT (Riabi et al., 2021)</td><td>86.80 ± 2.01</td><td>68.53 ± 6.09</td><td>65.36 ± 2.74</td><td><b>45.16</b> ± 13.60</td><td>58.96 ± 10.42</td><td>95.00 ± 1.32</td></tr><tr><td>CharacterBERT (Ours)</td><td><b>87.98</b> ± 1.77</td><td><b>70.16</b> ± 3.63</td><td><b>69.35</b> ± 3.01</td><td>31.27 ± 9.30</td><td><b>64.19</b> ± 7.03</td><td><b>96.13</b> ± 0.70</td></tr></tbody></table>

<table><caption>Table 13: Offensive language detection F1 scores, *off* for offensive and *Non-Off* for non offensive</caption><thead><tr><th>Model</th><th>Off</th><th>Non-Off</th><th>macro avg</th></tr></thead><tbody><tr><td>CharacterBERT (Riabi et al., 2021)</td><td><b>36.29</b> ± 5.73</td><td>76.49 ± 3.81</td><td><b>56.39</b> ± 2.95</td></tr><tr><td>CharacterBERT (Ours)</td><td>24.58 ± 7.44</td><td><b>80.21</b> ± 3.66</td><td>52.39 ± 3.18</td></tr></tbody></table>