Faculty of Engineering and Technology
Master in Computing“Towards Building a Corpus for Palestinian Dialect”“‫”نحو بناء مدونة للعامية الفلسطينية‬By: Diyam Fuad Akra
Supervisor: Dr. Mustafa Jarrar
Committee:
Dr. Nizar Habash
Dr. Mahdi Arar
Dr. Abualseoud HananiThis Thesis was submitted in partial fulfillment of the requirements for
the Master’s Degree in Computing from the Faculty of Graduate Studies
at Birzeit University, Palestine
28-01-2015

Faculty of Engineering and Technology
Master in Computing“Towards Building a Corpus for Palestinian Dialect”“‫”نحو بناء مدونة للعامية الفلسطينية‬By: Diyam Fuad Akra

Committee:

Signature

Dr. Mustafa Jarrar (Chairman)………………..

Dr. Nizar Habash (Member)………………...

Dr. Mahdi Arar (Member)………………..

Dr. Abualseoud Hanani (Member)………………..

‫االهداء‬‫إليك يا نور الحياة‪ ،‬لوالك ما كنت وما كان هذا االنجاز‪ ،‬معك أمشي بخطوات ثابتة‬‫َ‬‫وبثقة تطاول عنان السماء‪ ،‬إليك أبي الحبيب‬‫إليكِ يا سر البسمة وفرحة العمر‪ ،‬يا من صنعتني على عينك فدونك أنا ال شيء‪ ،‬إليكِ أمي‬‫الحبيبة‪.‬‬‫إليك شادي‪.‬‬‫إلى شريك العمر‪ ،‬إلى أيامي القادمة ‪َ ،‬‬‫‪I‬‬Acknowledgment
Special thanks to my supervisor Dr. Mustafa Jarrar for his great supportand valuable discussions and feedback.
I would like to also thank Dr. Nizar Habash for his valuable ideas during
this research, and the other committee members Dr. Mahdi Arar and Dr.
Abualseoud Hanani for their valuable comments.
Special thanks to my colleagues at Sina Institute, specially Faeq
Alrimawi and Nasser Zalmout, for their input, support, and feedback.
Many special thanks to Faisal Alshargi and Owen Rambow and Ramy
Eskander, from Columbia University, for their cooperation and forproviding me with tools to support this research.
Special thanks to my sisters and my brothers, for helping in many ways.
This research is part of the Cuuras project, at Sina Institute, which isfunded by the Ministry of Higher Education – Scientific ResearchCouncil.

II

Abstract
The number of users of the internet is increasing exponentiallyevery year; most of these users are using social networks, blogs andforums. When users in the Arab world need communicate with eachother, they often use their colloquial dialect Arabic instead of the Modern
Standard Arabic (MSA). To retrieve information about a specific topic,
Modern Standard Arabic (MSA) Informational Retrieval will retrieve
only MSA relevant data, but maybe there is helpful information
published in Dialect Arabic, also if we need to directly translate from
Dialect to English, this may be done by translating the Dialect to standard
Arabic then to English.
This thesis aims to build an annotated corpus for Palestinian Dialectwith Relevant Meta Data. The proposed methodology includes studying
linguistic facts about Palestinian dialect and comparing it with Modern
Standard Arabic in terms of morphology, orthography and lexical. As
well as collecting Palestinian written text from different resources, thenanalyzing and annotating the corpus by using resources designed forIIIEgyptian dialect, after that annotating manually a list of words that can’t
be analyzed by existing resources, finally start using the existing
annotation tool to double check over annotated corpus.

IV‫ملخص‬‫يتزايد استخدام االنترنت ووسائل التواصل االجتماعي بشكل ملحوظ يوما بعد يوم‪،‬‬‫وكما هو معروف فإن الناس عادة تستخدم العاميات في حياتها اليومية بدال من اللغة الفصحى‪،‬‬‫وبالضرورة فإن هذه العادة انتقلت للعالم االفتراضي و وسائل التواصل االجتماعي‪ ،‬ومن هنا‬‫يتبادر إلى أذهاننا التساؤل التالي‪ :‬ماذا لو أردت أن ابحث عن كلمة أو معلومة معينة عبر‬‫االنترنت‪ ،‬هل أستطيع أن استرجع ما نشر حول هذه الكلمة أو المعلومة بالعامية أيضا‪ ،‬وماذا لو‬‫خطر ببالنا أننا نود الترجمة من العامية الى االنجليزية مباشرة مثال‪.‬‬‫من هذه التساؤالت برزت الفكرة المقدمة في هذه الرسالة أال وهي بناء مدونة للعامية‬‫الفلسطينية تتضمن العديد من الكلمات الفلسطينية موسومة بمجموعة من الصفات النحوية‬‫والمعجمية مثل أصل الكلمة والسوابق واللواحق وتصنيف الكلمة(فعل‪،‬اسم‪...‬الخ) ومعناها‬‫ومقابلها في اللغة العربية الفصحى‪.‬‬‫تمت عملية بناء المدونة من خالل عدة مراحل وهي ‪ :‬دراسة الفروقات النحوية والشكلية‬‫والمعجمية بين العامية الفلسطينية واللغة العربية الفصحى ومن ثم جمع مادة فلسطينية مكتوبة‬‫من مصادر مختلفة‪ ،‬ثم استخدام الوسائل التي صممت للعامية المصرية لتحليل النص‬‫الفلسطيني‪ ،‬وأخيرا توسيم ما لم يتم تحليله من قبل الوسائل المصرية يدويا واستخدام برنامج‬‫صمم لتسهيل عملية التوسيم لتدقيق ما تم تحليله من قبل الوسائل المصرية‪.‬‬‫‪V‬‬

Table of Contents‫ االهداء‬......................................................................................................................... I
Acknowledgment ..................................................................................................... II
Abstract ................................................................................................................... III‫ ملخص‬........................................................................................................................ V
Introduction ............................................................................................................... 1
1.1

Scope and Motivation .................................................................................. 1

1.2

Problem Statement and Thesis Goals ........................................................... 9

1.3

Summary of Contributions .......................................................................... 9

1.4

Summary of Structural Work ..................................................................... 10

Literature Review .................................................................................................... 12
2.1

Corpus Collection and Annotation ............................................................. 12

2.1.1 Development of a pilot Levantine Arabic Treebank (LATB) of Jordanian Arabic ... 122.1.2 Arabic Dialects Parser ......................................................................................... 14
2.1.3 Cross Lingual Arabic Blog Alerts (COLABA) .......................................................... 18

2.2 Dialectal Orthography.................................................................................... 25
2.3 Dialectal Morphological Annotation .............................................................. 31
2.3.1 MADAMIRA ........................................................................................................ 32

2.4 Remarks on the related works and literature ................................................... 38
Corpus Collection.................................................................................................... 40
3.1

Corpus....................................................................................................... 40

3.1.1 Corpus collection methodology .......................................................................... 40
3.1.2 Corpus Statistics ................................................................................................. 41

3.2

Processing and Storing collected text ......................................................... 50

3.2.1 Storing methodology .......................................................................................... 50

VI

3.2.2 Storing Schema Statistics .................................................................................... 55

Corpus Annotation .................................................................................................. 57
4.1

Annotation Methodology........................................................................... 57

4.2 Annotation process ........................................................................................ 64
4.2.1 Using MADAMIRA ............................................................................................... 64
4.4.2 Manual Annotation............................................................................................. 67
4.2.3 Correct errors and fill in gaps using DIWAN......................................................... 68

4.3 Discussion ..................................................................................................... 70
4.4 Evaluation ..................................................................................................... 72
Conclusion and Future Work ................................................................................... 75
References ............................................................................................................... 77
Appendices.............................................................................................................. 83
Appendix #1 ............................................................................................................ 84VIIList of Figures
Figure ‎2.1: Morphological/Part-of-speech/gloss (MPG) tagging [2]………………..13
Figure ‎2.2: “‫ ”انا مش عارفة‬sentence Treebank [2]……………………………………..13
Figure ‎3.3: “‫ ”رايح اشترك فيه‬sentence Treebank [2] .................................................... 14
Figure ‎2.4:Sample of Levantine-MSA dictionary [1] ............................................... 15
Figure ‎2.5: tree pair from MSA-dialect synchronous grammar [1][3] ....................... 18
Figure ‎2.6: Sample of Dialect Arabic queries [5] ..................................................... 19
Figure ‎2.7: Typo identification and Fixing interface [4] ........................................... 21
Figure ‎2.8: Specifying word dialect interface [4] ..................................................... 22
Figure ‎2.9: Lemma creation interface [4] ................................................................. 23
Figure ‎2.10: Morphological Profile creation interface [4] ........................................ 23
Figure ‎2.11: EGY CODA Example[8] ..................................................................... 30
Figure ‎2.12: TUN CODA Example [9] ................................................................... 30
Figure ‎2.13: MADA output sample[15]……………………………………………..33
Figure ‎2.14: MADAMIRA architecture[16]............................................................. 36
Figure ‎2.15: MADAMIRA Demo online interface[16] ............................................ 37
Figure 3.1: Sample from Facebook Document………..…….....................................42
Figure 3.2: Sample from Twitter Document………………………...........................43
Figure 3.3: Sample from "‫ "عبد الحميد عبد العاطي‬blog………….....................................44
Figure 3.4: Sample from Palestinian Stories…………………..................................45
Figure 3.5 : Sample from Forum ……………………………...................................46

VIII

Figure 3.6: Sample from Palestinian terms….…………………..............................46
Figure 3.7 : one of "‫ "وطن ع وتر‬episodes……………………...................................47
Figure 3.8 : Sample from Dictionary…………..………….....................................48
Figure 3.9 : Sample 2 from Dictionary……………………….................................49‎Figure 3.10 : Sample 3 from Dictionary……………………………………..….…49
Figure ‎3.11: Database ER model…………………………………………….…….51
Figure 3.12: Document Table………………………..……………….…………....52
Figure ‎3.13: Palestinian Dialect words Table………………………………….….53
Figure ‎3.14: Bigram Palestinian Dialect words Table……...……………….…….54
Figure ‎3.15: Trigram Palestinian Dialect words Table………………………...….54
Figure ‎3.16: Fourgram Palestinian Dialect words Table………………………….55
Figure 4.1: Sample of MADAMIRA_EGY Result………………………………..68
Figure 4.2: Sample of Manual Annotated List…………………...………………..68
Figure 4.3: DIWAN Interface…………………...…………………….…….…….69
Figure ‎4.4: DIWAN Output Sample……………………………………………...70
Figure 4.5 : Annotators differences Sample ………………………………………74

IX

List of Tables
Table 3.1: Data base tables’ statistics ...................................................................... 55
Table 4.1: Sample of Exceptional Palestinian words .............................................. 60
Table 4.2: Annotation Specification Example .......................................................... 63
Table 4.3:MADAMIRA EGY Result ....................................................................... 65
Table 4.4: MADAMIRA MSA Result .................................................................... 65
Table 4.5: experiment result .................................................................................... 65
Table 4.6: inter-annotator agreement result………………………………………….73

X

Chapter One

Introduction

This chapter gives a quick introduction to the thesis.
Section 1.1 presents the scope and the motivation of ourresearch. The problem statement and the thesis goals arepresented in section 1.2. Section 1.3 summarizes our main
contributions, and section 1.4 gives a structural overview
of the thesis.

1.1 Scope and Motivation
The need for processing the Arabic language texts recently became an
important issue due to the necessity for many application types, such as1

search engines, spelling checker, morphological analyses, machinetranslation, among others.
The Arabic language is spoken by 300 million people all over the world[4]. Arabic language has two forms: Modern Standard Arabic (MSA) andDialectal Arabic; MSA is the superordinate of language which is usedparticularly -the in media, education, and formal communications. MSA
is not used in daily life expressions; it is also more written rather thanspoken. On the other hand, Dialectal Arabic is an informal colloquial,which is used mainly in daily communications among Arabs. Besides
that, it is a spoken language rather than written language. So the term
dialect is used to describe differences in speech that are associated withdifferent regions or different social groups, as you wander around, you
can find variations in speech that are associated with their place ofresidence (urban or rural). In general, Dialectal Arabic is classified into
five main categories [4]: Egyptian Dialect, which is used in Egypt andSudan, Gulf Arabic, which is used in Gulf countries, Moroccan Arabic,
which is used in Western Arab countries (Morocco, Libya, Tunisia,
Algerian), Iraqi Arabic, which is used in Iraq, and ultimately Levantine

2

Arabic which is used in Levant Countries (Lebanon, Syria, Jordan, andPalestine).
Dialectal Arabic is found as a verbal language among people only, andrecently in these days Arab peoples start using their dialectal Arabic in
web area. The emergence of social networks, blogs and forums, users
tend to use the Arabic dialect language in their communications,
therefore, the need for processing the Arabic dialect is getting urgently
more important day by day.
The need to deal with these Arabic Dialectal data is becoming
significantly important for the upcoming uses. However, using Natural
Language Processing (NLP) tools which are basically designed forStandard Arabic to deal with Dialectal Arabic will not solve the problem,
but it will lead to lower performance due to the fact that [4]:x Standard

Arabichasdifferent

morphological,

lexical,orthographical, and phonological aspects rather than dialectal
Arabic;3

x Dialectal Arabic is less controlled; because it is used by people
in daily life, which means that there are no standard
orthographic rules for it.
x Lack of annotated dialectal Arabic corpus and resources.
Our Next Step is to explore some differences between MSA and dialects.
Arabic Variants
The forms of Standard Arabic and its dialects are called Arabic Variants
and classified in [6, 10, 11, 12, and 13] as follows:
Phonological level variants, which is the sound of character; Arabic
people utter same character in different sounds. For example: In Standard
Arabic, "‫ ”ق‬is varied between Arabic dialects such as in Levantine is
replaced by “‫”أ‬, “‫”ك‬, “,‫ ;"ﭽ‬in Egyptian is replaced by “‫”أ‬, “,‫ "ﭽ‬in Gulf is
replaced by “.‫ "ﭽ‬This variant is coming over phonological level. Anotherphonological level is “‫ ”ث‬which becomes as "‫"ت‬,”‫ ”س‬in the Levantine
and Egyptian; "‫ "ذ‬which has variants ”‫”ز‬,”‫ ”ج“ ;”د‬which has variants“‫"ﭽ‬,”‫”ي‬.

4

Orthographic level variant, which is the way of writing/spelling acharacter. Arabic people write the same character in different spellings.
Same phonological level variants also occur in orthographic level
because each phoneme may be written in different forms as it is spoken.
Another issue in orthographic level is lack of standard of orthographic, so
as an example the sentence “‫ ”لم أقل لك‬in standard Arabic may be written
in Levantine Arabic as follows: “‫”ما بقلكش‬, “‫”م بقلكش‬, “‫”مبقلكش‬, “‫”ما بألكش‬,“‫”م بألكش‬, “‫”مبألكش‬,”‫”ما بكلكش‬,”‫”م بكلكش‬, “‫”مبكلكش‬. Also, sometimes people
write the same word in different forms, as an example “‫”الضو‬, “‫”الضوء‬.
Lexicon variants, which meaning that lexical term between the variants
is completely different, examples of lexical variants: The word “ ‫ “فقط‬inStandard Arabic is replaced by the word “‫ ”بس‬in Levantine Arabic. The
word “‫ ”اريد‬in Standard Arabic is replaced by the word “‫ ”بدي‬in Levantine
Arabic, “‫ ”عاوز‬in Egyptian Arabic, “‫ ”ابي‬in Gulf Arabic, “‫ ”بغيت‬in
Moroccan Arabic, and “‫ ”اريد‬in Iraqi Arabic. Reference names, as “‫”هؤالء‬becomes “‫ ”هدول‬in Levantine Arabic.

5

Morphological level variants, which meaning that there is a difference inthe morphology of the same word, examples of morphological variants:
present progressive is different between Arabic variants, as an example
the verb “‫ ”يلعب‬in Standard Arabic is replaced by “ ‫ “بيلعب‬in the Levantine
and Egyptian Arabic, and “‫ ”ديلعب‬In Iraqi Arabic, and “‫ ”كيلعب‬in
Moroccan Arabic. Future also differs, as an example “‫ ”سيلعب‬in Standard
Arabic became “‫ ”حيلعب‬in Levantine, “‫ ”هيلعب‬in Egyptian, “‫ ”رح يلعب‬in
Iraqi, and “‫ ”غيلعب‬in Moroccan.. Furthermore, pronouns are different
among Arabic variants, such as “‫ ”كم‬in MSA may be replaced by “‫ ”كوا‬In
Levantine.
Sentence structure level variants, which means that the structure forsame sentence differs, examples of syntactic variants: “ ‫ ”شركة محمد‬in
MSA sometimes becomes “‫ ”الشركة تبعت محمد‬in Levantine; word order, as“‫ ”هذا الولد‬becomes “‫ ”الولد دا‬as in Egyptian Dialect. MSA Negations forms
such as “‫ ”لم‬which may replaced by “‫ ”ما‬before a verb and “‫ ”ش‬at the endof the verb; Example: “‫ ”لم أقل لك‬in MSA equivalent to “‫ ”ما قلتلكش‬in
Levantine.

6

So, because of these Arabic variants using existing MSA NLP tools for
Arabic dialects may lead low accuracy results [1,14]. This becomes aproblematic issue and there is a need for tools that have the ability to deal
with dialectal Arabic. So from here comes the importance of building anannotated corpus of Palestinian dialect, which we are going to work on
this thesis.
The importance of building an annotated corpus considers as thefoundation stone of many applications, some of them are as follows:x Information Retrieval and Extraction applications, there aremany applications, which are responsible for retrieving or
extracting information about something. Building annotatedcorpus will help these applications to go through dialectal
Arabic data not only over Standard Arabic.
x Search engines, through which users can search the required
things using dialectal Arabic words, and using an annotatedcorpus to get its equivalent from Standard Arabic.

7

x Machine Translation, by converting Dialectal Arabic words to
its equivalent or synonym Standard Arabic then translate themto the corresponding languages.
x AutoComplete with possible equivalent Standard Arabic word;
In other words, if a user starts entering some Dialectal Arabic
word, then -an application starts giving him/her possible
Standard Arabic word
x Part-Of-speech tagging applications.
x Dialectal Arabic Parsers.
According to the variants for Arabic, which are discussed above, there isa need to deal with each Arabic language dialect separately, for thisreason, this thesis will focus on the Palestinian dialect, Palestinian dialect
which is considered a sub-dialect of Levantine Arabic.

8

1.2 Problem Statement and Thesis Goals
This thesis aims to collect a corpus of written Palestinian dialect andannotate it with relevant metadata. More specifically, we did this by:
x Collecting Palestinian dialect texts manually from different resources
such as blogs, forums, social networks, dialect dictionaries, and TV
series.
x Annotating each word in the corpus manually using the DIWAN tool,
which is linked with the Egyptian MADAMIRA [15, 16]. This also
includes writing each word in the CODA (Conventional Orthography
for Dialectal Arabic) standard [8, 17].

1.3 Summary of Contributions
Our contributions can be summarized as follows:x A corpus of Palestinian dialect text (43K words, 15K of them unique)was manually collected from different resources, like blogs, forums,
Facebook, Twitter, books and Palestinian series.

9x The corpus was parsed automatically and stored in a proper database
schema.
x A list of 1234 unique corpus words is annotated manually.
x 18743 words (with 500 unique words) from this corpus were selected
and fully annotated using the DIWAN tool. The POS of each word
was specified, as well as it was written according to the CODA rules[8,17]. The rest of the corpus will be future work.
x It is worth noting that this work was done in cooperation withresearchers from the Curras project, and a joint paper [20] waspublished about it (see appendix 1), as well as a technical report [17].
The achievements listed above are our own contributions, butfollowing the project’s methodology, tools, and resources.

1.4 Summary of Structural Work
Chapter Two presents the literature review of related work done in thisarea. A special focus given to work done on conventional orthography
for dialectal Arabic (CODA) [8,9]. The work was done on developing
morphological analyzer for Arabic and its dialects (MADAMIRA)

10[15,16,18,19]. The work was done on developing web application for
Dialectal Arabic Text annotation [5]. Finally The work was done on
Parsing Arabic Dialects [1,2,3,4].
Chapter Three described our corpus. It will cover our methodology that
we followed to collect our corpus, it covers corpus statistics such as typeof documents, number of documents, number of threads in eachdocument, and number of words in each document.
Chapter Four presented approaches, methods and tools used in the
annotation process. It will cover annotation concept, annotation
methodology which includes a list of meta data that each word will beannotated with it. The benefits of using MADAMIRA in our annotation
process, a sample of our manual annotation list, and finally it covers how
DIWAN tool will speed up the annotation process.
Chapter Five This chapter discussed the results we -have gotten until
now, and it also presented the future work that can be done in this area.
And finally covered using of DIWAN tool to complete our annotatedcorpus.

11

Chapter 2

Literature Review

This chapter gives a quick review over works done on Arabic
Dialects. Section 2.1, presents work done on corpus
collection and annotation. Section 2.2, presents work done ondialectal orthography. Section 2.3, presents work done ondialectal morphological annotation.

2.1

Corpus Collection and Annotation

2.1.1 Development of a pilot Levantine Arabic Treebank (LATB) of
Jordanian Arabic

Levantine Arabic Treebank (LATB) contains 33,000 words and it is built
for development and testing purposes. LATB is built using Jordanian
Levantine Conversational Telephone speech.

12Developing Levantine Arabic Treebank approach begins by analyzing
corpus data morphologically; so for each word in corpus generates
manually a tag called Morphological/Part-of-speech/gloss (MPG) tag.
Figure 2.1 shows an example of this tagging.

Figure 2.1: Morphological/Part-of-speech/gloss (MPG) tagging [2]Then analyzing corpus data syntactically; which is done manually andraises many issues such as detecting conversational speech effects anddisfluencies, active Participles which may be adjectival or verbal,
according to context; Figures 2.2, and 2.3 show examples when it isadjectival, or verbal

Figure 2.2: “‫ ”انا مش عارفة‬sentence Treebank [2]13Figure 2.3: “‫ ”رايح اشترك فيه‬sentence Treebank [2]After developing Levantine Treebank, the authors apply automatic tree
tool over it which are: Cat-tree which is responsible for detecting
inconsistency and unwell-formed sentences and it separates multiple
trees, Clean-trees which are responsible for removing resulting null
sentences (unfinished sentences) and null constituents, Tregex andTsurgeon which are responsible for traversing constituents and eithertransform it or remove it. Applying these tools reduced Levantine Arabic
Treebank from 6639 trees to 3979 trees.
2.1.2 Arabic Dialects Parser
In [1] and [3] authors’ presented three approaches to building aparser for Arabic Dialects, but before starting work on these approaches,
the authors need to prepare linguistic resources that will be used in theseapproaches. The first linguistic resource is Levantine-MSA dictionary,
this dictionary will be used to translate Levantine sentence to Standard

14

Arabic sentence and to convert Standard Arabic Treebank to Levantine
Treebank. Each dictionary entry contains Levantine word with its Partof-Speech, equivalent MSA word and English gloss as shown in Figure
2.4. The dictionary is relatively small. It contains 2201 words.

Figure 2.4: Sample of Levantine-MSA dictionary [1]The Second linguistic resource is part-of-speech tagging for corpus data,
and the basic idea here is to assume Standard Arabic (MSA) tagger as abaseline then make adoption on it. So the first step is to run the MSA
tagger over Levantine data, the accuracy of applying MSA tagger overLevantine was 69%. According to accuracy result a number of adaption
making on MSA tagger were added, which are: Adoption using
probabilities and normalization after applying this step the accuracy of

15

MSA tagger became 73%, replace Levantine words that have an entry inthe dictionary presented previously with MSA word, and run the MSA
tagger another time then the accuracy became 80%, Manually tagged partof Levantine corpus, and used the tagged set as training set, the accuracy
after applying this step reached to 80%.
After preparing linguistic resources, the authors started applying parsing
approaches. The first approach is sentence transduction which depends
on translating Levantine sentence to MSA sentence, then parsed the MSA
translated sentence using MSA parser. This approach has many
disadvantages such as lack of resources for translation from Levantine
Arabic to MSA, and sometimes two words in Levantine Arabic areTranslated to the same word in MSA ‫مين تترجم إلى ِمن و َمن“نفس‬-‫مثل مِن‬“‫الكلمة‬. Another approach is Treebank Transduction which depends on
converting Standard Arabic Treebank into Levantine-Like Treebank
using linguistic knowledge, then train a statistical parser on converted
Treebank, then test parsing performance over new Levantine corpus data.
There are a lot of transformations that were done over Standard Arabic
Treebank to convert it to Levantine-like Treebank such as unifying two

16

blocks with the same meaning in one and separating nested node,replacing all negations in Arabic Treebank with Levantine negation
forms (“‫ ”مش\ما‬precedes verb, “‫ ”ش‬after verb), replacing (Verb-SubjectObject) order in Standard Arabic Treebank with Levantine order which is(Subject-Verb-Object), replacing (demonstrative pronoun – noun) orderin Standard Arabic Treebank with Levantine order which is (noundemonstrative pronoun), replacing every lexical in Arabic Treebank with
its equivalent in Levantine using dictionary that described above,
replacing every “‫ ”احتاج\اريد‬verb and all derivatives in Arabic Treebank
with “‫ ”بد‬and its derivatives, adding prefix “‫ ”ب‬to every verb that has“VBP” tag, Converting “‫ ”ليس‬in Arabic Treebank to particle.
Final approach used in parsing Arabic dialects is Grammar transduction
which depends on developing an MSA-dialect synchronous grammar.
This grammar contains pairs of elementary trees which combine MSA
elementary tree with corresponding Levantine elementary tree. Figure 2.5
shows a tree pair from this grammar.

17

Figure 2.5: Tree pair from MSA-dialect synchronous grammar [1] [3]This grammar developed by extracting Standard Arabic elementary trees
from Arabic Treebank then translates it using handwritten rules tocorresponding Levantine elementary trees.
2.1.3 Cross Lingual Arabic Blog Alerts (COLABA)
In this section, we present COLABA (Cross Lingual Arabic Blog
Alerts) project, which is done by Diab, Habash, Rambow, Altantawy, andBenajiba in [5], and aims to processing Arabic social data on the web,through COLABA project many tools were developed in order to achieve
COLABA goal which is to retrieve all dialect Arabic Blogs data and all
MSA blogs data that are related to the required MSA query.
COLABA goal is achieved by applying many stages; In the first stage,
COLABA project asked 25 annotators to design a dialect query that is18

responsible for harvesting large amount of dialectal data from the web,annotators designed 40 queries with its dialect, corresponding MSA
translation, and English translation. Figure 2.6 shows a sample of these
dialect queries.

Figure 2.6: Sample of Dialect Arabic queries [5]Then COLABA Project goes through annotation stage, which is
responsible for removing HTML markup, spam, advertisements,
encoding issues, and every Meta data from blogs data that's retrieved in
the first stage.
After that, COLABA project gives each blog initial rank according to its
degree of dialectness. This was done through a simple module called
Dialect Identification pipeline which is responsible for determining thedegree to which a text includes Dialect Arabic words. It works asfollows: it takes an input text, and then analyzes each word in the input
text by Buckwalter MSA morphological analyzer (BAMA), if BAMA

19

returns a result; then a word is MSA. Otherwise, it is potentially a dialect
word, and then it gives rank to a blog according to the number of words
that aren’t classified by BAMA.
Then COLABA project Applies typographical clean up over blogs withhigh ranking. This is done by correcting every MSA word with nonstandard orthography by writing it in standard orthography, remove
speech effects from words, add missing spaces between words, and thenapplies Dialect Identification pipeline another time.
Then COLABA project applies COLABA Conventional Orthography(CCO), which is responsible for providing orthography for each word in
Dialects, as an example the word “‫ ”باب‬has Levantine orthography whichis “be:b”, and Egyptian orthography which is “ba:b”.
After that, COLABA project applies Dialect annotation over highly
ranked blogs, this step is done using a COLANN_GUI web application
which was presented by Benajiba and Diab in [4], and it is a web
application used by annotators to annotate text. It is browser independent,
it uses PHP to interact with a server database and JavaScript for GUI, it
contains three types of users: Annotator which is responsible for20

annotation, Lead Annotator which is responsible for assign annotation
tasks to annotators, Super User Which is the administrator of the system.
Annotation process using COLANN_GUI passes through many steps.
The first step, if there is a misspelling in a word, an annotator mustcorrect it, and if there is also a speech effect, annotators must remove it
as well remove the missing spaces, then the annotator must do sentenceboundary detection. Figure 2.7 shows the interface that annotators usedin this step.

Figure 2.7: Typo identification and Fixing interface [4]Second, the annotator is asked to choose the dialect of a word, level of
dialectalness, and enters phonetic transcription of the word. Figure 2.8
shows the interface that annotators used it in this step.

21

Figure 2.8: Specifying word dialect interface [4]Third; annotators must enter the underlying lemma form of each word(derived from), in this step the application provides an annotator with a
Dialect Arabic word and a list of usage examples from blogs data, thenasks an annotator to provide a corresponding lemma, MSA equivalent,
English equivalent, dialect Id for each word in usage examples, andassociate each lemma with its usage example. Figure 2.9 shows theinterface that annotators used in this step.

22

Figure 2.9: Lemma creation interface [4]Finally; the application asks annotators to enter POS tag for it for eachlemma created in the previous step. Figure 2.10 shows the interface thatannotators used in this step.

Figure 2.10: Morphological Profile creation interface [4]23All information entered by annotators using the application is stored in
the database. This database contains 22 relational tables that are
responsible for saving basic information, annotation information,
assignment information, user permissions information, and user
connection information.
The Final Stage in COLABA project is using DIRA (Dialectal
Information Retrieval for Arabic) which is responsible for retrieving all
MSA and dialect data for a word, DIRA works as follows: It takes a verb
as input, then generates three surface forms for it; these surface forms
are: MSA inflected forms, as example of the verb “‫”أصبح‬, it generates“‫”يصبح‬, “‫”سيصبح‬,”‫… ”أصبحنا‬etc, MSA with dialectal morphemes, as
example for verb “‫”أصبح‬, it also generates “‫”بيصبح‬.”‫”حيصبح‬,”‫”هيصبح‬,“‫…”حيصبحوا‬etc, MSA lemma is translated to Dialect lemma, as exampleof the word “‫”أصبح‬, it translated to “‫”بقى‬with all morphemes such as“‫”حيبقى‬, “‫”بيبقى‬, Then DIRA re-injected all generated forms into original
query to retrieve Dialectal data.

24

2.2 Dialectal Orthography
This section reviews works done on building Conventional Orthography
for Dialectal Arabic (CODA) [8,9, 17]. The most important problem thathas been noted in all works done under Dialectal Arabic area is the lackof conventional orthography; in Arabic dialect, there are many ways towriting the same words, and there are also a lot of commons that areshared among all Arabic dialects. So there is a need for writing in astandard way over all Arabic dialects. From these issues come the needs
for CODA, which is a conventional orthography for dialectal Arabic.
CODA aimed to satisfy many Goals such as; consistent and coherent
standards for writing Arabic Dialects, build for computational purposes,
unified framework for writing all Arabic Dialects, and Save a level of
uniqueness over each dialect. CODA builds conventions according tosimilarities between Modern Standard Arabic and its dialect.
In order to achieve these goals, CODA teams made many design
decisions such as; using Arabic Script, unique orthographic form for eachdialect; this form represents dialect phonology, morphology, and lexical25

semantics, Orthographic decisions are displayed in MSA-like format
such as spelling the Definite article “‫ ”ال‬morphologically.
Generally speaking, CODA tries to preserve phonology as it in word; e.g.
‫ اتكتب‬، ‫ كتاب‬،‫ راجل‬, but there is some exceptions, such as a word that
contains /q/ which differs between dialects, then the word is written in
MSA form; e.g.: ‫قصر‬, and if the word in dialect has short vowel thatcomes from long vowel in MSA then it is written in MSA Form; such as‫ طبور‬will be written ‫طابور‬.
CODA also tries to preserve the morphology as it in word, But there is an
exception here if there is a negation or pronoun that comes with a verb
CODA separates it from the word; e.g. ‫ ما كتبتلوش‬will write ‫ ما كتبت لوش‬in
CODA.
Also CODA team decides to keep the word order in a sentence as in theoriginal sentence, to develop CODA in a way that is easy to learn andwrite, and to build the unique CODA Map for each dialect that contains
rules and exceptions.

26

Until now, CODA has been applied for Egyptian Dialect (EGY) andTunisian dialect (TUN). So, according to the CODA design principles aword is written as pronounced unless there is an exception on; thisexception maybe:
x Morphological exception; in EGY there are many morphological
exceptions, such as; in noun and adjectives‫ ة‬CODA teams

decided to -writ as ‫ ة‬not ,‫ه‬and if the word has clitics then it
becomes either ‫ ت‬or ‫ا‬, ‫ ين‬suffix (dual & plural) does not vary
according to its case as MSA, add vowels for geminates in verbs
such as ‫ مديت‬not ‫مددت‬, the plural suffix (‫وا‬-‫ )توا‬spelled with silent
Alif in CODA, in affixes that relate to feminine, there must be an
extra ‫ ي‬at the end. e.g. ‫اكتبي‬if particles is single then according to
CODA it will be attached to the word such as ‫بالطول‬, also in CODA
pronominal pronouns and negations part must be attached at theend of word, ‫ ال التعريف‬written as is, in some cases we removed
some letters from the original word such as : ‫ال‬+‫ ل‬written as ‫ ة‬,‫لل‬
becomes ‫ ا‬or‫ ت‬e.g.

‫هم‬+‫ معلمة‬written as‫ معلمتهم‬or‫معلماهم‬according to context, ‫ ا‬in ‫ واو الجماعة‬is removed e.g. ‫ها‬+‫ كتبوا‬is27

written as ‫كتبوها‬, ‫ ى‬becomes ‫ ا‬or ‫ي‬, and finally there are dialectal
clitics such as ,progressive part ‫ ب‬this progressive part add tooriginal verb such as ‫اكتب‬+‫ ب‬written as ‫باكتب‬, Future part ‫ ح‬e.g.
such as ‫حامشي‬,Second person plural ‫ كو‬e.g. ‫شافوكو‬, and Feminine
clitics ‫ ي‬e.g. ‫شافوكي‬. Also in TUN there are many exceptions, suchas; interrogation proclitic ‫شي‬, e.g. ‫وشريتوهاشي‬, Negation part ‫ش‬,and Single litter colitis ‫ع‬, ‫م‬x Phonological exception, in EGY there are the following
phonological exceptions such as; CODA EGY team decided that
there are 9 consonants that spelled differently from theirphonology in DA if these constants have a dialect Arabic rootradical and its dialect Arabic root has MSA root ; these consonant
are :/‫ ء‬/ spelled /‫ق‬/ ; /‫س‬/-/‫ت‬/ spelled /‫ث‬/; /‫د‬/-/‫ز‬/ spelled /‫ذ‬/; /‫ظ‬/-/‫ز‬//‫د‬/-/‫ض‬/ spelled /‫ض‬/; /‫ز‬/-/‫ذ‬/-/‫ض‬/ spelled /‫ ظ‬/; /‫س‬/-/‫ص‬/ spelled /‫ص‬/ ;/‫ت‬/-/‫ط‬/ spelled /‫ط‬/, preserve long vowels as is such as “‫ كتبين‬which written in CODA as ‫”كاتبين‬, if there is multiple long vowels thatshorten to one in EGY; then CODA team decided to write twolong vowels as MSA; e.g. ‫ قنون‬is written as ‫ قانون‬in CODA,

28

different writing for ‫ي‬and ‫و‬in Arabic script according to context.
E.g. ‫دور‬that maybe /du:r/ or /do:r/ according to context it appears
on it, CODA team decided to keep Hamza as it; in other words if
peoples write word without Hamza or with Hamza CODA team
decided to be as is, for example if CODA team face “‫ ”مايل‬in text,
then will preserve it as is, and if they face “‫ ”مائل‬will also keep as
is -, and finally ‫ى‬،‫ ي‬in the end of words in CODA will be spelledcorrectly, as example”‫ ” الكالم ده علي مين‬will be written in CODA”‫”الكالم ده على مين‬. Also in TUN there are the following phonological
exceptions, such as same exception in 9 consonants as EGY with
two special cases in TUN which are /‫ق‬/, /‫غ‬/ spelled /‫ق‬/ as e.g.
/bagra/ is written in CODA as /baqrah/, and Consonants withmultiple punctuations are written in a form closes to MSA if there
is MSA, and Finally TUN CODA adds ‫ ن‬after some numerals insome cases such as ‫خمسطاشن راجل‬.
x Dialectal lexical exception, such as ‫برضه‬not ‫برضو‬, ‫ده‬not ‫ ذه‬in
EGY, and ‫هذاك‬not ‫هاذاكة‬, ‫عالسالمة‬not ‫عسالمة‬, and Foreign words

29

that use non-Arabic phoneme /g/, /v/ and /p/ will be written in
CODA as q, f and b. e.g. ‫ قازوز‬,‫ فيسته‬in TUN.
Figure 2.11 & 2.12 shows examples of EGY CODA and TUN CODA

Figure 2.11: EGY CODA Example [8]Figure 2.12: TUN CODA Example [9]30So as you have noticed in Figures 2.11 & 2.12, the first part of the figurecontains text as appeared; in the second part, the text rewritten in CODA;
the last part of the figure contains the corresponding English text.
CODA guidelines will be extended to cover PAL in this thesis, as
discussed in chapter 4.

2.3 Dialectal Morphological Annotation
Most of the previous work that is done under the area of Morphology inArabic focused on MSA, but in Dialects the works are relatively small or
it depends on MSA to deal with Dialects. But as stated previously
available MSA tools cannot be easily extended or transferred to work
properly for Dialects according to varying between MSA and its Dialects.
So it is important to develop annotated and morpheme-segmented
resources and morphological analysis tools to deal with Dialects. One of
the most recent contributions that deals with Dialects is CALIMA which
is a morphological analyzer for EGY [19]. CALIMA and MSA analyzer
SAMA are also used in EGY morphological tagger MADA-ARZ and in
MADAMIRA.

31

2.3.1 MADAMIRAMADAMIRA is a morphological analyzer for Arabic and its dialect,
MADAMIRA combines two systems for Arabic processing. Thesesystems are:x MADAA morphological analysis and disambiguation for Arabic (MADA)
depends on presenting many analyses for each word, then selecting thesuitable analysis according to context. In order to do this, MADA
contains 19 orthogonal features that suitable analysis is selected depends
on it. These features are: Part_Of_Speech(POS/pos); such as Noun, Verb,…etc, presence of conjunction(CNJ/conj); such as w, and f, presence of aparticle clitics(PRT/part); such as b, k, and l, presence of pronominal
clitics(PRO/clitic); such as object, and possessive, presence of definite
article(DET/art) ; such as Al, gender(GEN/gen) ; such as FEM, and
MASC, number(NUM/num) ; such as SG ,DU, and PL, Person(PER/per); such as 1,2, and 3, voice(VOX/voice) ; such as PASS, and ACT,
Aspect(ASP/aspect) ; such as IV, CV, and PV, mood(MOD/mood) ; such
as I, S, J, and SJ, presence of nunation (NUN/def) ; such as DEF, and32

INDEF, construct state(CON/idafa) ; such as POSS, and NOPOSS, andFinally case(CAS/case) ; such as ACC, GEN, and NOM
Each of these features is weighted. So when a new word comes, MADA
provides a list of potential analyses by usinga Buckwalter Arabic

morphological analyzer (BAMA), and then the analysis that gains mostof the features will be selected by MADA. Figure 2.13 shows a sample of
MADA outputs

Figure 2.13: MADA output sample [15]Then MADA was extended to MADA ARZ which is the Egyptianversion of MADA, the main major changes in this extension occurred inmorphological analyzer used. In MADA ARZ, the morphological33

analyzer that used is CALIMA; CALIMA is a Morphological analyzer
for Egyptian Arabic;
Building CALIMA was passed through many stages. Firstly, choose
Egyptian Colloquial Arabic Lexicon (ECAL) which contains 27K verbs,
36K nouns and adjectives, 1.5K proper nouns, and 1K closed class; Each
ECAL entry consists of phonological form, undiacritized orthography,
lemma, and morphological features, then diacritrize each ECAL entry,
then write rules for converting from diacritized form to CODA and rules
for converting from ECAL morphology to LDC EGY POS tags, thenupdate lemma according to lemma Specification in SAMA, after thateach mapped ECAL entry was converted to SAMA-like representation.
Applying approach described above generates six tabled for CALIMA.
These tables are: complex prefixes, complex suffixes, complex stem,
prefix-stem, prefix-suffix, and stem suffix and each table was also
extended to contain non CODA variants.
CALIMA has 100K stem corresponding to 36K lemmas, 2421 complex
prefixes, 1179 complex suffixes.

34

x AMIRAAMIRA is a tool that is used for processing Arabic; it is based onsupervised learning with no dependency on any morphologicalknowledge, AMIRA toolkit contains: Clitic token (TOK), Part of Speech
Tagger (POS), and Base phrase chunker (BPC).
So when an input text is entered to MADAMIRA, it is cleaned andconverted to Buckwalter representation schema, then MADAMIRA
builds all possible analysis for each word using either SAMA analyzer
for MSA or CALIMA analyzer for EGY Arabic, after that MADAMIRA
gives predictions for the word’s morphological features and then gives
score for each word analysis and sorts the analyses according to thescore, after that MADAMIRA tokenizes the top score analysis according
to the schema requested by the user, then MADAMIRA divides the input
text into chunks, Finally MADAMIRA passes the input text to Named
Entity Recognizer in order to mark and categorize the named entities.
Figure2.14 shows MADAMIRA architecture

35

Figure 2.14: MADAMIRA architecture [16]In order to evaluate MADAMIRA ; authors choose a set of 25,000 words
for standard Arabic and 20,000 words for EGY Arabic then ask
MADAMIRA to analyze them and compare MADAMIRA results withgold annotation list ; the results are : the percentage of words that36

diacritized correctly in MADAMIAR is 86.3% for MSA and 83.2% for
EGY, the percentage of words that MADAMIRA gives its lemmacorrectly is 96.0% for MSA and 87.8% for EGY, the percentage ofwords that MADAMIRA gives its part-of-speech tag correctly is 95.9%
for MSA and 92.4% for EGY, and the percentage of words that
MADAMIRA gives all morphological features (match gold entry
exactly) correctly is 84.1% for MSA and 77.3% for EGY. Figure2.15
shows MADAMIRA interface

Figure 2.15: MADAMIRA Demo online interface [16]37

2.4 Remarks on the related works and literature
As we seen previously there are many works done in dialects but most ofthem have a lot of issues. These issues may be related to small size of
dialectal data as work presented in sub-section 2.1.1 which has only
33.000 words, subsection 2.1.2 which depends in too small dictionary,also numbers of queries that designed to collect data from web in
subsection 2,1,3 is relatively small because there are a huge data
published on the web and 40 queries couldn’t collect all of it.
Also, There is another issue comes from that some works were dependedon Standard Arabic to deal with dialects which didn’t lead to significant
results as work presented in subsection 2.1.2.
Also, all works presented in section 2.1 didn’t take into account the lackof orthographic and inconsistency in dialectal data.
These previous efforts start being on the train when CODA, and
MADAMIRA which presented in section 2.2 and 2.3 start, but also these
two work have many issues such as CODA need to be extended to cover
more dialects, also MADAMIRA need to be trained on more data in both38

MSA and EGY because sometimes it couldn’t give results for MSA
word.

39

Chapter 3

Corpus Collection

This chapter gives a detailed description about the corpus.
Section 3.1 presents corpus collection process and corpus
statistics. Section 3.2 presents corpus database

3.1 Corpus
3.1.1 Corpus collection methodology
As we stated previously, dialects data are founded as oral data, notas written data, so it is too difficult to find resources for written dialectal
content. Dialect data have also a lot of noise and inconsistency due to thelack of orthographic standards for dialects. Hence, the same word may bein different formats in dialects. So lack of resources and noise led to40

many challenges in the collection of high-coverage and high-accuracy
dialect corpora. We decided to focus on precision and variety more thanon size in our corpus, so when we collected our corpus, we tried to covera variety of topics and contexts, localities and sub-dialects, including thesocial class and gender of the speakers and writers [20].
3.1.2 Corpus Statistics
As we stated above, we are collecting our corpus manually from different
resources, and different context. The most important part of our corpus is
the famous Palestinian series “Watn E Watr ‫”وطن ع وتر‬. Our corpus
documents are:x Facebook
The text has been manually collected from different Palestinian pages on
Facebook. This is done by crawling many Palestinian Facebook pages
such

as:

‫بديش‬“‫أتجوز‬‫”يما‬(https://www.facebook.com/mombdesh2tjwaz?ref=br_tf);

page
this

page

contains several Palestinian jokes in different Palestinian sub-dialects anditcollected

in

June,

2013,

and

“‫للحوار‬‫فلسطين‬‫”شبكة‬page(https://www.facebook.com/paldf); this page contains several political41

discussions in Palestinian dialect. Total number of threads in thisdocument is 35, total number of word tokens are 3120 and total numberof word types is 1985. Figure 3.1 shows a sample of Facebook document.

Figure 3.1: Sample from Facebook documentx Twitter
The text has been manually collected from accounts of Palestinian
peoples who are tweet in Palestinian dialects; those peoples come fromdifferent sub-dialects. Accounts chosen here are reflecting different
culturesanddifferent

sub-dialects

such

as:

“‫الخطيب‬‫”مروة‬account(https://twitter.com/Marwa_101 ); this account is for girl from
umm-alfahm, “‫ ”هبة عبد السالم الحايك‬account(https://twitter.com/Hebahayek); this account is for girl from Gaza, and “Tamer Hammam” account42

(https://twitter.com/itamer83); this account is for young man from Gaza.
Total number of threads in this document is 38, total number of word
tokens is 3541 and total number of word types is 2133. Figure 3.2 showsa sample of Twitter document.

Figure 3.2: Sample from Twitter documentx Blogs
The text has been manually collected from “‫ ”عبد الحميد عبد العاطي‬blog, it
contains many of poets that have written in Palestinian dialect. Total
number of threads in this document is 37, total number of word tokens is8748 and total number of word types is 4454. Figure 3.3 shows a sample
of Blogs document

43

Figure 3.3: Sample from “‫ ”عبد الحميد عبد العاطي‬blog

x Palestinian Stories
The text has been manually collected from different forums, it contains
six stories that written in Palestinian dialect such as: “‫”قصة صبحية و الظبع‬(http://www.paledream.com/vb/showthread.php?t=16157 ); this story is
written in rural Palestinian sub-dialects, “‫”قصة جبينة‬(http://www.omaniyat.com/vb/showthread.php?t=21995 ); this story is
also written in rural Palestinian sub-dialects, “‫(”قصة سندريال‬http://forum.sedty.com/t428456.html ) ; which is written in rural
Palestinian dialect, “‫ ”قصة الملك و بناته الثالث‬which is written in standard
Palestinian dialect, “‫(”قصة الغرة و الجرة‬44

http://pulpit.alwatanvoice.com/articles/2012/05/10/260140.html) which is
written in Jaffa rural Palestinian dialect, and finally a dialogue between
number of Palestinian rural women. The total number of word tokens is2407 and total number of word types is 1422. Figure 3.4 shows a sampleof Stories document

Figure 3.4: Sample from Palestinian story

x Forums
The text has been manually collected from “‫”منتديات شبكة فلسطين للحوار‬(https://www.paldf.net/forum/index.php), it contains many of the
discussions in Palestinian Dialect. Total number of threads in thisdocument is 33, total number of word tokens is 1092 and total number ofword types is 798. Figure 3.5 shows a sample of forums document

45

Figure 3.5: Samples from Forumx Palestinian Terms
It is one document that contains 556 Palestinian terms with its meaning;these terms are collected manually from the web(https://www.paldf.net/forum/showthread.php?t=119188&page=2).
Figure 3.6 shows a sample from these terms.

Figure 3.6: Samples from Palestinian terms

46

x “‫ ”وطن ع وتر‬Series
Episodes of “‫ ”وطن ع وتر‬series, which was broadcast on the “‫”الفلسطينية‬channel, that have been obtained from those in charge of the series inthe“‫”مسرح حياة‬, and the idea in series based on a critique of the conditionsof Palestinian society in Comic way. Total number of episodes is 41,
total number of word tokens is 23423 and total number of word types is8459. Figure 3.7 shows one of “‫ ”وطن ع وتر‬episodes

Figure 3.7: one of “‫ ”وطن ع وتر‬episodes47

x Dictionary of Palestinian Vocabularies and Loan words” ‫معجم‬‫”العامي و الدخيل‬It contains Palestinian Dialect words from “‫ ”أ‬to “‫”ي‬, and for each word,
author was listed the meanings of a word, and where to use it, along with
its corresponding Standard Arabic

word,

it

also contains 53

morphological rules for Palestinian Dialect. Total number of pages in thisbook is 646, and total number of interpreted terms as ( ...‫ طوطح‬،‫ إجر‬، ‫معس‬‫ )الخ‬is 5595. The Author of Book is Hussien Ali Lubany, and Publisher of
The Book is Lebanon Library. Figures 3.8 -3.10 show samples from thedictionary

Figure 3.8: Sample from Dictionary48

Figure 3.9: Sample 2 from Dictionary

Figure 3.10: Sample 3 from Dictionary

49

3.2

Processing and Storing collected text

This section presents our work on parsing and storing collected texts in
N-gram models. This work is conducted at the beginning of the research,but it turned out that is not needed as DIWAN tool was used to annotate
the corpus. It is presented here as we think it is important when extending
our work to for example annotating phrases, rather than only words in the
corpus.
3.2.1 Storing methodology
Storing the corpus data is a preliminary step to achieve our goals, so
we retrieve and manipulate the dialect words easily. To maximize thedialect words that we could extract from the available resources we
decided to store the data using N-gram model (considering N [1-4]),
this - due to the fact that -people -do not only use- single words but alsophrases to express every things. We also decided to store the position of
the word in -the document; this position can be used later to generate anequivalent document in Modern Standard Arabic (MSA) after translating50

the dialect words into their correspondence MSA words. Figure 3.11
shows an ER model for corpus database

Figure 3.11: Database ER model51

As shown in Figure 3.12, “Document” table is used to store theinformation about each document in the corpus; those information –are:document name, document description, number of pages, and number ofwords.
Figure 3.12 shows a sample of data in Document Table.

Figure 3.12: Document Table

52

Tables (Palestinian_Dialect_words, Bigram_Palestinian_Dialect_words,
Trigram_Palestinian_Dialect_words,

andFourgram_Palestinian_Dialect_words) are used to store N-gram model;
each of these tables has: N-gram data (1, 2, 3, or 4 words), document thatN-gram data related to, paragraph that N-gram data located on it, andposition of N-gram data in the document. Figures 3.13-3.16 show
samples from these tables

Figure 3.13: Palestinian Dialect words Table

53

Figure 3.14: Bigram Palestinian Dialect words Table

Figure 3.15: Trigram Palestinian Dialect words Table

54

Figure 3.16: Fourgram Palestinian Dialect words Table

3.2.2 Storing Schema Statistics
Database statistics are shown in Table 3.2
Table Name

Contents

Document Table

50 documents

Palestinian_Dialect_words Table

43090 words

Dictionary of Palestinian

5595 terms

Vocabularies and Loan words” ‫معجم‬55

‫”العامي و الدخيل‬Bigram_Palestinian_Dialect_words

37043 pair of words

Table
Trigram_Palestinian_Dialect_words

32022 triple of words

Table
Fourgram_Palestinian_Dialect_words 27684 four of words
Table
Table 3.1: Data base tables statistics

Finally, The total Number of the distinct words in the corpus is 25402

56

Chapter 4

Corpus Annotation

This chapter gives a detailed description about annotation
approach followed in this thesis. Section 4.1 presents theannotation

specification.

Section

4.2

discusses

theannotation process that followed. Section 4.3 discusses theimportance of using annotation tool.

4.1 Annotation Methodology
Annotation is a process that aims to annotate each word with relevant
Meta data; these relevant Meta data are helpful in many applications suchas translation, morphological analyses; these Meta data -could be part-ofspeech tagging, stem, gloss, equivalent MSA… etc. The annotation will57

be more helpful if it is done in a context because the same word may
have different analysis and different meanings in different contexts.
So imagine that we have a word with its Metadata such as an equivalent
word in MSA then we can use it in application that translate from dialect
to MSA, on the other side we can use English meanings in applicationthat translate from Dialect to English. We followed the annotation
methodology provided in [20], which requires annotating every word in
the corpus with Metadata, such as prefix, suffix, stem, lemma, part-ofspeech, gloss and MSA lemma, defined in [20, 17] as the following:x Word: it is the row data as it appeared in the input data; and it isrepresented in Arabicx Word (Buckwalter): it is the row data as it appeared in the input
data; but it is represented in the Buckwalter transliteration [22].
Examples of Buckwalter transliteration: letter "‫ "ط‬is written as "T"
in Buckwalter, letter "‫ "ذ‬is written as "*", the letter "‫ "أ‬is written as">", and letter "‫ "ص‬is written as "S".

x Surface (Unicode): It is written in Arabic characters, and it isreflecting the word written as CODA specification presented in58chapter 2, and there is a work in progress to extending it to cover
Palestinian Dialect; So As we stated in [17] Palestinian dialect
does not have a standard orthography and we can’t also use the
MSA orthography because there are many differences inphonological, morphological and lexical. Also, Palestinian people
write the dialect in different ways that reflect the differences inphonology; for example the word “‫ ”دقيقة‬-might be written in fourways, such as “‫”دقيقة‬,”‫”دئيئة‬,”‫”دكيكة‬,”‫”دغيغة‬. Similarly, if a word has
many long vowels, then it may be written in different ways, forexample the word “‫ ”مساكين‬maybe also written as “‫”مسكين‬byshorting the first vowel. PAL also has some qualities that do notexist in MSA, which may be written in different ways, such as thePalestinian future particle ‫ ح‬which is written attached to the verb
that followed it such as “‫ ”حاروح‬or separate from the verb that
follows it such as “‫”ح اروح‬. Finally, there are words in thePalestinian dialect may be written

in different forms such as"‫"برضه‬,"‫"برضو‬, and ”‫” برضة‬.

59

All of these problems can be covered by applying the guidelines of
EGY version of CODA, but there are unique Palestinian problems that
we need to be deal with, such as: adding the letter “‫ ”ك‬to the list ofletters that spelling in different forms in dialect because it may be
spelling as “‫ك‬/k” or as “/tš/, adding the non-EGY to the list of clitics
such asdemonstrative proclitic ‫ه‬h+ ; e.g.: ‫ هالبنت‬، ‫ هالحياة‬، ‫ هالشغل‬andConjunction proclitic ‫ت‬t+ ‘so as to’, e.g., ‫تيوكل‬،‫ لتروحوا‬، ‫تيشوف‬, andFinally extending the list of exceptional words to cover additional
Palestinian words; an example of these words is word "‫ "هيو‬whichcorresponding to "‫ "ها هو‬in Standard Arabic, Table 4.1 provides a sample
from it

Table 4.1: Sample of Exceptional Palestinian words [17]xSurface (Buckwalter): The Buckwalter transliteration of the
Surface.

60

xLemma: The lemma is the dictionary entry that abstracts all
inflectional morphology. The lemma for the verbs is the third
person masculine singular perfective form and for nouns is the
masculine singular form if available or feminine singular form if
masculine is not available. The lemma of the word is written in
Buckwalter transliteration. E.g.: for the word “‫ ”يرى‬the lemma is“‫”رأى‬, for the word “‫ ”أوالد‬the lemma is “‫”ولد‬, for the word “‫”سيارات‬the lemma is “‫”سيارة‬, and for the word “‫ ”بيجي‬the lemma is “‫”أجا‬x Buckwalter POS: it is the fullest part-of-speech (POS) of theword; it combines prefixes, stem, and suffixes of a word with theirtags. Tags that used here are the tags that define in[21], these tags
are classified to many groups such as Nouns which have basic tags
as for example NOUN and NOUN_PROP and also have related
tags that come as prefix e.g. “DET” for “‫ ”ال‬or as suffixes
e.g.”NSUFF_FEM_PL” for “‫ (جمع المؤنث)"ات‬,another group is

Pronouns group which have many tags(PRON-1S e.g. “‫”أنا‬,DEM_PRON e.g. “‫(هالشغل)”ه‬,REL_PRON e.g. “‫”اللي‬, pronouns
that reflect objects of the verb e.g. PVSUFF_DO: 1S for “‫ ”ي‬in61

“‫”عزمني‬, pronouns that reflect subjects of the verb e.g.
IVSUFF_SUBJ: 1P for “‫ ”نا‬in “‫…”حيشوفنا‬etc), another group isgroup for adjectives which have tags(ADJ,ADJ_COMP), verbs
group which have tags ( IV for imperfective verb”‫”فعل مضارع‬, PV
for perfective verb “‫ ”فعل ماضي‬,CV for command verb “ ‫فعل‬‫…”أمر‬etc), Adverbs group that have tags(ADV, REL_ADV),
particles group that has tags(NEG_PART for negations e.g.
"‫"ما‬,”‫”ش‬, PROG_PART for progressive part in imperfective verbs
e.g. “‫ ”ب‬in “‫…”بيحكي‬etc), and Finally there is a tag “PREP” forprepositions as ,"‫"من‬CONJ for conjunctions such as “‫”و‬, INTERJ
for interjections such as "‫"اوكي‬, PSEUDO_VERB such as “‫”ياريت‬,TYPO such as “‫”اخص‬, and VERB such as “‫”حاشا‬.
Examples of pB:- ‫صفحة‬Æ SfH/NOUN+ p/NSUFF_FEM_SG- ‫أتجوز‬Æ A/IV1S+ tjwz/IV- ‫يما‬Æ ymA/NOUN- ‫بديش‬Æ bd/IV+ y/IVSUFF_SUBJ:1S+$/NEG_PART62

x MSA lemma: the equivalent lemma in Modern Standard Arabic(MSA). e.g.: for lemma “‫ ”أجا‬in the Palestinian dialect; thecorresponding lemma in MSA is “‫”جاء‬, for lemma “‫ ”بد‬thecorresponding lemma in MSA is “‫”أراد‬.

x Gloss it is the corresponding meaning of the lemma in English.
x Annotator A specification of the source of the annotation; e.g.:
diwan_approved meaning that annotation approved by annotatorusing DIWAN tool, MADA meaning that annotated automatically
By MADAMIRA

Table 4.2: Annotation Specification Example

One of the decisions that was taken in the annotation methodology [20] is
to discard diacritizion when write Buckwalter POS, this decision was
taken in order to minimize the load on the human annotator, and also
because EGY and MSA generates morpheme and lexical items that differ

63

from PAL only in the short vowels. Finally, in the methodology
considers diacritized Lemma only because it has a critical role ingenerating suitable gloss.

4.2 Annotation process
4.2.1 Using MADAMIRA
Now, after collecting our corpus, and specifying our annotation
specification, the question up to mind is: how can we annotate our
corpus? shall we annotate it manually or by using certain tool thenannotate it automatically? In order to answer these questions we decided
to make an experiment and then according to the experiment's results we
decided to use DIWAN tool to annotate our corpus which internallyannotate the words using MADAMIRA which presented in chapter 2.
This experiment starts by choosing randomly an episode of the PAL TV
show “Watan Aa Watar” (460 words), then entering it through both
MADAMIRA-MSA and MADAMIRA-EGY to analyze it, after that weanalyzed the output that we get from both systems to determine if it's64usable to annotate our corpus or not. Tables 4.3 and 4.4 show samples
from experiment results for both EGY and MSA, and Table 4.5 shows
experimental results

Table 4.3: MADAMIRA EGY Result

Table 4.4: MADAMIRA MSA Result

Table 4.5: Experiment result65

As we noticed in Table 4.5, Egyptian dialect looks close to Palestinian
dialect; this is because we focus here on written data not in oral data.
Palestinian and Egyptian people write, for example, “‫ ”قلتلك‬in the sameway, although both pronounce it differently.
The last column "Status" in tables 4.3, and 4.4 isn't generated
automatically by MADAMIRA, we added this column manuallyaccording to analysis that generated by MADAMIRA; we gave "No
ANALYSIS" to words that MADAMIRA fails on it, also we gave
"Wrong" to words that MADAMIRA returned wrong analysis insomewhere on it such as gloss, POS, lemma, and finally we gave
"Correct" to words that MADAMIRA returned correct analysis for it inall parts(correct gloss, correct POS, correct lemma).
Words with status No Analysis in Tables 4.3, and 4.4 refer to the words
that the morphological analyzer couldn't analysis it. There are manyreasons that cause failure in analysis in MADAMIRA such as the word is
totally Palestinian (it is used in the Palestinian dialect only), e.g. word inentry #2 “‫”شحال‬, and word in entry #15 “‫ ”لتنغنغ‬in Table 4.3.

66

While the wrongly analyzed words are words MADAMIRA gives it
incorrect part-of-speech (POS) or incorrect lemma or incorrect gloss. An
example of wrong analysis is the word in entry#1 “‫”بدها‬, which
MADAMIRA-EGY analyze it as noun while it is a verb and it is a totally
Palestinian word. In general, the results show that we can use
MADAMIRA-EGY to annotate our corpus because almost it gives
correct analysis or give analysis that need some modifications.
4.4.2 Manual Annotation
After annotating the corpus using MADAMIRA-EGY the total
Number of words annotated By MADAMIRA-EGY is 55586 (16,334
unique);number

of

words

that

MADAMIRA_EGY

returned

NO_ANALYSIS is 1689 word from 55586; these words which
equivalent to 1244 unique words are annotated manually. Figure 4.1
shows sample from MADAMIRA result

67

Figure 4.1: Sample of MADAMIRA_EGY Result

Figure 4.2 shows a sample from words that annotated manually

Figure 4.2: Sample of Manual Annotated List

4.2.3 Correct errors and fill in gaps using DIWAN
After we annotated NO_ANALYSIS words manually; we started
with the next step which is to make a double check over other words that68

annotated by MADAMIRA_EGY to verify it if it is analyzed correctly ornot, then to correct words that are annotated wrongly. A decision wasmade to do this step using DIWAN tool. DIWAN is a tool developed atColumbia University for text annotating. DIWAN takes a text as input; inthis case a text is a MADAMIRA_EGY output file, then DIWAN shows
every word with its annotation for annotator; annotator may choose tosave it as is or to make some updates on it. Figure 4.3 shows DIWANInterface.

Figure 4.3: DIWAN Interface

So as shown in Figure 4.3, annotator chooses the word, then clicks
on it, then check the annotation appeared and either save as it or makesome changes on it; most of the time the changes are occurring in POS,

69

English Gloss, and MSA. Output saved in files that looks like
MADAMIRA_EGY file with two extra attributes which are: Source mod
to indicate if the annotator updated it or not (No: annotator does not
update it, Yes: annotator update it), ANNO

which has value

diwan_approved to state that annotation is verified By DIWAN. In This
Thesis the number of words that double checked By DIWAN are
18743(501 unique words). Figure 4.4 shows sample from DIWAN output

Figure 4.4: DIWAN Output Sample

4.3 Discussion
As we noticed previously, using DIWAN to do a double check on
MADAMIRA annotations will increase the productivity in building
annotated corpus, also using DIWAN will preserve the quality of the70

annotated corpus because it minimizes the entries that users will enter it
manually. So if we suppose that there is no tool that helps us inannotating our corpus then the annotation process will take too long timeand maybe also cause a lot of errors in annotation. From our experiencein manual annotation step, annotating of NO_ANALYSIS words (1244words) takes about two months, which is too long time and also manyerrors appear in first iteration then in the second iteration. We minimize
these errors and finally we make a double check on it using DIWAN. So
if we don’t use DIWAN then annotator will be ask to enter all things
manually and be very careful when entering tags in pB part, because any
error will cause a problem, and as we know there are a lot of tags, all ofthese tags DIWAN presents it as auto-complete list, also gender, number,
aspect and person are presented as list which will also minimize error
because if annotator enters it then sometimes will enter gender as female,
male, another time as f, and m, and also may enter person sometimes as
one, second, third, another time as 1,2, and 3, and may enter number as
singular, plural, Dual or as s, p, and d, so all of these errors will cause areal problem because we need it in standard format to make our corpus

71

applicable in another applications, So using DIWAN will save time andquality.
till now we annotate about 500 unique words which equivalent to 18743
non unique words; which means that we annotated all duplicates of 500
unique words; as an example the word "‫ "يابا‬is repeated about 26 times in
the corpus in a different context, also these 500 unique words are themost frequent words in our corpus which means that some of them are
duplicated more than 500 times in our corpus. Finally, every time we use
DIWAN we accelerate process more and more.

4.4 Evaluation
To evaluate the quality of annotations, we consider the inter-annotator
agreement. We chose a sample of 59 words from our corpus that weannotated, and asked a colleague (Faeq Rimawi, from Sina Institute) toannotate this sample separately, then we compared the annotations.
As we notice in Table 4.6, the percentage of differences in annotations
regardless of cause of difference is 20.3% which comes from differencesin MSA lemma, lemma, and from Buckwalter POS with POS, also72

sometimes the two annotator gives two different values for many
attributes in the same word such as different values for both POS and
Buckwalter POS, or for MSA lemma and gloss.
Number of words

Percentage

Difference in total

12 word20.30%

Difference in Lemma

2 word3.30%

Difference in MSA Lemma

4 word6.70%

Difference in gloss

4 word6.70%

Difference in BW, POS

6 word10.10%

Table 4.6: Inter-annotator agreement result

Figure 4.5 shows a sample for differences between two annotators; thissample explained the number of words in Table 4.6. For example, thetwo annotators gave the word “ ‫ ”دشر‬different values for both MSA
lemma, and gloss, so we considered it in the differences of both.

73

Figure 4.5 : annotators differences Sample

74

Chapter 5

Conclusion and Future Work

This Thesis presented a sample from our annotated corpus; it presented
500 words that annotated very well according to our annotation
methodology. It discussed the linguistic variations between Palestinian
dialect and Modern Standard Arabic especially in terms of morphology,
orthography, and lexicon. It also discussed our annotation methodology,
the benefits of using MADAMIRA-EGY, and annotation tool DIWAN,
to semi-automate and speed up the annotation process.
This Thesis raised many issues that need to be addressed in the future
work and researches such as; Complete using DIWAN tool to approve
MADAMIRA_EGY annotations by annotator; the result of this step isfully annotated PAL corpus ; this corpus contains 16334 unique words
annotated with relevant meta data such as ( Surface, Lemma, POS,
Buckwalter POS,

MSA

lemma,English gloss),

Completethedevelopment of Palestinian-specific morphological annotation tags and75

CODA guidelines, build A Palestinian lexicon, which will be extracted
from surface, lemma, MSA lemma, and English gloss attributes in
corpus, extend MADAMIRA to analyze Palestinian text by learning
MADAMIRA_EGY with DIWAN output data, Corpus will be extended
to include more text; these texts will be collected from different resources
such as the new parts(2014) from Palestinian TV show “‫”وطن ع وتر‬, andfrom TV Shows “‫”فنجان البلد‬, “‫”حروف وطن‬, and the famous Palestinian
series “‫”التغريبة الفلسطينية‬, also from others Palestinian Facebook andTwitter pages and also collecting more text from Palestinian forums andFinally by collecting poems that written in Palestinian dialect such as thecollection of poems“‫ ”ميجنا‬for the Palestinian poet “‫”تميم البرغوثي‬.
Finally, all lexical annotations for our corpus such as Lemma will belinked with Arabic ontology resources. The corpus will finally be
published on public for researchers.

76

References[1] Rambow, O., Chiang, D., Diab, M., Habash, N., Hwa, R., Sima’an,
K., Lacey, V., Levy, R., Nichols, C., and Shareef, S. 2006. Parsing
Arabic Dialects. Final Report-Version 1, January 18, 2006.
[2] Maamouri, M., Bies, A., Buckwalter, T., Diab, M., Habash, N.,
Rambow, O., and Tabessi, D. 2006. Developing and using a Pilot
Dialectal Arabic Treebank. In Proceedings of the Fifth International
Conference on Language Resources and Evaluation, LREC’06[3] Chiang, D., Diab, M., Habash, N., Rambow, O., and Shareef, S.
2006.Parsing Arabic Dialects. In Proceedings of European Chapter ofthe Association for Computational Linguistics (EACL), Trento, pages
369–376.
[4] Benajiba, Y., and Diab, M. 2010.A Web Application for Dialectal
Arabic Text Annotation. In Proceedings of the LREC Workshop for
Language Resources (LRs) and Human Language Technologies(HLT) for Semitic Languages: Status, Updates, and Prospects. 201077

[5] Diab, M., Habash, N., Rambow, O., Altantawy, M., and Benajiba, Y.
2010. COLABA: Arabic dialect annotation and processing. In
Proceedings of LREC Workshop on Semitic Language Processing.
2010. p. 66-74[6] Diab, M., and Habash, N.: Presentation about Arabic Dialect
Processing. MEDAR 2009, Cairo, Egypt, April 21, 2009[7] Lubany, A.: Dictionary of Palestinian Vocabularies and Loan Words(Arabic-Arabic) ‫معجم العامي و الدخيل في فلسطين عربي –عربي‬.2006.Lebanon
Library.
[8] Habash, N., Diab, M., and Rambow, O. Conventional Orthography
for

Dialectal

Arabic:Principles

andGuidelines–EgyptianArabic.2012. Columbia university.
[9] Zribi, I., Boujelbane, R., Masmoudi, A., Ellouze, M., Belguith, L.,andHabash,N.

Conventional

Orthography

for

Tunisian

Arabic.2014.ANLP Research group, MIRACL Lab, University ofSfax, Tunisia.

78[10] Habash, N., Rambow, O., and Kiraz, G. 2005. Morohologicalanalysis and generation for Arabic Dialects. Semitic '05 Proceedingsof the ACL Workshop on Computational Approaches to Semitic
Languages. Pages 17-24[11] Habash, N., and Rambow, O. 2006. MAGEAD: Morphological
Analyzer and Generator for Arabic Dialects. In proceedings of the
21st International conference on Computational linguistics and 44 th
Annual Meeting of ACL, pages 681-688, Sydney, July 2006.
[12] Altantawy, M., Habash, N., Rambow, O., and Saleh, I. 2010.
Morphological Analysis and Generation of Arabic Nouns: A
Morphemic Functional Approach. In Proceedings of the International
Conference on Language Resources and Evaluation, LREC 2010, 1723 May 2010, Valletta, Malta; 01/2010.
[13] Almeman, K., and Lee, M. 2012.Towards Developing a MultiDialect Morphological Analyzer for Arabic. In Proceedings of 4thInternational Conference on Arabic Language Processing, May 2-3,
2012, Rabat, Morocco.

79[14] Habash, N., Rambow, O., Diab, M., and Kanjawi-Faraj, R.
2008.Guidelines
Proceedings

of

for

Annotation

workshopofonArabicArabicDialectness.
and

its

In
local

Languages,Marrakech,Mococco,2008.
[15] Habash, N., Rambow, O., and Roth, R. .MADA+TOKEN: A Toolkit
forArabicTokenization,

Diacritization,

Morphological

Disambiguation, POS Tagging, Stemming, and Lemmatization.
Center for Computational Learning Systems, Columbia University,
New York, NY, USA.
[16] Pasha, A., Al-Badrashiny, M., Diab, M., El Kholy, A., Eskandar, R.,
Habash,N.,

Pooleery,M.,

Rambow,

O.,

and

Roth,

R.2014.MADAMIRA: A Fast, Comprehensive Tool Morphological
Analysis and Disambiguation of Arabic. Center for Computational
Learning Systems, Columbia University, New York, NY, USA.,
Department of Computer Science, The George Washington
University, Washington, DC., and Google, Inc.

80

[17] Habash, N., Jarrar, M., Akra, D., Rimawi, F., and Arrar, M.
Conventional

Orthography

for

Dialectal

Arabic:PalestinianArabic.2014. Birzeit university.
[18] Habash, N., Roth, R., Rambow, O., Eskander, R., and Tomeh, N.
Morphological

AnalysisandArabic.2013.In Proceedings

Disambiguationof NAACL-HT,

for

Dialectal

pages

426-432,

Atlanta,Georgia,9-14 June 2013, Association for Computational
Linguistics.
[19] Habash, N., Eskander, R., and Hawwari, A. Morphological Analyzer
for Egyptian Arabic.2012.In Proceedings of the Twelfth Meeting ofSpecialInterest

Group

on

Computational

Morphology

andPhonology(SIGMORPHON2012), pages 1-9, Montreal, Canada,
June7, 2012, Association for Computational Linguistics.
[20] Jarrar, M., Habash, N., Akra, D., and Zalmout, N.: Building A
corpus for Palestinian Arabic: Preliminary Study. 2014. In
Proceedings of the EMNLP 2014 Workshop on Arabic Natural
Language Processing, pages 18-27, Doha, Qatar, October, 2014,
Association for Computational Linguistics.

81[21] Maamouri, M., Krouna, S., Slimane, D., Hamrouni, N, and Habash,
N. Egyptian Arabic (ARZ) Morphological Analysis & POS
Annotation.2012.
[22] Buckwalter, T.: Buckwalter Arabic Morphological analyzer version
2.0.2004. LDC catalog Number LDC2004L02, ISBN1-58563-342-0.

82Appendices83

Appendix #1

84

Building a Corpus for Palestinian Arabic:
a Preliminary Study
Mustafa Jarrar, *Nizar Habash, Diyam Akra, Nasser Zalmout
Birzeit University, West Bank, Palestine{mjarrar,nzalmout}@birzeit.edu, diyam@student.birzeit.edu*

New York University Abu Dhabi, United Arab Emirates
nizar.habash@nyu.edu

the Arab World. Using such tools to understandand process Arabic dialects (DAs) is achallenging task because of the phonological andmorphological differences between DAs andMSA. In addition, there is no standard
orthography for DAs. Moreover, DAs have
limited standardized written resources, sincemost of the written dialectal content is the resultof ad hoc and unstructured social conversations
or commentary, in comparison to MSA’s vast
body of literary works.

Abstract
This paper presents preliminary results inbuilding an annotated corpus of thePalestinian Arabic dialect. The corpus
consists of about 43K words, stemmingfrom diverse resources. The paper
discusses some linguistic facts about thePalestinian dialect, compared with theModern Standard Arabic, especially interms of morphological, orthographic,
and lexical variations, and suggests somedirections to resolve the challenges thesedifferences pose to the annotation goal.
Furthermore, we present two pilot
studies that investigate whether existing
tools for processing Modern Standard
Arabic and Egyptian Arabic can be used
to speed up the annotation process of ourPalestinian Arabic corpus.

1.

The rest of this paper is structured as follows:
We present important linguistic background in
Section 2, followed by a survey of related workin Section 3. We then present the process ofcollecting the Curras Corpus (Section 4) and thechallenges of annotating it (Section 5).

2.

Linguistic BackgroundIn this section we summarize some important
linguistic facts about PAL that influence thedecisions we made in this project. For more
information on PAL and Levantine Arabic ingeneral, see (Rice and Sa’id, 1960; Cowell,1964; Bateson, 1967; Brustad, 2000; Halloun,
2000; Holes, 2004; Elihai, 2004). For a
discussion of differences between Levantine andEgyptian Arabic (EGY), see Omar (1976).

Introduction and Motivation

This paper presents preliminary results towards
building a high-coverage well-annotated corpusof the Palestinian Arabic dialect (henceforth
PAL), which is part of an ongoing project called
Curras. Building such a PAL corpus is a firstimportant step towards developing natural
language processing (NLP) applications, forsearching, retrieving, machine-translating, spellchecking PAL text, etc. The importance ofprocessing and understanding such text isincreasing due to the exponential growth ofsocially generated dialectal content at recent
Social Media and Web 2.0 breakthroughs.

2.1 Arabic and its dialects
The Arabic language is a collection of variants
among which a standard variety (MSA) has a
special status, while the rest are considered
colloquial dialects (Bateson, 1967, Holes, 2004;
Habash, 2010). MSA is the official written
language of government, media and education in
the Arab World, but it is not anyone’s native
language; the spoken dialects vary widely acrossthe Arab World and are the true native varieties

Most Arabic NLP tools and resources weredeveloped to serve Modern Standard Arabic(MSA), which is the official written language in

18
Proceedings of the EMNLP 2014 Workshop on Arabic Natural Langauge Processing (ANLP), pages 18–27,
October 25, 2014, Doha, Qatar. c 2014 Association for Computational Linguistics

of Arabic, yet they have no standard orthography
and are not taught in schools (Habash et al.,2012, Zribi et al., 2014).

dialects. The Druze dialect retains the /q/
pronunciation. Another example is the /k/phoneme (corresponding to MSA ! k), which
realizes as /tš/ in rural dialects. These difference
cause the word for ‫ ﻗﻠﺐ‬qlb ‘heart’ to bepronounced as /qalb/, /’alb/, /kalb/ and /galb/ andto be ambiguous out of context with the word ‫ﻛﻠﺐ‬klb ‘dog’ /kalb/ and /tšalb/. And similarly to
EGY (but unlike Tunisian Arabic), the MSA
phoneme /θ/ (! θ) becomes /s/ or /t/, and the
MSA phoneme /ð/ (! ð) becomes /z/ or /d/ in
different lexical contexts, e.g., MSA !‫ ﻛﺬ‬kðb/kaðib/ ‘lying’ is pronounced /kizib/ in PAL and/kidb/ in EGY.

PAL is the dialect spoken by Arabic speakers
who live in or originate from the area of
Historical Palestine. PAL is part of the South
Levantine Arabic dialect subgroup (of which
Jordanian Arabic is another dialect). PAL is
historically the result of interaction between
Syriac and Arabic and has been influenced bymany other regional language such as Turkish,
Persian, English and most recently Hebrew. The
Palestinian refugee problem has led to additional
mixing among different PAL sub-dialects as well
as borrowing from other Arabic dialects. We
discuss next some of the importantdistinguishing features of PAL in comparison to
MSA as well as other Arabic dialects. We
consider the following dimensions: phonology,
morphology, and lexicon. Like other Arabic
dialects, PAL has no standard orthography.

Similar to many other dialects, e.g. EGY andTunisian (Habash et al., 2012; Zribi et al., 2014),
the glottal stop phoneme that appears in many
MSA words has disappeared in PAL: compare
MSA !"# rÂs /ra’s/ ‘head’ and ‫ ﺑﺌﺮ‬bŷr /bi’r/‘well’ with their Palestinian urban versions: /rās/and /bīr/. Also, the MSA diphthongs /ay/ and/aw/ generally become /ē/ and /ō/; thistransformation happens in EGY but not in other
Levantine dialects such as Lebanese, e.g., MSA‫ ﺑ"ﺖ‬byt /bayt/ ‘house’ becomes PAL /bēt/.

2.2 Phonology
PAL consists of several sub-dialects thatgenerally vary in terms of phonology andlexicon preferences. Commonly identified subdialects include urban (which itself varies mostly
phonologically among the major cities such as
Jerusalem, Jaffa, Gaza, Nazareth, Nablus andHebron), rural, and Bedouin. The Druze
community
has
also
some
distinctive
phonological features that set it apart. The
variations are a miniature version of thevariations in Levantine Arabic in general.
Perhaps the most salient variation is thepronunciation of the /q/ phoneme (corresponding
to MSA ! q1), which realizes as /’/ in most urban
dialects, /k/ in rural dialects, and /g/ in Bedouin

PAL also elides many short vowels that appear
in the MSA cognates leading to heavier syllabic
structure, e.g. MSA !‫ ﺟﺒﺎ‬/jibāl/ ‘mountains’ (andEGY /gibāl/) becomes PAL /jbāl/. Additionallylong vowels in unstressed positions in some PAL
sub-dialects shorten, a phenomenon shared with
EGY but not MSA: e.g., compare /zāru/ (!"#!$
zAr+uwA) ‘they visited’ with /zarū/ (!"#$%
zAr+uw+h) ‘they visited him’. Finally, PAL has
commonlyinserted
epenthetic
vowels(Herzallah, 1990), which are optional in somecases leading to multiple pronunciations of thesame word, e.g., /kalb/ and /kalib/ (‫ ﻛﻠﺐ‬klb‘dog’). This multiplicity is not shared with MSA,
which has a simpler syllabic structure and more
limited epenthesis than PAL.

1

Arabic orthographic transliterations are provided in the
Habash-Soudi-Buckwalter (HSB) scheme (Habash et al.,2007), except where indicated. HSB extends Buckwalter’stransliteration scheme (Buckwalter, 2004) to increase its
readability while maintaining the 1-to-1 correspondencewith Arabic orthography as represented in standard
encodings of Arabic, i.e., Unicode, etc. The following are
the only differences from Buckwalter’s scheme (indicatedin parentheses): Ā ! (|), Â ! (>), ŵ ! (&), Ǎ ! (<), ŷ ! (}), ħ !
(p), θ ! (v), ð ! (*), š ! ($), Ď ! (Z), ς ! (E), γ ! (g), ý !
(Y), ã ً ‫( ـ‬F), ũ ٌ ‫( ـ‬N), ĩ ٍ‫( ـ‬K). Orthographic transliterations arepresented in italics. For phonological transcriptions, wefollow the common practice of using ‘/.../’ to represent
phonological sequences and we use HSB choices with someextensions instead of the International Phonetic Alphabet(IPA) to minimize the number of representations used, as
was done by Habash (2010).

2.3 Morphology
PAL, like MSA and its dialects and other
Semitic languages, makes extensive use of
templatic morphology in addition to a large setof affixations and clitics. There are however
some important differences between MSA andPAL in terms of morphology. First, like manyother dialects, PAL lost nominal case and verbal
mood, which remain in MSA. Additionally, PAL
in most of its sub-dialects collapses the feminineand masculine plurals and duals in verbs and

19

3.

most nouns. Some specific inflections areambiguous in PAL but not MSA, e.g., ‫ ﺣﺒ"ﺖ‬Hbyt/Habbēt/ ‘I (or you [m.s.]) loved’.

Related Work

3.1 Corpus Collection and Annotation
There have been many contributions aiming todevelop annotated Arabic language corpora, withthe main objective of facilitating Arabic NLP
applications. Notable contributions targeting
MSA include the work of Maamouri and Cieri,(2002), Maamouri et al. (2004), Smrž and Hajič(2006), and Habash and Roth (2009). These
efforts developed annotation guidelines for
written MSA content producing large-scale
Arabic Treebanks.

Second, some specific morphemes are slightly orquite different in PAL from their MSA forms,
e.g., the future marker is /sa/ in MSA but /Ha/ or/raH/ in PAL. Another prominent example is the
feminine singular suffix morpheme (TaMarbuta), which in MSA is pronounced as /at/
except at utterance final positions (where it is/a/). In some PAL urban sub dialects, it has
multiple allomorphs that are phonologically andsyntactically conditioned: /a/ (after non-front andemphatic consonants), /e/ (after front nonemphatic consonants), /it/ (nouns in constructstate such as before possessive pronouns) and /ā/(in deverbals before direct objects): e.g. ‫ ﺑﻄﺔ‬bTħ/baTT+a/ ‘duck’, ‫ ﺣﺒﺔ‬Hbħ /Habb+e/ ‘pill’, ‫ﺑﻄﺘﻨﺎ‬bTnA /baTT+it+na/ ‘our duck’ and /mdars+ā
+hum/ ‘she taught them’.

Contributions that are specific to DA include thedevelopment of a pilot Levantine Arabic
Treebank (LATB) of Jordanian Arabic, whichcontainedmorphologicalandsyntactic
annotations of about 26,000 words (Maamouri etal., 2006). To speed up the process of creating
the LATB, Maamouri et al. (2006) adapted MSA
Treebank guidelines to DA and experimented
with extensions to the Buckwalter ArabicMorphological Analyzers (Buckwalter, 2004).
The LATB was used in the Johns Hopkins
workshop on Parsing Arabic Dialect (Rambow etal., 2005; Chiang et al., 2006), which
supplemented the LATB effort with anexperimental Levantine-MSA dictionary. The
LATB effort differs from the work presented
here in two respects. First, the LATB corpus
consists of conversational telephone speechtranscripts, which eliminated the orthographic
variations issues that we face in this paper.
Secondly, when the LATB was created, there
were no robust tools for morphological analysisof any dialects; this is not the case any more. We
plan to exploit existing tools for EGY to help the
annotation effort.

Third, PAL has many clitics that do not exist in
MSA, e.g., the progressive particle /b+/ (as in/b+tuktub/ ‘she writes’), the demonstrative
particle /ha+/ (as in /ha+l+bēt/ ‘this house’), thenegation cirmcumclitic /ma+ +š/ (as in/ma+katab+š/ ‘he did not write’) and the indirect
object clitic (as in /ma+katab+l+ō+š/ ‘he did notwrite to him’). All of these examples except forthe demonstrative particle are used in EGY.
2.4 Lexicon
The PAL lexicon is primarily Arabic withnumerous borrowings from many different
languages. MSA cognates generally appear withsome minor phonological changes as discussed
above; a few cases include more complex
changes, e.g. /biddi/ ‘I want’ is from MSA/bi+widd+i/ ‘in my desire’ or /illi/ ‘relative
pronoun which/who/that’ which corresponds to aset of MSA forms that inflect for gender andnumber (!‫ﻟﺬ‬$ Alðy, ‫ﻟﺘﻲ‬$ Alty, etc.). Some common
PAL words are portmanteaus of MSA words,
e.g., /lēš / ‘why?’ corresponds to MSA /li+’ayy+i
šay’/ ‘for what thing?’. Examples of common
words that are borrowed from other languages
include the following:• !‫ﻧﺎﻣ‬%&' /roznama/ ‘calendar’ (Persian)• !"‫ ﻛﻨﺪ‬/kundara/ ‘shoe’ (Turkish)• !"#‫ ﺑﻨﺪ‬/banadora/ ‘tomato’ (Italian)• ‫ ﺑﺮ"ﻚ‬/brēk/ ‘brake (car)’ (English)• !‫ﻮ‬#‫ﺰ‬%‫ﻔ‬%‫ ﺗﻠ‬/talifizyon/ ‘television’ (French)• !‫ ﻣﺤﺴﻮ‬/maHsūm/ ‘checkpoint’ (Hebrew)

Other DA contributions include the Egyptian
Colloquial Arabic Lexicon (ECAL) (Kilany, etal., 2002), which was developed as part of the
CALLHOME Egyptian Arabic (CHE) corpus(Gadalla, et al., 1997). In addition to YADAC(Al-Sabbagh and Girju, 2012), which was basedon dialectal content identification and web
harvesting of blogs, micro blogs, and forums of
EGY content. Similarly, the COLABA project(Diab et al., 2010) developed annotated dialectal
content resources for Egyptian, Iraqi, Levantine,and Moroccan dialects, from online weblogs.

20

et al., 2006; Habash, et al., 2012b). Therefore, it
is important to develop annotated andmorpheme-segmented resources, along withmorphological analysis tools, that are specificand tailored for DAs. One of the notable recent
contributions for EGY morphological analysis
was CALIMA (Habash et al., 2012b). The
CALIMA analyzer for EGY and the commonlyused SAMA analyzer for MSA (Graff et al.,2009) are central in the functioning of the EGY
morphological tagger MADA-ARZ (Habash etal., 2013), and its successor MADAMIRA(Pasha et al., 2014), which supports both MSA
and EGY.

3.2 Dialectal Orthography
Due to the lack of standardized orthography
guidelines for DA, along with the phonologicaldifferences in comparison to MSA, and dialectal
variations within the dialects themselves, there
are many orthographic variations for written DA
content. Writers in DA, regardless of the context,
are often inconsistent with others and even withthemselves when it comes to the written form of
a dialect; writing with MSA driven orthography,
or writing words phonologically sometimes.
These orthography variations make it difficult
for computational models to properly identify
and reason about the words of a given dialect(Habash et al, 2012a), hence, a conventional
form for the orthographic notations is important.
Within this scope, we can view this problem forLevantine dialects as an extension of the work ofHabash et al. (2012a) who proposed the socalled CODA (Conventional Orthography for
Dialectal Arabic). CODA is designed for the
purposeofdeveloping
conventional
computational models of Arabic dialects ingeneral. Habash et al. (2012a) provides adetailed description of CODA guidelines asapplied to EGY. Eskander et al. (2013) identify
five goals for CODA: (i) CODA is an internallyconsistent and coherent convention for writingDA; (ii) CODA is created for computational
purposes; (iii) CODA uses the Arabic script; (iv)
CODA is intended as a unified framework forwriting all DAs; and (v) CODA aims to strike an
optimal balance between maintaining a level of
dialectal
uniquenessandestablishingconventions based on MSA-DA similarities.
CODA guidelines will be extended to cover PAL
in this paper, as discussed in Section 5.3.

The work we present in this paper builds on theshoulders of these previous efforts from thedevelopment of guidelines for orthography andmorphology (in MSA and EGY) to the use ofexisting tools (specifically MADAMIRA MSA
and EGY) to speed up the annotation process.

4.

Corpus Collection

Written dialects in general tend to have scarce
resources in terms of written literature; writtenmaterials usually involve informal conversations
or traditional folk literature (stories, songs, etc.).
It is therefore often difficult to find resources for
written dialectal content. In addition, resourcesof dialectal content are prone to significant noise
and inconsistency because they tend to lackstandard orthographies and rely on ad hoctranscriptions and orthographic borrowing fromthe standard variety. In the case of Arabic,
unlike MSA that dominates the formal andwritten content outlets, as in the press, scientificarticles, books, and historical narration, DAs are
more naturally used in traditional and informal
contexts, such as conversations in TV series,
movies, or on social media platforms, providing
socially powered commentary on different
domains and topics. And given the lack ofstandard orthography, there is common mixingof phonetic spelling and MSA-cognate-based
spelling in addition to the so-called Arabizi
spelling – writing DAs in Roman script, rather
than Arabic script (Darwish, 2014 and AlBadrashiny et al., 2014). Such noise imposes
many challenges regarding the collection ofhigh-coverage high-accuracy DA corpora. It istherefore important to remark that althoughbigger is better when it comes to corpus size, we
focus more in this first iteration of our PAL
corpus on precision and variety rather than mere

3.3 Dialectal Morphological Annotation
Most of the work that explored morphology inArabic focused on MSA (Al-Sughaiyer and AlKharashi, 2004; Buckwalter, 2004; Habash andRambow, 2005; Graff et al., 2009; Habash,2010). The contributions for DA morphology
analysis, however, are relatively scarce and areusually based on either extending available MSA
tools to tackle DA specificities, as in the work of(Abo Bakr et al., 2008; Salloum and Habash,2011), or modeling DAs directly, without relying
on existing MSA contributions (Habash andRambow, 2006). Due to the variations between
MSA and DAs, available MSA tools andresources cannot be easily extended ortransferred to work properly for DA (Maamouri,

21

size. That is, we tried not only to manually selectand review the content of the corpus, but also toassure that we covered a variety of topics andcontexts, localities and sub-dialects, including
the social class and gender of the speakers andwriters. This is because such aspects help us
discover new language phenomena in the dialect
as will be discussed in the next section.

annotation of a new dialectal
orthography and morphology.
5.1 Annotation Specification

The words are annotated in context. As such, thesame word may receive different annotations indifferent contexts. We define the annotation of aword as a tuple <w, wB, c, cB, l, pB, g, i>
described as follow. (Examples of suchannotations are illustrated in Table 5.):

Table 1 presents the resources that we manually
collected to build the PAL Curras corpus. There
are 133 social media threads (about 16k words)
from blogs (e.g., ‫ﻧﺔ ﻋﺒﺪ &ﻟﺤﻤ(ﺪ &ﻟﻌﺎ"ﻲ‬/‫ﻣﺪ‬
Abdelhameed Alaaty’s blog), forums (e.g., ‫ﺷﺒﻜﺔ‬‫ﻨﻲ‬#‫ )ﻟﺤﻮ)* )ﻟﻔﻠﺴﻄ‬The Palestinian dialogue network),
Twitter, and Facebook. The collection was doneby reading many discussion threads andselecting the relevant ones to assure diversity
and PAL representative content. Content that isheavily written in a mix of languages, or a mixof other dialects was excluded. In the same way,
we also manually collected some PAL stories,
and a list of PAL terms and their meanings,
which reflect additional diversity of topics,
contexts, and social classes. About half of our
corpus comes from 41 episode scripts from thePalestinian TV show ‫ﺗﺮ‬# $ ‫&ﻦ‬# “Watan Aa
Watar”. Each episode discusses and provides
satirical critiques regarding different topics ofrelevance to the Palestinian viewers about daily
life issues. The show’s importance stems fromthe fact that the actors use a variety ofPalestinian local dialects, hence enriching thecoverage of the corpus.

•

•

•

•
•

•

Table 1. The Curras Corpus Statistics
Document Type
Word Word Documents
Tokens Types
Facebook
3,120
1,985 35 threads
Twitter
3,541
2,133 38 threads
Blogs
8,748
4,454 37 threads
Forums1,092
798 33 threads
Palestinian Stories
2,407
1,422 6 stories
Palestinian Terms
759
556 1 doc
TV Show: ‫ﺗﺮ‬# $ ‫&ﻦ‬# 23,423
8,459 41 episodes
Watan Aa Watar
Curras Total
43,090 19,807
191

5.

corpus:

Corpus Annotation Challenges

This section presents our approach toannotating the Curras corpus. We start with a
specification of our annotation goals, followed
by a discussion of our general approach. We
then discuss in more details two important
challenges that need to be addressed for22

w: Raw (Unicode) The raw input word
defined as a string of letters delimited bywhite space and punctuation. The word isrepresented in Arabic script (Unicode).
wB: Raw (Buckwalter) The same raw input
word in the commonly used Buckwaltertransliteration (Buckwalter, 2004).
c: CODA (Unicode) The Conventional
Orthography (Habash et al., 2012) version of
the input word.
cB: CODA (Buckwalter) The Buckwaltertransliteration of the CODA form.
l: Lemma The lemma of the word in
Buckwalter transliteration. The lemma is thecitation form or dictionary entry that
abstracts over all inflectional morphology(but not derivational morphology). The
lemma is fully diacritized. We follow thedefinition of lemma used in BAMA
(Buckwalter, 2004) and CALIMA-ARZ(Habash et al., 2012b).
pB: Buckwalter POS The Buckwalter full
POS tag, which identifies all clitics andaffixes and the stem and assigns each a subtag. This representation treats clitics as
separatetokensandabstracts
theorthographic rewrites they undergo when
cliticized. See the handling of thel/PREP+Al/DET in word #6 in Table 5.
This representation is used by the LDC in
the Penn Arabic Treebank (PATB)(Maamouri et al., 2004) and tools such as
MADAMIRA (Pasha et al., 2014). It is ahigh granularity representation that allows
researchers to easily go to coarser
granularity POS (Diab 2007; Habash, 2010;
Alkuhlani et al., 2013). The Buckwalter POS
tag can be fully diacritized or undiacritized.
Given the added complexity of producing
diacritized text manually by annotators, we
opted at this stage to only use undiacritized
forms.

••that most Arabic speakers do not do and thus itrequires a lot of training and precious attention
to detail; (ii) MSA and EGY produce manymorphemes and lexical items that are quite
similar to PAL except in terms of the shortvowels (compare the lemmas for word #5 inTables 3, 4 and 5); (iii) PAL has many cases ofmultiple valid diacritizations as mentioned
above. While we think a convention should be
defined to explain the variation and model it, it isperhaps the topic of a future effort that is more
focused on PAL phonology. We make an
exception for the lemmas and diacritize themsince lemmas are important in indicating thecore meaning of the word. In case of different
pronunciations of the lemma, we choose theshortest.

g: Gloss The English gloss, an informal
semantic denotation of the lemma. In Tables
3-5, we only use one English word for space
limitations.
i: Analysis A specification of the source of
the annotation, e.g., ANNO is a humanannotator, and MADA is the MADAMIRA
system with some minor or no automatic
post-processing. In Tables 3 and 4, which
are produced automatically, the Analysis
field is replaced with a status indicating how
usable the automatic annotation is.

5.2 General Approach
To speed up the process of annotating our
corpus, we made the following decisions. First,
and quite obviously from the previous section,
we made a conscious decision to follow on thefootsteps of previous efforts for MSA and EGY
annotation done at the Linguistic Data
Consortium and Columbia’s Arabic Modelinggroup in terms of guidelines for orthography
conventionalization
andmorphologicalannotation. This allows us to exploit existing
guidelines with only essential modification toaccommodate PAL and produce annotations thatare comparable to those done for MSA and
EGY. This, we hope, will encourage research indialectal adaptation techniques and will makeour annotations more familiar and thus usable by
the community.
Second, and closely related to the first point,
we exploit existing tools to speed up the
annotation process. In this paper, we specificallyuse the MADAMIRA tool (Pasha et al., 2014)
for morphological analysis and disambiguationof MSA and EGY. Our choice of using this tool
is motivated by the assumption that EGY/MSA
and PAL share many orthographic andmorphological features. This assumption was
validated by pilot experiments, presented below,
and which show most of the PAL annotations
can be generated automatically. However, amanual step is then needed to verify every
annotation, to correct errors and fill in gaps. The
manual annotation has not been completed yet asof the writing of this paper submission.

5.3 A Conventional Orthography for PAL
As explained in Section 2, PAL, like otherArabic dialects, does not have a standard
orthography. Furthermore, there are numerous
phonological, morphological and lexical
differences between PAL and MSA that makethe use of MSA spelling as is undesirable. PAL
speakers who write in the dialect produce
spontaneousinconsistentspellings
that
sometimes reflect the phonology of PAL, andother times the word’s cognate relationship with
MSA. For example, the word for ‘heart’ (MSA!‫ ﻗﻠ‬qalb) has four spellings that correspond tofour sub-dialectal pronunciations: !‫ ﻗﻠ‬qlb /qalb/,!‫ﻟ‬# Âlb /’alb/, !‫ ﻛﻠ‬klb /kalb/, and !‫ ﺟﻠ‬jlb /galb/.
Similarly, the common shortening of some longvowels (from MSA to PAL) leads to different
orthographies as in !"‫ ﻗﺎﻧ‬qAnwn ‘law’ (MSA
/qānūn/), which can also be written with ashortened first vowel !"‫ ﻗﻧ‬qnwn /’anūn/reflecting the PAL pronunciation. PAL also hassome clitics that do not exist in MSA, whichleads to different spellings, e.g. the PAL future
particle ! H /Ha/ can be written attached to or
separate from the verb that follows it. Even
when a morpheme exists in MSA and PAL, it
may have additional forms or pronunciations.
One example is the definite article morpheme !"Al /il/ which has a non-MSA/non-EGY
allomorph /li/ when attached to nominals withinitial consonant clusters. As a result, a word
like /li+blād/ ‘the homeland/countries’ can bespelled to reflect the morphology as !‫ﻟﺑﻼ‬% AlblAd
or the phonology !‫ ﻟﺑﻼ‬lblAd, with the latter being
ambiguous with ‘for countries’ (in PAL/la+blād/). Finally, there are words in PAL thathave no cognate in MSA and as such have noFinally, we made one major simplification to
the annotations to minimize the load on thehuman annotator: we do not produce diacritizedmorphological analyses in the Buckwalter POS
tag. The reasons for this decision are thefollowing: (i) full diacritization is a complex task

23

clear obvious spelling to go with, e.g., the word
/barDo/ ‘additionally’ is spontaneously written
as !‫ﺿ‬#‫ ﺑ‬brDw, !‫ﺿ‬#‫ ﺑ‬brDh and ‫ﺿﺔ‬#‫ ﺑ‬brDħ.

In Tables 3 and 4 (column CODA), we show theresults of using the MADAMIRA-MSA and
MADAMIRA-EGY systems on a set of ten
words, while Table 5 shows the manually
selected or corrected CODA. MADAMIRA
generates a CODA version (contextually) by
default. We expect the EGY version to be more
successful than the MSA version in producing
the CODA for PAL given the shared presence ofmany morphemes in EGY and PAL. However,when we ran the same set of words through
MADAMIRA-EGY, we encountered manyerrors in words, morphemes and spelling choices
in PAL that are different from EGY, e.g., theraw word !‫ ﻣﻧﺣ‬mnHb ‘we love’ (CODA !‫ﺑﻧﺣ‬bnHb) is analyzed as the EGY !‫ ﻣﺎ ﻧﺣ‬mA nHb‘we do not love’!

This, of course, is not a unique PAL problem.
Researchers working on NLP for EGY andTunisian dialects developed CODA guidelines
for them (Habash et al., 2012a; Zribi et al.,2014). These guidelines were by design intended
to apply (or be easily extended) to all Arabic
dialects, but were only demonstrated for two.
Our challenge was to take these guidelines(specifically the EGY version) and extend them.
There were three types of extensions. First, in
terms of phonology-orthography, we added theletter ! k to the list of root letters to be spelled in
the MSA cognate to cover the PAL rural subdialects that pronounces it as /tš/. Second, in
terms of morphology, we added the non-EGY
demonstrative proclitic ! h+ and the conjunction
proclitic ! t+ ‘so as to’ to the list of clitics, e.g.,!"‫ ﺑ&ﺎﻟﺑ‬bhAlbyt ‘in this house’ and !"‫ﺷ‬$‫ ﺗ‬tyšwf ‘sothat he can see’. Finally, we extended the list of
exceptional words to cover problematic PAL
words. All of the basic CODA rules for EGY(and Tunisian) are kept the same.

5.4 Morphological Annotation Process andChallenges
To study the value of using an existing
morphological analyzer for MSA or EGY in
creating PAL annotations, we conducted thefollowing pilot study.
Pilot Study (II): We ran the words from arandomly selected episode of the PAL TV show
“Watan Aa Watar” (460 words) through both
MADAMIRA-MSA and MADAMIRA-EGY.
We analyzed the output from both systems todetermine its usability for PAL annotations. We
consider all analyses that are correct for PAL
annotation or usable via simple post processing(such as removing CASE endings on MSA
words) to be correct (as in word #2 in Tables 35). Words that receive incorrect analyses or no
analyses require manual modifications.

Pilot Study (I): We conducted a small pilot
study in annotating the CODA for PAL words.
We considered 1,000 words from 77 tweets in
Curras. The CODA version of each word wascreated in context. 15.9% of all words had adifferent CODA form from the input raw wordform. 42% of these changes involve consonants(two-fifths of the cases), vowels (one-fifth of thecases) and the hamzated/bare forms of the letter
Alif ! A. Examples of consonant change can beseen in Table 5 (words #4 and #10). An
additional 29% word changes involve thespelling of specific morpheme. The mostcommon change (over half of the time) was forthe first person imperfect verbal prefix ! A when
following the progressive particle ! b: !‫ ﺑﻛﺗ‬bktb
as opposed to !‫ ﺑﺎﻛﺗ‬bAktb. About 18% of thechanged words experience a split or a merge(with splits happening five time more thanmerges). An example of a CODA split is seen in
Table 5 (word #9). Finally, only about 8% of thechanged words were PAL specific terms; andless than 7% involved a typo or speech effectelongation. These results are quite encouraging
as they suggest the differences between CODA
and spontaneously written PAL are notextensive. Further analysis is still needed ofcourse.

The results of this experiment are summarized inTable 2. Table 3 and 4 illustrate sample results
for ten words and Table 5 includes the manuallycreated results.2
Table 2. Accuracy of automatic annotation of PAL text
Statistics
No Analysis
Wrongly Analyzed
Correctly Analyzed

MADAMIRA MSA MADAMIRA EGY
17.78%
7.24%
18.43%
14.75%
63.79%
78.01%The No Analysis (NA) words in Tables 2, 3 and4 refer to the words that the morphologicalanalyzer couldn't recognize. This failure may be
2The examples in Tables 3-5 are presented in the
Buckwalter transliteration (Buckwalter, 2004) to match theforms as they appear in the annotated corpus.

24

a result of missing lexical entry, specific PAL
morphologyortypos.
As
expected,
MADAMIRA-MSA had 2.5 times the number of
NA cases compared to MADAMIRA-EGY.
Examples include dialectal lexical terms (word#7) or dialectal morphology (words # 1 and #9).

analysis provided by MADAMIRA-EGY iscorrect for other contexts than the one illustrated
here. Another example is word #8, which is a
Levantine specific term hardly used in EGY and
not used at all in MSA. MADAMIRA-MSA has
a higher proportion of wrongly analyzed words
than MADAMIRA-EGY.

The wrongly analyzed words are words thatwere assigned incorrect POS tag in context. For
example, word #3 in Tables 3 and 4 is the resultof mis-analyzing the proclitic l- as the
preposition ‘for/to’ as opposed to the non-CODA
spelling of the definite article in PAL. The

Overall MADAMIRA-EGY produced analyses
that were either correct and ready to use for PAL
or requiring some minor modifications such as
adjusting the vowels on the lemmas (e.g., word#5) in one of every five words.

Table 3 Automatic annotations by the MADAMIRA-MSA system. Entries with Status NA had no analysis.
Raw

1 !‫!ﺑﻮﻛﻮ‬2 ‫ﻻﻛﻞ‬$3 !‫ﻟﺒﻨﻮ‬4 ‫&ﻟﺘﺎﻧﻲ‬5 !‫&ﻟﺤﻤﺎ‬6 ‫ﺗﺐ‬#‫ﻟﻠﺮ‬
7 !‫ﻮ‬#$
8 ‫ﺑﺪ"ﺎ‬9 ‫ﻟﻚ‬#‫ﺑﻨﺮ‬
10 !"‫ﺪ‬$

CODA

Lemma

Buckwalter POS (Diacritized)

Gloss

StatusAbwkwA
AlAkl‫ﻷﻛﻞ‬$ Al>kl>akol
Al/DET+>akol/NOUN+a/CASE_DEF_ACC
eatinglbnwk!‫ ﻟﺒﻨﻮ‬lbnwk
banok
li/PREP+bunuwk/NOUN+K/CASE_INDEF_GEN
bank
AltAny ‫ &ﻟﺘﺄﻧﻲ‬Alt>ny ta>an~iy Al/DET+ta>an~iy/NOUN
prudence
AlHmAr !‫ &ﻟﺤﻤﺎ‬AlHmAr HimAr Al/DET+HimAr/NOUN+u/CASE_DEF_NOM
donkeyllrAtb‫ﺗﺐ‬#‫ ﻟﻠﺮ‬llrAtb
rAtib
li/PREP+Al/DET+rAtib/NOUN+i/CASE_DEF_GEN
salaryAywp
bdhA‫ ﺑﺪ"ﺎ‬bdhA
bud~bud~/NOUN+i/CASE_DEF_GEN+hA/POSS_PRON_3FS escapebnrdlkhdwl

NAUsableWrong
Wrong
UsableUsable
NA
Wrong
NA
NA

Table 4 Automatic annotations by the MADAMIRA-EGY system. Entries with Status NA had no analysis.
Raw

CODA

Lemma

Buckwalter POS (Diacritized)

Gloss

Status1 !‫ !ﺑﻮﻛﻮ‬AbwkwA ‫ﺑﻮﻛﻮ‬$ Abwkw Abuw
Abuw/NOUN+kuw/POSS_PRON_3MS
father Correct
2 ‫ﻻﻛﻞ‬$ AlAkl‫ﻷﻛﻞ‬$ Al>kl>akl
Al/DET+>akol/NOUN
eating Correct
3 !‫ ﻟﺒﻨﻮ‬lbnwk!‫ ﻟﺒﻨﻮ‬lbnwk
bankli/PREP+bunuwk/NOUN
bank
Wrong
4 ‫ &ﻟﺘﺎﻧﻲ‬AltAny ‫ &ﻟﺘﺎﻧﻲ‬AltAny tAniy
Al/DET+tAniy/ADJ_NUM
second Usable
5 !‫ &ﻟﺤﻤﺎ‬AlHmAr !‫ &ﻟﺤﻤﺎ‬AlHmAr HumAr Al/DET+HumAr/NOUN
donkey Usable
6 ‫ﺗﺐ‬#‫ ﻟﻠﺮ‬llrAtb‫ﺗﺐ‬#‫ ﻟﻠﺮ‬llrAtb
rAtib
li/PREP+Al/DET+rAtib/NOUN
salary Correct
7 !‫ﻮ‬#$ Aywp!‫ﻮ‬#$>ywh>ayowah >ayowah/INTERJ
yes
Correct
8 ‫ ﺑﺪ"ﺎ‬bdhA‫ﺑﺪ"ﺎ‬
bdhA
bud~bud~/NOUN+hA/POSS_PRON_3FS
escape Wrong
9 ‫ﻟﻚ‬#‫ ﺑﻨﺮ‬bnrdlk ‫_ﻟﻚ‬#‫ ﺑﻨﺮ‬bnrd_lk rad~bi/PROG_PART+nu/IV1P+rud~/IV+li/PREP+ak/PRON_2MS answer Usable
10 !"‫ﺪ‬$ hdwl
NA
Table 5 Manual Annotations in Curras. Entries with Analysis MADA were automatically converted and validated by
the annotator. Entries with Analysis ANNO required some modification of the MADAMIRA output or were createdfrom scratch.
Raw

CODA

Lemma

Buckwalter POS (Undiacritized)

1 !‫ !ﺑﻮﻛﻮ‬AbwkwA ‫ﺑﻮﻛﻮ‬$ Abwkw Abuw
Abw/NOUN+kw/POSS_PRON_3MS
2 ‫ﻻﻛﻞ‬$ AlAkl‫ﻷﻛﻞ‬$ Al>kl>akl
Al/DET+>kl/NOUN
3 !‫ ﻟﺒﻨﻮ‬lbnwk!‫ &ﻟﺒﻨﻮ‬Albnwk bank
Al/DET+bnwk/NOUN
4 ‫ &ﻟﺘﺎﻧﻲ‬AltAny ‫ &ﻟﺜﺎﻧﻲ‬AlvAny vAniy
Al/DET+vAny/ADJ_NUM
5 !‫ &ﻟﺤﻤﺎ‬AlHmAr !‫ &ﻟﺤﻤﺎ‬AlHmAr HmAr Al/DET+HmAr/NOUN
6 ‫ﺗﺐ‬#‫ ﻟﻠﺮ‬llrAtb‫ﺗﺐ‬#‫ ﻟﻠﺮ‬llrAtb
rAtib
l/PREP+Al/DET+rAtb/NOUN
7 !‫ﻮ‬#$ Aywp!‫ﻮ‬#$>ywh>ayowah >ywh/INTERJ
8 ‫ ﺑﺪ"ﺎ‬bdhA‫ﺑﺪ"ﺎ‬
bdhA
bid~
bd/NOUN+hA/POSS_PRON_3FS
9 ‫ﻟﻚ‬#‫ ﺑﻨﺮ‬bnrdlk ‫_ﻟﻚ‬#‫ ﺑﻨﺮ‬bnrd_lk rad~b/PROG_PART+n/IV1P+rd/IV+l/PREP+k/PRON_2MS
10 !"‫ﺪ‬$ hdwl!"‫ﺬ‬$ h*wl
ha*A
h*wl/DEM_PRON

25

Gloss

Analysisfather MADA
eating MADA
bank
ANNO
second ANNO
donkey MADA
salary MADA
yes
MADA
want
ANNO
answer MADA
these ANNO5

M. C. Bateson. Arabic Language Handbook. Center
for Applied Linguistics, Washington D.C., USA,1967.
W. Black, Elkateb, S., & Vossen, P. (2006).
Introducing the Arabic wordnet project. In In
Proceedings of the third International WordNet
Conference (GWC-06).
K. Brustad. The Syntax of Spoken Arabic: A
Comparative Study of Moroccan, Egyptian,
Syrian, and Kuwaiti Dialects. Georgetown
University Press, 2000.
T. Buckwalter. Buckwalter Arabic morphologicalanalyzer version 2.0. LDC catalog number
LDC2004L02, ISBN 1-58563-324-0, 2004.
D. Chiang, M. Diab, N. Habash, O. Rambow, and S.
Shareef. Parsing Arabic Dialects. In Proceedingsof the European Chapter of ACL (EACL), 2006.
M. W. Cowell. A Reference Grammar of Syrian
Arabic. Georgetown University Press, 1964.
Kareem Darwish. Arabizi Detection and Conversion
to Arabic. In the Arabic Natural Language
Processing Workshop, EMNLP, Doha, Qatar,
2014.
M. Diab. Towards an Optimal POS tag set forModern Standard Arabic Processing. In Proc. of
Recent Advances in Natural Language Processing(RANLP), Borovets, Bulgaria, 2007.
M. Diab, N. Habash, O. Rambow, M. Altantawy,
and Y. Benajiba. COLABA: Arabic Dialect
Annotation and Processing. LREC Workshop on
Semitic Language Processing, Malta, 2010.
Y. Elihai. The olive tree dictionary: a transliterated
dictionary of conversational Eastern Arabic(Palestinian). Washington DC: Kidron Pub, 2004.
R. Eskander,
N. Habash,
O. Rambow,
andN. Tomeh. Processing Spontaneous Orthography.
In Proceedings NAACL-HLT, Atlanta, GA, 2013.
H. Gadalla, H. Kilany, H. Arram, A. Yacoub, A. ElHabashi, A. Shalaby, K. Karins, E. Rowson, R.
MacIntyre, P. Kingsbury, D. Graff, and C.
McLemore. CALLHOME Egyptian Arabic
Transcripts. Linguistic Data Consortium, Catalog
No.: LDC97T19, 1997.
D. Graff, M. Maamouri, B. Bouziri, S. Krouna, S.
Kulick, and T. Buckwalter. Standard ArabicMorphological Analyzer (SAMA) Version 3.1.
Linguistic Data Consortium LDC2009E73, 2009.
N. Habash and O. Rambow. Arabic Tokenization,
Part-of-Speech Tagging and Morphological
Disambiguation in One Fell Swoop. In ACL, Ann
Arbor, Michigan, 2005.
N. Habash, A. Soudi, and T. Buckwalter. On Arabic
Transliteration. In A. van den Bosch and A.
Soudi,
editors,
ArabicComputational
Morphology: Knowledge-based and Empirical
Methods. Springer, 2007.
N. Habash and R. Roth. CATiB: The Columbia
Arabic Treebank. In ACL, 2009.
N. Habash. Introduction to Arabic natural language
processing, volume 3. Morgan & Claypool
Publishers, 2010.

Conclusion and Future Work

We presented our preliminary results towards
building an annotated corpus of the PalestinianArabic dialect. The challenges and linguistic
variations of the Palestinian dialect, comparedwith Modern Standard Arabic, were discussed
especially in terms of morphology, orthography,
and lexicon. We also discussed and showed thepotential, and limitations, of using existing
resources, especially MADAMIRA-EGY, tosemi-automate and speed up the annotation
process.
The paper has also pointed out several issues thatneed to be considered and researched further,
especially the development of Palestinianspecific morphological annotation and CODA
guidelines, a Palestinian lexicon, and theextension of MADAMIRA to analyze
Palestinian text. Our corpus will be further
extended to include more text, and all lexical
annotations (i.e., Lemmas) will be linked withexisting Arabic ontology resources such as the
Arabic WordNet (Black et al., 2006). The corpus
will be publicly available for research purposes.

Acknowledgement
This work is part of the ongoing project Curras,
funded by the Palestinian Ministry of Higher
Education, Scientific Research Council. Nizar
Habash performed most of his work on thispaper while he was in the Center for
Computational Learning Systems at Columbia
University.

References
H. Abo Bakr, K. Shaalan, and I. Ziedan. A Hybrid
Approach for Converting Written Egyptian
Colloquial Dialect into Diacritized Arabic. In The
6th International Conference on Informatics andSystems, INFOS2008. Cairo University, 2008.
M. Al-Badrashiny, R. Eskander, N. Habash, andO. Rambow. Automatic Transliteration of
Romanized Dialectal Arabic. CoNLL, 2014.
S. Alkuhlani, N. Habash and R. M. Roth. AutomaticMorphological Enrichment of a Morphologically
Underspecified Treebank. In Proc. of Conferenceof the North American Association for
Computational Linguistics (NAACL), Atlanta,
Georgia, 2013.
R. Al-Sabbagh and R. Girju. YADAC: Yet anotherdialectal Arabic corpus. In Proc. of the Language
Resources and Evaluation Conference (LREC),
pages 2882–2889, Istanbul, 2012.

26

N. Habash, M. Diab, and O. Rabmow. (2012a)
Conventional Orthography for Dialectal Arabic.
In Proc. of LREC, Istanbul, Turkey, 2012.
N. Habash, R. Eskander, and A. Hawwari. (2012b)
A Morphological Analyzer for Egyptian Arabic.
In Proc. of the Special Interest Group on
Computational Morphology and Phonology,
Montréal, Canada, 2012.
N. Habash, R. Roth, O. Rambow, R. Eskander, andN. Tomeh. Morphological Analysis andDisambiguation for Dialectal Arabic. In Proc. of
NAACL, Atlanta, Georgia, 2013.
M. Halloun. A Practical Dictionary of the Standard
Dialect Spoken in Palestine. Bethlehem
University, 2000.
R. Herzallah. Aspects of Palestinian Arabic
Phonology: A Nonlinear Approach. Ph.D. thesis,
Cornell University. Distributed as Working Papersof the Cornell Phonetics Laboratory No. 4, 1990.
C. Holes. Modern Arabic: Structures, Functions,and Varieties. Georgetown Classics in Arabic
Language and Linguistics. Georgetown University
Press, 2004.
H. Kilany, H. Gadalla, H. Arram, A. Yacoub, A. ElHabashi, and C. McLemore. Egyptian Colloquial
Arabic Lexicon. Linguistic Data Consortium,
Catalog No.: LDC99L22, 1999.
M. Maamouri, A. Bies, T. Buckwalter, M. Diab,
N. Habash,
O. Rambow,
and
D. Tabessi.
Developing and using a pilot dialectal Arabic
treebank. In Proc. of LREC, Genoa, Italy, 2006.
M. Maamouri,
A. Bies,T. Buckwalter,
andW. Mekki. The Penn Arabic Treebank: Building aLarge-Scale Annotated Arabic Corpus. In
NEMLAR Conference on Arabic Language
Resources and Tools, Cairo, Egypt, 2004.
M. Maamouri, A. Bies, S. Kulick, M. Ciul, N.
Habash and R. Eskander. Developing an Egyptian
Arabic Treebank: Impact of Dialectal Morphology
on Annotation and Tool Development. In Proc. of
LREC, Reykjavik, Iceland, 2014.
M. Maamouri, and C. Cieri. Resources for Arabic
Natural Language Processing at the Linguistic
Data Consortium. In Proc. of the International
Symposium on Processing of Arabic. Faculté desLettres, University of Manouba, Tunisia, 2002.
M. Omar. Levantine and Egyptian Arabic:
Comparative Study. Foreign Service Institute.
Basic Course Series, 1976.
A. Pasha, M. Al-Badrashiny, M. Diab, A. El Kholy,
R. Eskander, N. Habash, M. Pooleery, O.
Rambow and R. M. Roth. MADAMIRA: A Fast,
Comprehensive Tool for Morphological Analysis
and Disambiguation of Arabic. In Proc. of LREC,
Reykjavik, Iceland, 2014.
O. Rambow, D. Chiang, M. Diab, N. Habash, R.
Hwa, K. Sima’an, V. Lacey, R. Levy, C. Nichols,
and S. Shareef. 2005. Parsing Arabic Dialects.
Final Report, 2005 JHU Summer Workshop.
F. Rice and M. Sa'id. Eastern Arabic: an
introduction to the spoken Arabic of Palestine,
Syria and Lebanon. Beirut: Khayat's 1960.

W. Salloum and N. Habash. Dialectal to Standard
Arabic Paraphrasing to Improve Arabic-English
Statistical Machine Translation. In Proc. of theFirst Workshop on Algorithms and Resources for
Modeling of Dialects and Language Varieties,
Edinburgh, Scotland, 2011.
O. Smrž and J. Hajič. The Other Arabic Treebank:Prague Dependencies and Functions. In Ali
Farghaly,
editor,ArabicComputational
Linguistics. CSLI Publications, 2006.
I. Zribi, R. Boujelbane, A. Masmoudi, M. Ellouze
Khmekhem, L. Hadrich Belguith, and N. Habash.
A Conventional Orthography for Tunisian Arabic.
In Proc. of LREC, Reykjavik, Iceland, 2014.

27