Arabic NLP: Unique Challenges and Their Solutions | by Richard Pelgrim | Towards Data Science

In this article, I provide a concise and to-the-point overview of the challenges of working with Arabic text in NLP projects...and the tools available to overcome them. I rely heavily on the camel-tools Python package developed at the NYU Abu Dhabi CAMeL Lab and this excellent webinar by its director, Dr. Nizar Habash. Big shout-out to them for doing groundbreaking work in the field and making their tools accessible to the public!
Working with Arabic text in NLP projects presents (at least) 5 unique challenges:
The form of characters and spelling of words can vary depending on their context (fancy term: Orthographic Ambiguity)The same verb can have thousands (literally) of different forms (fancy term: Morphological Richness)There are many dialects of Arabic and there are big differences between them (Dialectal Variation)Since Arabic is a phonetic language (what you write is what you say), there can be different ways to write the same word when writing in dialectal Arabic, for which there is no agreed-upon standard (Orthographic Inconsistency).
The form of characters and spelling of words can vary depending on their context (fancy term: Orthographic Ambiguity)
The same verb can have thousands (literally) of different forms (fancy term: Morphological Richness)
There are many dialects of Arabic and there are big differences between them (Dialectal Variation)
Since Arabic is a phonetic language (what you write is what you say), there can be different ways to write the same word when writing in dialectal Arabic, for which there is no agreed-upon standard (Orthographic Inconsistency).
These four different characteristics of the Arabic language all contribute to data sparsity. Because of the many different forms of words, the potential different spellings, and the high number of dialects, one could end up with an NLP vocabulary running into the millions of words. This is not an overstatement: a single verb in Arabic can have up to 5400 forms (!) compared to a maximum of 6 in English and only 1 in Chinese.
But fear not! Right when you’re stuck in the Desert of Low-Resource Languages and your pipeline perched for some Liquid Code to keep you going...there materialises a magical, user-friendly CAMeL** to help you out:
pip install camel-tools
** we’ll save our post-colonial commentary on the blatantly self-orientalising name of this package for another day, shall we?
Below I’ll share the steps I took to pre-process my Arabic text and deal with the unique challenges posed by the Arabic language using the camel-tools package.
NOTE: I’m assuming that you’ve already done all the basic, universal NLP pre-processing such as removing repeating characters, stop words, emoji, hashtags, digits, and any of the other best-practice data cleaning tasks that are part of any NLP project. The Github repo referenced below contains a comprehensive list of Arabic stop words that account for the orthographic ambiguity mentioned earlier. Important note: make sure to add the word “يا” to the file as this is not currently included and is a very common stop word.
Note that NLTK also contains a library of Arabic stop words but this collection misses many of these common spelling inconsistencies.
github.com
Alright, now let’s get crackin’!
The first step is to cut down some serious data sparsity by removing the diacritics of the text. Diacritics are the symbols (in some cases comparable to vowels in the English language) that are located above or below the letters of your Arabic text — the blue marks in the image below.
The same word may have different diacritics depending on its context and therefore it is common practice to remove these symbols to reduce data sparsity.
Technically, this actually creates a new problem because the same word-root can have totally different meanings depending on the diacritics you give it...but we’ll deal with that in STEP 4.
To account for a number of common spelling inconsistencies across dialects (and because of the gap between spoken and written Arabic in general), the next step is to reduce orthographic ambiguity. Specifically, camel-tools does this by removing particular symbols from particular letters (the dots from the teh-marbuta and the hamza from the alef).
The next step is just a simple word tokenizer. We need this in order to be able to input our text into the functions of our next step.
Now this is where things get interesting. Remember how I said at the end of STEP 1 that removing the diacritics actually creates a new problem? We now have just the root characters but, technically, no way of knowing which of the many different words it could be. For example, the word below could alternatively mean: ‘and with our contract / necklace / psychoses’ or ‘and he stresses us out’ — depending on the diacritics used, the context and the dialect it’s written in.
So...which form of the word should we choose?
The camel-tools package comes with a nifty ‘morphological analyzer’ which — in a nutshell — compares any word you give it to a morphological database (it comes with one built-in) and outputs a complete analysis of the possible forms and meanings of the word, including the lemma, part of speech, English translation if available, etc.
Below we run a morphological analysis on the word ‘وبعقدنا’.
I’ve included a portion of the output below; these are only the top 3 of over 20 analyses that it returns. Notice that it also ‘disambiguates’ the word by restoring diacritics.
{'diac': 'وَبِعُقَدُنا', 'lex': 'عُقْدَة_1', 'bw': 'وَ/PART+بِ/PREP+عُقَد/NOUN+ُ/CASE_DEF_NOM+نا/POSS_PRON_1P', 'gloss': '[part.]_+_by;with+complexes+our', 'pos': 'noun', 'prc3': '0', 'prc2': 'wa_part', 'prc1': 'bi_prep', 'prc0': '0', 'per': 'na', 'asp': 'na', 'vox': 'na', 'mod': 'na', 'stt': 'c', 'cas': 'n', 'enc0': '1p_poss', 'rat': 'i', 'source': 'lex', 'form_gen': 'm', 'form_num': 's', 'pattern': 'وَبِ1ُ2َ3ُنا', 'root': 'ع.ق.د', 'catib6': 'PRT+PRT+NOM+NOM', 'ud': 'PART+ADP+NOUN+PRON', 'd1seg': 'وَبِعُقَدُنا', 'd1tok': 'وَ+_بِعُقَدُنا', 'atbseg': 'وَ+_بِ+_عُقَدُ_+نا', 'd3seg': 'وَ+_بِ+_عُقَدُ_+نا', 'd2seg': 'وَ+_بِ+_عُقَدُنا', 'd2tok': 'وَ+_بِ+_عُقَدُنا', 'atbtok': 'وَ+_بِ+_عُقَدُ_+نا', 'd3tok': 'وَ+_بِ+_عُقَدُ_+نا', 'bwtok': 'وَ+_بِ+_عُقَد_+ُ_+نا', 'pos_lex_logprob': -4.923429, 'caphi': 'w_a_b_i_3_u_q_a_d_u_n_aa', 'pos_logprob': -0.4344233, 'gen': 'f', 'lex_logprob': -4.923429, 'num': 'p', 'stem': 'عُقَد', 'stemgloss': 'complexes', 'stemcat': 'N'} {'diac': 'وَبِعُقَدنا', 'lex': 'عُقْدَة_1', 'bw': 'وَ/PART+بِ/PREP+عُقَد/NOUN+نا/POSS_PRON_1P', 'gloss': '[part.]_+_by;with+complexes+our', 'pos': 'noun', 'prc3': '0', 'prc2': 'wa_part', 'prc1': 'bi_prep', 'prc0': '0', 'per': 'na', 'asp': 'na', 'vox': 'na', 'mod': 'na', 'stt': 'c', 'cas': 'u', 'enc0': '1p_poss', 'rat': 'i', 'source': 'lex', 'form_gen': 'm', 'form_num': 's', 'pattern': 'وَبِ1ُ2َ3نا', 'root': 'ع.ق.د', 'catib6': 'PRT+PRT+NOM+NOM', 'ud': 'PART+ADP+NOUN+PRON', 'd1seg': 'وَبِعُقَدنا', 'd1tok': 'وَ+_بِعُقَدنا', 'atbseg': 'وَ+_بِ+_عُقَد_+نا', 'd3seg': 'وَ+_بِ+_عُقَد_+نا', 'd2seg': 'وَ+_بِ+_عُقَدنا', 'd2tok': 'وَ+_بِ+_عُقَدنا', 'atbtok': 'وَ+_بِ+_عُقَد_+نا', 'd3tok': 'وَ+_بِ+_عُقَد_+نا', 'bwtok': 'وَ+_بِ+_عُقَد_+نا', 'pos_lex_logprob': -4.923429, 'caphi': 'w_a_b_i_3_u_q_a_d_n_aa', 'pos_logprob': -0.4344233, 'gen': 'f', 'lex_logprob': -4.923429, 'num': 'p', 'stem': 'عُقَد', 'stemgloss': 'complexes', 'stemcat': 'N'} {'diac': 'وَبِعُقَدِنا', 'lex': 'عُقْدَة_1', 'bw': 'وَ/PART+بِ/PREP+عُقَد/NOUN+ِ/CASE_DEF_GEN+نا/POSS_PRON_1P', 'gloss': '[part.]_+_by;with+complexes+our', 'pos': 'noun', 'prc3': '0', 'prc2': 'wa_part', 'prc1': 'bi_prep', 'prc0': '0', 'per': 'na', 'asp': 'na', 'vox': 'na', 'mod': 'na', 'stt': 'c', 'cas': 'g', 'enc0': '1p_poss', 'rat': 'i', 'source': 'lex', 'form_gen': 'm', 'form_num': 's', 'pattern': 'وَبِ1ُ2َ3ِنا', 'root': 'ع.ق.د', 'catib6': 'PRT+PRT+NOM+NOM', 'ud': 'PART+ADP+NOUN+PRON', 'd1seg': 'وَبِعُقَدِنا', 'd1tok': 'وَ+_بِعُقَدِنا', 'atbseg': 'وَ+_بِ+_عُقَدِ_+نا', 'd3seg': 'وَ+_بِ+_عُقَدِ_+نا', 'd2seg': 'وَ+_بِ+_عُقَدِنا', 'd2tok': 'وَ+_بِ+_عُقَدِنا', 'atbtok': 'وَ+_بِ+_عُقَدِ_+نا', 'd3tok': 'وَ+_بِ+_عُقَدِ_+نا', 'bwtok': 'وَ+_بِ+_عُقَد_+ِ_+نا', 'pos_lex_logprob': -4.923429, 'caphi': 'w_a_b_i_3_u_q_a_d_i_n_aa', 'pos_logprob': -0.4344233, 'gen': 'f', 'lex_logprob': -4.923429, 'num': 'p', 'stem': 'عُقَد', 'stemgloss': 'complexes', 'stemcat': 'N'}
Now, in my case I’m working with over 6 million Tweets so running a word-by-word analysis is not going to be very informative or effective. Instead, we can use the Morphological Disambiguator (watch out, Arnold!) to do this for us. This will take a list of tokens as input (hence the simple word tokenizer in STEP 3) and output all the disambiguated forms of the tokens. Each analysis is a dictionary and we can access the forms and components we want using the dictionary keys. The analyses are listed from most likely to least likely, so common practice is to simplify specify the first analysis as your output.
Which outputs the diacritized form, Part-Of-Speech tag, and lemma for each word in our sentence:
('نَجَحَ', 'verb', 'نَجَح-a_1')('بايدن', 'noun_prop', 'بايدن_0')('فِي', 'prep', 'فِي_1')('الاِنْتِخاباتِ', 'noun', 'ٱِنْتِخاب_1')
This means we could, for example, get all of the lemmas of our Arabic text using a function such as the one below:
For my project which conducts Topic Modelling, this is the way I chose to conduct my tokenization. Other projects may call for different approaches using the Morphological Tokenizer, which will tokenize the strings differently depending on the scheme you select.
So... just find the tokenization / lemmatization approach that best fits your project, execute it, and then you’re all done pre-processing your Arabic text!
The camel-tools package offers more features for pre-processing (such as transliteration, unicode normalization, etc.) that may be useful for your project. Check out the complete documentation for more info.
camel-tools.readthedocs.io
Also, I just want to point out that I did my fair share of research trying to find the best tool for pre-processing Arabic text for NLP. And while there are a number of other good options out there (like Farasa, MADAMIRA and Stanford CoreNLP) I found camel-tools to be the most versatile, comprehensive, and easy to use. Besides the core pre-processing functions for dealing with the challenges mentioned above, it also comes with some cool bonus features: it can perform sentiment analysis and — this is really one of the coolest features —it can identify the dialect in which the text is written (out of a maximum of 25 dialects). I’ll definitely be using the Dialect Identifier to add ‘dialect’ as a feature in my NLP pipeline.
Thank you for reading! Follow me on LinkedIn for regular distributed computing updates and hacks.
And please consider becoming a Medium member to support the writing community: