Introduction
The proliferation of social media has made it possible to study large online communities at scale, thus making important discoveries that can facilitate decision making, guide policies, improve health and well-being, aid disaster response, etc. The wide host of languages, languages varieties, and dialects used on social media and the nuanced differences between users of various backgrounds (e.g., different age groups, gender identities) make it especially difficult to derive sufficiently valuable insights based on single prediction tasks. For these reasons, it would be desirable to offer NLP tools that can help stitch together a complete picture of an event across different geographical regions as impacting, and being impacted by, individuals of different identities. We offer AraNet as one such tool for Arabic social media processing.
Introduction ::: 
For Arabic, a collection of languages and varieties spoken by a wide population of $\sim 400$ million native speakers covering a vast geographical region (shown in Figure FIGREF2), no such suite of tools currently exists. Many works have focused on sentiment analysis, e.g., BIBREF0, BIBREF1, BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6, BIBREF7 and dialect identification BIBREF8, BIBREF9, BIBREF10, BIBREF11, BIBREF12, BIBREF13. However, there is generally rarity of resources on other tasks such as gender and age detection. This motivates our toolkit, which we hope can meet the current critical need for studying Arabic communities online. This is especially valuable given the waves of protests, uprisings, and revolutions that have been sweeping the region during the last decade.
Although we create new models for tasks such as sentiment analysis and gender detection as part of AraNet, our focus is more on putting together the toolkit itself and providing strong baselines that can be compared to. Hence, although we provide some baseline models for some of the tasks, we do not explicitly compare to previous research since most existing works either exploit smaller data (and so it will not be a fair comparison), use methods pre-dating BERT (and so will likely be outperformed by our models) . For many of the tasks we model, there have not been standard benchmarks for comparisons across models. This makes it difficult to measure progress and identify areas worthy of allocating efforts and budgets. As such, by publishing our toolkit models, we believe model-based comparisons will be one way to relieve this bottleneck. For these reasons, we also package models from our recent works on dialect BIBREF12 and irony BIBREF14 as part of AraNet .
The rest of the paper is organized as follows: In Section SECREF2 we describe our methods. In Section SECREF3, we describe or refer to published literature for the dataset we exploit for each task and provide results our corresponding model acquires. Section SECREF4 is about AraNet design and use, and we overview related works in Section SECREF5 We conclude in Section SECREF6
Methods
Supervised BERT. Across all our tasks, we use Bidirectional Encoder Representations from Transformers (BERT). BERT BIBREF15, dispenses with recurrence and convolution. It is based on a multi-layer bidirectional Transformer encoder BIBREF16, with multi-head attention. It uses masked language models to enable pre-trained deep bidirectional representations, in addition to a binary next sentence prediction task. The pre-trained BERT can be easily fine-tuned on a wide host of sentence-level and token-level tasks. All our models are trained in a fully supervised fashion, with dialect id being the only task where we leverage semi-supervised learning. We briefly outline our semi-supervised methods next.
Self-Training. Only for the dialect id task, we investigate augmenting our human-labeled training data with automatically-predicted data from self-training. Self-training is a wrapper method for semi-supervised learning BIBREF17, BIBREF18 where a classifier is initially trained on a (usually small) set of labeled samples $\textbf {\textit {D}}^{l}$, then is used to classify an unlabeled sample set $\textbf {\textit {D}}^{u}$. Most confident predictions acquired by the original supervised model are added to the labeled set, and the model is iteratively re-trained. We perform self-training using different confidence thresholds and choose different percentages from predicted data to add to our train. We only report best settings here and the reader is referred to our winning system on the MADAR shared task for more details on these different settings BIBREF12.
Implementation & Models Parameters. For all our tasks, we use the BERT-Base Multilingual Cased model released by the authors . The model is trained on 104 languages (including Arabic) with 12 layer, 768 hidden units each, 12 attention heads, and has 110M parameters in entire model. The model has 119,547 shared WordPieces vocabulary, and was pre-trained on the entire Wikipedia for each language. For fine-tuning, we use a maximum sequence size of 50 tokens and a batch size of 32. We set the learning rate to $2e-5$ and train for 15 epochs and choose the best model based on performance on a development set. We use the same hyper-parameters in all of our BERT models. We fine-tune BERT on each respective labeled dataset for each task. For BERT input, we apply WordPiece tokenization, setting the maximal sequence length to 50 words/WordPieces. For all tasks, we use a TensorFlow implementation. An exception is the sentiment analysis task, where we used a PyTorch implementation with the same hyper-parameters but with a learning rate $2e-6$.
Pre-processing. Most of our training data in all tasks come from Twitter. Exceptions are in some of the datasets we use for sentiment analysis, which we point out in Section SECREF23. Our pre-processing thus incorporates methods to clean tweets, other datasets (e.g., from the news domain) being much less noisy. For pre-processing, we remove all usernames, URLs, and diacritics in the data.
Data and Models ::: Age and Gender
Arab-Tweet. For modeling age and gender, we use Arap-Tweet BIBREF19 , which we will refer to as Arab-Tweet. Arab-tweet is a tweet dataset of 11 Arabic regions from 17 different countries. For each region, data from 100 Twitter users were crawled. Users needed to have posted at least 2,000 and were selected based on an initial list of seed words characteristic of each region. The seed list included words such as <برشة> /barsha/ ‘many’ for Tunisian Arabic and <وايد> /wayed/ ‘many’ for Gulf Arabic. BIBREF19 employed human annotators to verify that users do belong to each respective region. Annotators also assigned gender labels from the set male, female and age group labels from the set under-25, 25-to34, above-35 at the user-level, which in turn is assigned at tweet level. Tweets with less than 3 words and re-tweets were removed. Refer to BIBREF19 for details about how annotation was carried out. We provide a description of the data in Table TABREF10. Table TABREF10 also provides class breakdown across our splits.We note that BIBREF19 do not report classification models exploiting the data.
Data and Models ::: Age and Gender ::: 
We shuffle the Arab-tweet dataset and split it into 80% training (TRAIN), 10% development (DEV), and 10% test (TEST). The distribution of classes in our splits is in Table TABREF10. For pre-processing, we reduce 2 or more consecutive repetitions of the same character into only 2 and remove diacritics. With this dataset, we train a small unidirectional GRU (small-GRU) with a single 500-units hidden layer and $dropout=0.5$ as a baseline. Small-GRU is trained with the TRAIN set, batch size = 8, and up to 30 words of each sequence. Each word in the input sequence is represented as a trainable 300-dimension vector. We use the top 100K words which are weighted by mutual information as our vocabulary in the embedding layer. We evaluate the model on TEST set. Table TABREF14 show small-GRU obtain36.29% XX acc on age classification, and 53.37% acc on gender detection. We also report the accuracy of fine-tuned BERT models on TEST set in Table TABREF14. We can find that BERT models significantly perform better than our baseline on the two tasks. It improve with 15.13% (for age) and 11.93% acc (for gender) over the small-GRU.
UBC Twitter Gender Dataset. We also develop an in-house Twitter dataset for gender. We manually labeled 1,989 users from each of the 21 Arab countries. The data had 1,246 “male", 528 “female", and 215 unknown users. We remove the “unknown" category and balance the dataset to have 528 from each of the two `male" and “female" categories. We ended with 69,509 tweets for `male" and 67,511 tweets for “female". We split the users into 80% TRAIN set (110,750 tweets for 845 users), 10% DEV set (14,158 tweets for 106 users), and 10% TEST set (12,112 tweets for 105 users). We, then, model this dataset with BERT-Base, Multilingual Cased model and evaluate on development and test sets. Table TABREF15 shows that fine-tuned model obtains 62.42% acc on DEV and 60.54% acc on TEST.
We also combine the Arab-tweet gender dataset with our UBC-Twitter dataset for gender on training, development, and test, respectively, to obtain new TRAIN, DEV, and TEST. We fine-tune the BERT-Base, Multilingual Cased model with the combined TRAIN and evaluate on combined DEV and TEST. As Table TABREF15 shows, the model obtains 65.32% acc on combined DEV set, and 65.32% acc on combined TEST set. This is the model we package in AraNet .
Data and Models ::: Dialect
The dialect identification model in AraNet is based on our winning system in the MADAR shared task 2 BIBREF20 as described in BIBREF12. The corpus is divided into train, dev and test, and the organizers masked test set labels. We lost some tweets from training data when we crawled using tweet ids, ultimately acquiring 2,036 (TRAIN-A), 281 (DEV) and 466 (TEST). We also make use of the task 1 corpus (95,000 sentences BIBREF21). More specifically, we concatenate the task 1 data to the training data of task 2, to create TRAIN-B. Again, note that TEST labels were only released to participants after the official task evaluation. Table TABREF17 shows statistics of the data. We used tweets from 21 Arab countries as distributed by task organizers, except that we lost some tweets when we crawled using tweet ids. We had 2,036 (TRAIN-A), 281 (DEV) and 466 (TEST). For our experiments, we also make use of the task 1 corpus (95,000 sentences BIBREF21). More specifically, we concatenate the task 1 data to the training data of task 2, to create TRAIN-B. Note that both DEV and TEST across our experiments are exclusively the data released in task 2, as described above. TEST labels were only released to participants after the official task evaluation. Table TABREF17 shows statistics of the data. More information about the data is in BIBREF21. We use TRAIN-A to perform supervised modeling with BERT and TRAIN-B for self training, under various conditions. We refer the reader to BIBREF12 for more information about our different experimental settings on dialect id. We acquire our best results with self-training, with a classification accuracy of 49.39% and F1 score at 35.44. This is the winning system model in the MADAR shared task and we showed in BIBREF12 that our tweet-level predictions can be ported to user-level prediction. On user-level detection, our models perform superbly, with 77.40% acc and 71.70% F1 score on unseen MADAR blind test data.
Data and Models ::: Emotion
We make use of two datasets, the LAMA-DINA dataset from BIBREF22, a Twitter dataset with a combination of gold labels from BIBREF23 and distant supervision labels. The tweets are labeled with the Plutchik 8 primary emotions from the set: {anger, anticipation, disgust, fear, joy, sadness, surprise, trust}. The distant supervision approach depends on use of seed phrases with the Arabic first person pronoun انا> (Eng. “I") + a seed word expressing an emotion, e.g., فرحان> (Eng. “happy"). The manually labeled part of the data comprises tweets carrying the seed phrases verified by human annotators $9,064$ tweets for inclusion of the respective emotion. The rest of the dataset is only labeled using distant supervision (LAMA-DIST) ($182,605$ tweets) . For more information about the dataset, readers are referred to BIBREF22. The data distribution over the emotion classes is in Table TABREF20. We combine LAMA+DINA and LAMA-DIST training set and refer to this new training set as LAMA-D2 (189,903 tweets). We fine-tune BERT-Based, Multilingual Cased on the LAMA-D2 and evaluate the model with same DEV and TEST sets from LAMA+DINA. On DEV set, the fine-tuned BERT model obtains 61.43% on accuracy and 58.83 on $F_1$ score. On TEST set, we acquire 62.38% acc and 60.32% $F_1$ score.
Data and Models ::: Irony
We use the dataset for irony identification on Arabic tweets released by IDAT@FIRE2019 shared-task BIBREF24. The shared task dataset contains 5,030 tweets related to different political issues and events in the Middle East taking place between 2011 and 2018. Tweets are collected using pre-defined keywords (i.e., targeted political figures or events) and the positive class involves ironic hashtags such as #sokhria, #tahakoum, and #maskhara (Arabic variants for “irony"). Duplicates, retweets, and non-intelligible tweets are removed by organizers. Tweets involve both MSA as well as dialects at various degrees of granularity such as Egyptian, Gulf, and Levantine.
IDAT@FIRE2019 BIBREF24 is set up as a binary classification task where tweets are assigned labels from the set {ironic, non-ironic}. A total of 4,024 tweets were released by organizers as training data. In addition, 1,006 tweets were used by organizers as test data. Test labels were not release; and teams were expected to submit the predictions produced by their systems on the test split. For our models, we split the 4,024 released training data into 90% TRAIN ($n$=3,621 tweets; `ironic'=1,882 and `non-ironic'=1,739) and 10% DEV ($n$=403 tweets; `ironic'=209 and `non-ironic'=194). We use the same small-GRU architecture of Section 3.1 as our baselines. We fine-tune BERT-Based, Multilingual Cased model on our TRAIN, and evaluate on DEV. The small-GRU obtain 73.70% accuracy and 73.47% $F_1$ score. BERT model significantly out-performance than small-GRU, which achieve 81.64% accuracy and 81.62% $F_1$ score.
Data and Models ::: Sentiment
We collect 15 datasets related to sentiment analysis of Arabic, including MSA and dialects BIBREF25, BIBREF26, BIBREF27, BIBREF1, BIBREF28, BIBREF29, BIBREF30, BIBREF31, BIBREF32, BIBREF33, BIBREF34. Table TABREF28 shows all the corpora we use. These datasets involve different types of sentiment analysis tasks such as binary classification (i.e., negative or positive), 3-way classification (i.e., negative, neutral, or positive), and subjective language detection. To combine these datasets for binary sentiment classification, we normalize different types of label to binary labels in the set $\lbrace `positive^{\prime }, `negative^{\prime }\rbrace $ by following rules:
{Positive, Pos, or High-Pos} to `positive';
{Negative, Neg, or High-Neg} to `negative';
Exclude samples which label is not `positive' or `negative' such as `obj', `mixed', `neut', or `neutral'.
After label normalization, we obtain 126,766 samples. We split this datase into 80% training (TRAIN), 10% development (DEV), and 10% test (TEST). The distribution of classes in our splits is presented in Table TABREF27. We fine-tune pre-trained BERT on the TRAIN set using PyTorch implementation with $2e-6$ learning rate and 15 epochs, as explained in Section SECREF2. Our best model on the DEV set obtains 80.24% acc and 80.24% $F_1$. We evaluate this best model on TEST set and obtain 77.31% acc and 76.67% $F_1$.
AraNet Design and Use
AraNet consists of identifier tools including age, gender, dialect, emotion, irony and sentiment. Each tool comes with an embedded model. The tool comes with modules for performing normalization and tokenization. AraNet can be used as a Python library or a command-line tool:
Python Library: Importing AraNet module as a Python library provides identifiers’ functions. Prediction is based on a text or a path to a file and returns the identified class label. It also returns the probability distribution over all available class labels if needed. Figure FIGREF34 shows two examples of using the tool as Python library.
Command-line tool: AraNet provides scripts supporting both command-line and interactive mode. Command-line mode accepts a text or file path. Interaction mode is good for quick interactive line-by-line experiments and also pipeline redirections.
AraNet is available through pip or from source on GitHub with detailed documentation.
Related Works
As we pointed out earlier, there has been several works on some of the tasks but less on others. By far, Arabic sentiment analysis has been the most popular task. Several works have been performed for MSA BIBREF35, BIBREF0 and dialectal BIBREF1, BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6, BIBREF7 sentiment analysis. A number of works have also been published for dialect detection, including BIBREF9, BIBREF10, BIBREF8, BIBREF11. Some works have been performed on the tasks of age detection BIBREF19, BIBREF36, gender detection BIBREF19, BIBREF36, irony identification BIBREF37, BIBREF24, and emotion analysis BIBREF38, BIBREF22.
A number of tools exist for Arabic natural language processing,including Penn Arabic treebank BIBREF39, POS tagger BIBREF40, BIBREF41, Buckwalter Morphological Analyzer BIBREF42 and Mazajak BIBREF7 for sentiment analysis .
Conclusion
We presented AraNet, a deep learning toolkit for a host of Arabic social media processing. AraNet predicts age, dialect, gender, emotion, irony, and sentiment from social media posts. It delivers state-of-the-art and competitive performance on these tasks and has the advantage of using a unified, simple framework based on the recently-developed BERT model. AraNet has the potential to alleviate issues related to comparing across different Arabic social media NLP tasks, by providing one way to test new models against AraNet predictions. Our toolkit can be used to make important discoveries on the wide region of the Arab world, and can enhance our understating of Arab online communication. AraNet will be publicly available upon acceptance.

Question: Did they experiment on all the tasks?
Answer: Yes

Question: What models did they compare to?
Answer:  we do not explicitly compare to previous research since most existing works either exploit smaller data (and so it will not be a fair comparison), use methods pre-dating BERT (and so will likely be outperformed by our models)

Question: What datasets are used in training?
Answer: Arap-Tweet BIBREF19 , an in-house Twitter dataset for gender, the MADAR shared task 2 BIBREF20, the LAMA-DINA dataset from BIBREF22, LAMA-DIST, Arabic tweets released by IDAT@FIRE2019 shared-task BIBREF24, BIBREF25, BIBREF26, BIBREF27, BIBREF1, BIBREF28, BIBREF29, BIBREF30, BIBREF31, BIBREF32, BIBREF33, BIBREF34

 Arap-Tweet , UBC Twitter Gender Dataset, MADAR , LAMA-DINA , IDAT@FIRE2019, 15 datasets related to sentiment analysis of Arabic, including MSA and dialects