SlowGuess/NAACL_2021_OCR huggingface dataset:
Abdelah El Mekki
Ismail Berrada<sup>1</sup>
Abdelkader El Mahdaouy
Ahmed Khoumsi²
$^{1}$ School of Computer Sciences, Mohammed VI Polytechnic University, Morocco
$^{2}$ Dept. Electrical & Computer Engineering, University of Sherbrooke, Canada
{firstname.lastname}@um6p.ma
ahmed.khoumsi@usherbrooke.ca
Finetuning deep pre-trained language models has shown state-of-the-art performances on a wide range of Natural Language Processing (NLP) applications. Nevertheless, their generalization performance drops under domain shift. In the case of Arabic language, diglossia makes building and annotating corpora for each dialect and/or domain a more challenging task. Unsupervised Domain Adaptation tackles this issue by transferring the learned knowledge from labeled source domain data to unlabeled target domain data. In this paper, we propose a new unsupervised domain adaptation method for Arabic cross-domain and cross-dialect sentiment analysis from Contextualized Word Embedding. Several experiments are performed adopting the coarse-grained and the fine-grained taxonomies of Arabic dialects. The obtained results show that our method yields very promising results and outperforms several domain adaptation methods for most of the evaluated datasets. On average, our method increases the performance by an improvement rate of  $20.8\%$  over the zero-shot transfer learning from BERT.
The Arabic language is characterized by two main language varieties: Modern Standard Arabic (MSA) and Arabic dialect. MSA has a standard written form and acquires an official status across the Arab countries, while Dialectal Arabic refers to the informal spoken dialects in the Arab World (Habash, 2010). These dialects are used in daily life but have no standard written form (Saadane and Habash, 2015; Habash et al., 2018; Eryani et al., 2020). Geographically and according to (Zaidan and Callison-Burch, 2014), Arabic dialects can be classified into five coarse-grained regional dialects: Egyptian, Levantine, Gulf, Iraqi, and Maghrebi. Recent studies have categorized dialectal Arabic into more fine-grained levels, including countries and cities (Bouamor et al., 2019;
Muhammad et al., 2020). These dialects differ from one another and from MSA, to a varying degree, at different linguistic levels (Salameh et al., 2018; Erdmann et al., 2018).
With the unprecedented reach of social media platforms, Sentiment Analysis (SA) has become a fundamental task for many applications. Most research work in this area has been devoted to English and other European languages, while some research studies have addressed the question of transfer learning from MSA to dialectal Arabic. However, Khaddaj et al. (2019) and Qwaider et al. (2019) have shown that zero-shot transfer learning, from models trained on MSA data, does not perform well for SA on dialectal Arabic data. So, existing works have focused on building resources and annotating corpora for a few dialects where most of them were collected from social media (Medhaffar et al., 2017; Al-Twairesh et al., 2017; Baly et al., 2018; Moudjari et al., 2020; Oueslati et al., 2020). Nevertheless, dealing with Arabic dialects as standalone languages is challenging since building manually such resources is costly and time-consuming.
It is well known that the generalization performance of Machine Learning (ML) models drops in the case of domain shift (out of distribution data). Hence, there is an imperative need to leverage existing labeled data from other related domains, in order to address this challenge. The aim is to accurately transfer the learned knowledge from a source domain labeled data to a new target domain data. On the one hand, adaptive pretraining of contextualized word embedding models has shown an effective transfer learning performance under domain shift (Han and Eisenstein, 2019; Rietzler et al., 2020). It consists of fine-tuning the pre-trained language models on large unlabeled corpus from the target domain using the MLM objective. On the other hand, self-training and domain-adversarial learning have been applied
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2824-2837 June 6-11, 2021. ©2021 Association for Computational Linguistics
successfully to many NLP applications (Li et al., 2020; Ramponi and Plank, 2020; Ye et al., 2020; Ganin et al., 2016). An effective method that combines domain-adversarial training and self-training is the Adversarial-Learned Loss for Domain Adaptation (ALDA) (Chen et al., 2020). The domain-adversarial training aligns both domains' output distributions, while self-training captures the discriminative features of the target domain data.
In this paper, we introduce a new unsupervised domain adaptation method for Arabic cross-domain and cross-dialect sentiment analysis based on AraBERT language model (Antoun et al., 2020) and the Adversarial-Learned Loss for Domain Adaptation (ALDA) (Chen et al., 2020). Due to limited amount of unlabeled data for most target domains-dialects, we do not rely on the adaptive pre-training of AraBERT model. Our method leverages the potentials of: i) contextualized word embeddings to learn high-level text representation, ii) adversarial domain training to match the output distributions of domains and dialects, and iii) self-training to capture the discriminative features of the target domain data.
- The proposition of a new unsupervised domain adaptation method for Arabic SA.
- The study of three possible challenging scenarios of domain adaptation for Arabic SA.
- The achievement of very promising results on several Arabic cross-domain and cross-dialect sentiment classification datasets.
To the best of our knowledge, this is the first study that investigates domain adaptation for cross-domain, cross-dialect and cross-domain & cross-dialect sentiment analysis, adopting the coarse-grained and the fine-grained taxonomies of Arabic dialects. The proposed method outperforms several state-of-the-art methods on most test datasets.
The rest of this paper is organized as follows. Section 2 presents related work. In Section 3, we introduce our method. Section 4 illustrates the conducted experiments, and discusses the obtained results. Finally, in Section 6, we conclude the paper and outline a few directions for future work.
Arabic sentiment classification. Recently, tangible progress has been made for Arabic senti
ment analysis (Badaro et al., 2019; Al-Ayyoub et al., 2019). This has been achieved by publishing datasets (Elnagar et al., 2018; Ashraf and Omar, 2016; Aly and Atiya, 2013; ElSahar and El-Beltagy, 2015; Nabil et al., 2015), sentiment lexicons (Badaro et al., 2014; El-Beltagy, 2016; Gilbert Badaro and Habash, 2018), and proposing models as well as architectures that reach decent accuracy scores (Al Sallab et al., 2015; Antoun et al., 2020; Abdul-Mageed et al., 2020). As an example, the pre-trained language model AraBERT (Antoun et al., 2020) has achieved state-of-the-art performance on Arabic sentiment classification tasks. Nevertheless, most of these achievements are still limited to the MSA, and to some Arabic dialects and domains (Badaro et al., 2019; Al-Ayyoub et al., 2019).
Unsupervised domain adaptation. In the past few years, there has been considerable interest in unsupervised domain adaptation for cross-domain NLP tasks, including cross-domain sentiment analysis (Ramponi and Plank, 2020). Previous work has focused on minimizing the discrepancy between domains by aligning the output distributions of the source and the target domains. Maximum Mean Discrepancy (MMD) (Gretton et al., 2012), KL-divergence (Zhuang et al., 2015), Correlation Alignment (CORAL) (Sun and Saenko, 2016), and domain-adversarial learning (Ganin et al., 2016) are among the most widely used methods to learn domain-invariant features. In the same vein, other researchers have adopted self-training approach in order to learn discriminative features of the target domain (Ramponi and Plank, 2020; Ye et al., 2020). The latter approach enables the model to be also trained on some samples of the target domain. The main idea is to select a subset of pseudo-labels, predicted on the target domain inputs, for which the model's confidence is higher than a fixed threshold, and to incorporate them into the model loss. However, pseudo-labels are generally noisy and may hurt the performance of the model. Chen et al. (2020) have tackled this issue by introducing the adversarial-learned loss for domain adaptation where the discriminator corrects the noise in the pseudo-labels by generating noise vectors that are specific for each domain.
Domain adaptation for cross-domain sentiment analysis. In order to learn cross-domain text representation, several domain adaptation methods have relied on pivot features extraction. In
spired from structural correspondence learning, Yu and Jiang (2016) have proposed a method to learn continuous sentence embedding employing CNN model across various domains. Li et al. (2018) have introduced a domain adaption method which can be extended to documents. The latter method uses a hierarchical attention transfer network for extracting the pivots and non-pivots features between source and target domains. Ziser and Reichart (2018) have proposed language modeling objective to learn a model scratch rather than adapting a pre-trained embedding model.
Recently, several methods have been introduced for domain adaptation based on adaptive pretraining of contextualized word embeddings (Han and Eisenstein, 2019; Li et al., 2020; Vu et al., 2020). The latter approach relies on the availability of a large amount of unlabeled data in the target domain to finetune/adapt the existing pre-trained language model using the MLM objective. Rietzler et al. (2020) have proposed an unsupervised domain adaptation method for aspect-target sentiment classification based on BERT adaptive pretraining. Vu et al. (2020) have presented an adaptive pre-training method that adversarially masks out tokens that are hard to be reconstructed by the MLM. In another work, (Du et al., 2020) have proposed to combine BERT domain-aware training and adversarial-domain learning (Ganin et al., 2016) for cross-domain sentiment analysis. The domain-aware training combines the adaptive pretraining using the MLM objective and a Domain Distinguish Task (DDT). For cross-domain and cross-lingual domain adaptation, Li et al. (2020) have introduced an unsupervised feature decomposition method based on the mutual information to extract domain-invariant and domain-specific features using the XLM language model (Lample and Conneau, 2019).
For the Arabic language, Khaddaj et al. (2019) have introduced a domain adaptation method for cross-domain and cross-dialect sentiment analysis, combining domain adversarial training (Ganin et al., 2016) with denoising autoencoder for representation learning. The input sentences of both domains are represented using the bag-of-words representation by selecting the top 5,000 most frequent unigrams and bigrams. The obtained results on the Levantine multi-topic ArSentD-LEV dataset (Baly et al., 2018) show that combining the reconstruction loss with the adversarial training has slightly
improved the performance in some cases. Nevertheless, the overall obtained results show that the zero-shot transfer from the SVM model achieves competitive results for some datasets. In another work, Qwaider et al. (2019) have shown that models that are trained on MSA for the task of sentiment classification generalize poorly to dialectal Arabic data. For improving the results, they have performed domain adaptation using feature engineering and sentiment lexicons.
In this section, we present our model architecture. The noise-correcting discriminator, the classifier and the generator losses, employed in our model, are those of ALDA model (Chen et al., 2020).
In unsupervised domain adaptation settings, for sentiment analysis, we are given a labeled source domain  $\mathcal{D}_S = \{(x_s^i,y_s^i)\}_{i = 1}^{n_s}$  having  $K$  classes and an unlabeled target domain  $\mathcal{D}_T = \{x_t^i\}_{i = 1}^{n_t}$ . The aim is then to transfer the learned knowledge from  $\mathcal{D}_S$  to  $\mathcal{D}_T$ . In other words, the objective is to train a robust classifier using the labeled source domain data that generalizes well on the target domain test data.
Figure 1 presents the general framework of our method. We aim to leverage the strength of both the domain adversarial training and the self-training in a unified framework. The adversarial training aligns both domains' output distributions, whereas the self-training considers the discriminative features of the target domain. Besides, AraBERT is used as a generator to extract high-level representation from both source and target domains sentences.
The generator  $G$ , the AraBERT encoder, is trained to extract features from the input sentences for both domains:  $h_{[CLS]} = G(x)$  corresponds to the hidden state of the  $[CLS]$  token. The weights of the generator are shared between both domain inputs.
The classifier  $C$  operates on the hidden states  $h_{[CLS]}$  to classify the input instances  $x$  and outputs a probability vector  $p(y = k|x) = \text{Softmax}(W_c h_{[CLS]} + b_c)$  for both domains ( $p_s$  and  $p_t$ ), where  $b_c$  and  $W_c$  are the bias vector and the weight matrix on the classification layer, respectively.
The generator  $G$  tries to confuse the discrimina
Figure 1: The general framework of our method
tor  $D$  by maximizing its loss. Thus, the generator aligns both domains' output distributions, whereas the discriminator must distinguish both domain features by generating different noise vectors for each domain. These noise vectors are employed to correct the pseudo-labels predicted by the classifier  $C$ . The Gradient Reversal Layer (GRL) reverses the gradient of the discriminator's loss during the back-propagation step.
3.2 Noise correcting discriminator
The input of the discriminator  $D$  corresponds to the hidden state  $h_{[CLS]}$  of the generator  $G$ .  $D$  is trained to produce a noise vector  $\xi^{(x)} = \sigma (D(h_{[CLS]})$  by applying  $\sigma$ , the Sigmoid activation, on its output layer. Note that, the output layer size is equal to  $K$ , the number of classes. Each component of the noise vector estimates the probability that the predicted label is the correct label  $\xi_k^{(x)} = p(y = k|\hat{y} = k,x)$ . Hence, instead of being trained to classify the source domain sentences and the ones of the target
domain,  $G$  is trained to generate different noise vectors for each domain. The noise vector is used to estimate the confusion matrix  $\eta = (\eta_{kl})$ , which is applied to correct the target domain's pseudolabels, predicted by the classifier  $C$ . The intuition behind the ALDA model is that, if we appropriately estimate the confusion matrix, the noise in the pseudo-labels predicted by the classifier, can be efficiently corrected (Chen et al., 2020).
Assuming that the noise in pseudo-labels is class-wise uniform with vector  $\xi_{k}^{(x_{t})}$ , the confusion matrix is then given by:
$$
\eta_ {k l} ^ {x _ {t}} = \left\{ \begin{array}{l l} \xi_ {k} ^ {(x _ {t})} & \text {i f} k = l \\ \frac {1 - \xi_ {l} ^ {(x _ {t})}}{K - 1} & \text {e l s e} \end{array} \right.
$$
The corrected label vector in the target domain is given by  $c^{(x_t)} = \sum_l\eta_{kl}^{x_t}p(\hat{y}_t = l|x_t)$  ( $l$  is the predicted pseudo label). For the source domain, the corrected label vector  $c^{(x_s)}$  is computed using the same procedure.
For the source domain, the discriminator minimizes the binary cross-entropy loss  $\mathcal{L}_{bce}$  between the corrected label vectors and the ground truth labels  $y_{s}$ :
$$
\mathcal {L} _ {a d v} (x _ {s}, y _ {s}) = \mathcal {L} _ {b c e} (c ^ {(x _ {s})}, y _ {s}) \qquad (1)
$$
For the target domain, the discriminator minimizes the binary cross-entropy loss  $\mathcal{L}_{bce}$  between the corrected label vector and the opposite distribution of the predicted pseudo-label  $u^{(\hat{y}_t)}$ :
$$
\mathcal {L} _ {a d v} \left(x _ {t}\right) = \mathcal {L} _ {b c e} \left(c ^ {\left(x _ {t}\right)}, u ^ {\left(\hat {y} _ {t}\right)}\right) \tag {2}
$$
where  $u^{(\hat{y}_t)}$  is computed as follows:
$$
u _ {k} ^ {(\hat {y} _ {t})} = \left\{ \begin{array}{l l} 0 & \text {i f} \hat {y} _ {t} = k \\ \frac {1}{K - 1} & \text {e l s e} \end{array} \right.
$$
To discriminate between both domains, the discriminator minimizes the following total adversarial loss:
$$
\mathcal {L} _ {a d v} \left(x _ {s}, y _ {s}, x _ {t}\right) = \mathcal {L} _ {a d v} \left(x _ {s}, y _ {s}\right) + \mathcal {L} _ {a d v} \left(x _ {t}\right) (3)
$$
In order to make the training more stable, ALDA incorporates the classification loss of the source domain as a regularization term into the discriminator. Thus, the discriminator must also correctly classify the source domain data. The regularization term is given by:
$$
\mathcal {L} _ {r e g} \left(x _ {s}, y _ {s}\right) = \mathcal {L} _ {c e} \left(p _ {d} ^ {\left(x _ {s}\right)}, y _ {s}\right) \tag {4}
$$
where  $p_d^{(x_s)} = \text{Softmax}(D(h_{[CLS]}^{(x_s)}))$  and  $\mathcal{L}_{ce}$  is the cross-entropy loss. Finally, the discriminator minimizes the following loss function:
$$
\mathcal {L} _ {D} = \mathcal {L} _ {a d v} (x _ {s}, y _ {s}, x _ {t}) + \mathcal {L} _ {r e g} (x _ {s}, y _ {s}) \quad (5)
$$
3.3 Classifier and generator losses
Following the principles of pseudo-labeling methods for domain adaptation, the ground truth label  $y_{t}$  for the target domain can be substituted by:
$$
\hat {y} _ {t} = \operatorname * {a r g m a x} _ {k} p _ {t} ^ {k} \text {i f} p _ {t} ^ {k} > \delta
$$
where  $\delta$  is a threshold. By using the learned confusion matrix  $\eta^{(x_t)}$  to correct the pseudo-label generated by the classifier  $C$ , ALDA approximates the loss in the target domain by:
$$
\mathcal {L} _ {T} \left(x _ {t}, \mathcal {L} _ {u n h}\right) = \sum_ {k} c _ {k} ^ {\left(x _ {t}\right)} \mathcal {L} _ {u n h} \left(p _ {t}, k\right) \tag {6}
$$
where  $\mathcal{L}_{unh}(p,k) = 1 - p_k$  is the unhinged loss. Then, the classifier  $C$  minimizes the following loss:
$$
\mathcal {L} _ {C} = \mathcal {L} _ {c e} \left(p _ {s}, y _ {s}\right) + \lambda \mathcal {L} _ {T} \left(x _ {t}, \mathcal {L} _ {u n h}\right) \tag {7}
$$
where  $\mathcal{L}_{ce}(p_s, y_s)$  is the cross-entropy loss of the source domain. Finally, the generator  $G$  minimizes the following loss function:
$$
\begin{array}{l} \mathcal {L} _ {G} = \mathcal {L} _ {c e} \left(p _ {s}, y _ {s}\right) + \lambda \mathcal {L} _ {T} \left(x _ {t}, \mathcal {L} _ {u n h}\right) \tag {8} \\ - \lambda \mathcal {L} _ {a d v} \left(x _ {s}, y _ {s}, x _ {t}\right) \\ \end{array}
$$
where  $\lambda \in [0,1]$  is a hyperparameter.
In this section, we present the experiments carried out to investigate the performance of our proposed method for Arabic cross-domain and cross-dialect sentiment analysis. We describe the used datasets and present the compared methods as well as the obtained results. We provide the experiments settings and implementation details of our method in Section A. The source code for reproducing the experimentations can be found in our github repository<sup>1</sup>.
We conduct three main sets of experiments to cover three possible scenarios.
Scenario 1: Domain adaptation for dialects of the same region. The set of experiments of this scenario aims to study our method's performance for cross-dialect and cross-domain sentiment analysis for Arabic dialects of the same region. To do so, we employ the existing multi-domain multi-dialect ArSentD-LEV dataset of the Levant region (Baly et al., 2018). ArSentD-LEV contains 1,000 tweets from each country of the Levant region (4,000 in total): Syria, Palestine, Jordan, and Lebanon. It is labeled into five classes and covers tweets from five topics: Personal  $(36\%)$ , Politics  $(23\%)$ , Religion  $(11\%)$ , Sport  $(6\%)$ , and Other  $(24\%)$ .
Scenario 2: Domain adaptation across regional dialects. In the set of experiments of this scenario, we investigate the performance of our method using the coarse-grained regional taxonomy of Arabic dialects. For this purpose,
<sup>1</sup>https://github.com/4mekki4/arabic-nlp-da
1. Firstly, we select three datasets, mixing Arabic dialects and MSA: BRAD (Elnagar and Einea), HARD (Elnagar et al., 2018), and TEAD (Abdellaoui and Zrigui, 2018) that are compiled from book reviews, hotel reviews, and Twitter, respectively. These datasets have sufficient samples to build a multi-dialect multi-domain dataset.
2. Secondly, we train an AraBERT-based dialect identification model, selecting data from some of the publicly available datasets, including MADAR (Bouamor et al., 2019), DART (Alsarsour et al., 2018), AOC (Zaidan and Callison-Burch, 2011), PADIC (Karima et al., 2018), and the multi dialect Arabic texts corpora proposed in (Khalid and Mark, 2013). The resulting Arabic dialect identification corpus consists of 353, 171 training sentences and a balanced test set of 50,000 sentences and covers MSA as well as dialectal sentences from Maghrebi, Levantine, Egyptian, and Gulf. It is worth mentioning that our trained dialect identification model achieves  $89\%$  accuracy.
3. Finally, we apply our dialect identification model on the three evaluated datasets to build our multi-dialect multi-domain corpus. Moreover, we select the Levantine and Gulf dialects and MSA, which yielded sufficient data across domains. For review datasets, the rating levels 1 and 2 are assigned to negative polarity, while ratings 4 and 5 are considered positives. Furthermore, we sample 1000 positive and 1000 negative instances for each dialect to build our final multi-dialect and multi-domain dataset.
Scenario 3: Domain adaptation from MSA to Arabic dialects using social media data. The set of experiments of this scenario tackles the transfer of learning from MSA to Arabic dialects, belonging to different regions, using corpora built from social media (see Table 7 of Section 1.5 for the datasets details). Thus,
1. For MSA, we use the ArSAS dataset (Elmadany et al., 2018).
2. For the Maghrebi region, we employ MSAC (Morocco) (Oussous et al., 2020) and TSAC (Tunisia) (Medhaffar et al., 2017) datasets. Note that, we have removed sentences that are written in Arabezi for TSAC.
3. For the Egyptian region, we use the ASTD dataset (Nabil et al., 2015).
4. For the Levant region, we utilize AJGT (Jordan) (Alomari et al., 2017) and TweetSYR (Syria) (Salameh et al., 2015) datasets.
5. For the Gulf region, we employ the Saoudi dialect AraSenti-Tweet dataset (Al-Twairesh et al., 2017).
Since some of these datasets are labeled using positive and negative classes only (TSAC and MSAC), we evaluate our method using positive and negative sentences for all the used datasets.
We use the train-test splits of the evaluated datasets whenever this information is available. Otherwise, we split the datasets into  $80\%$  train and  $20\%$  test. For the ArSentD-LEV and following the work of (Khaddaj et al., 2019), we evaluate our method on the full target domain/dialect dataset. For all our experiments, we utilize the accuracy evaluation measure and highlight the best accuracy performance using bold font.
In order to assess the performance of our method, we compare it with the state-of-the-art domain adaptation method, introduced by (Khaddaj et al., 2019), for Arabic sentiment analysis on the ArSentD-LEV dataset. Moreover, we evaluate BERT for zero-shot transfer from the source domain, denoted ZS-BERT. For a fair comparison, we investigate the performance of three state-of-the-art domain adaptation methods including MMD (Gretton et al., 2012), CORAL (Sun and Saenko, 2016), and DANN (Ganin et al., 2016). We implement the latter methods on top of AraBERT. We have also evaluated two state-of-the-art cross-domain sentiment analysis methods, namely PBLM (Ziser and Reichart, 2018) and HTAN (Li et al., 2018). It is worth to mention that for PBLM and HATN, we have used an extra 4000 unlabeled sentences from each domain/dialect. For HTAN, we have used Mazjak word embedding model (Abu Farha and Magdy, 2019)