SlowGuess/NAACL_2021_OCR huggingface dataset:
Mahmoud Nabil, Mohamed Aly, and Amir Atiya. 2015. ASTD: Arabic sentiment tweets dataset. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2515-2519, Lisbon, Portugal. Association for Computational Linguistics.
Oumaima Oueslati, Erik Cambria, Moez Ben HajHmida, and Habib Ounelli. 2020. A review of sentiment analysis research in arabic language. Future Generation Computer Systems, 112:408 - 430.
Ahmed Oussous, Fatima-Zahra Benjelloun, Ayoub Ait Lahcen, and Samir Belfkih. 2020. Asa: A framework for arabic sentiment analysis. Journal of Information Science, 46(4):544-559.
Chatrine Qwaider, Stergios Chatzikyriakidis, and Simon Dobnik. 2019. Can Modern Standard Arabic approaches be used for Arabic dialects? sentiment analysis as a case study. In Proceedings of the 3rd Workshop on Arabic Corpus Linguistics, pages 40-50, Cardiff, United Kingdom. Association for Computational Linguistics.
Alan Ramponi and Barbara Plank. 2020. Neural unsupervised domain adaptation in nlp—a survey. arXiv preprint arXiv:2006.00632.
Alexander Rietzler, Sebastian Stabinger, Paul Opitz, and Stefan Engl. 2020. Adapt or get left behind: Domain adaptation through BERT language model finetuning for aspect-target sentiment classification. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4933-4941, Marseille, France. European Language Resources Association.
Houda Saadane and Nizar Habash. 2015. A conventional orthography for algerian arabic. In Proceedings of the Second Workshop on Arabic Natural Language Processing, ANLP@ACL 2015, Beijing, China, July 30, 2015, pages 69-79. Association for Computational Linguistics.
Mohammad Salameh, Houda Bouamor, and Nizar Habash. 2018. Fine-grained Arabic dialect identification. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1332-1344, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Mohammad Salameh, Saif Mohammad, and Svetlana Kiritchenko. 2015. Sentiment after translation: A case-study on Arabic social media posts. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 767-777, Denver, Colorado. Association for Computational Linguistics.
Baochen Sun and Kate Saenko. 2016. Deep coral: Correlation alignment for deep domain adaptation. In Computer Vision - ECCV 2016 Workshops, pages 443-450, Cham. Springer International Publishing.
Thuy-Trang Vu, Dinh Phung, and Gholamreza Haffari. 2020. Effective unsupervised domain adaptation with adversarially trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 6163-6173. Association for Computational Linguistics.
Hai Ye, Qingyu Tan, Ruidan He, Juntao Li, Hwee Tou Ng, and Lidong Bing. 2020. Feature adaptation of pre-trained language models across languages and domains with robust self-training. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 7386-7399. Association for Computational Linguistics.
Jianfei Yu and Jing Jiang. 2016. Learning sentence embeddings with auxiliary tasks for cross-domain sentiment classification. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 236-246, Austin, Texas. Association for Computational Linguistics.
Omar F. Zaidan and Chris Callison-Burch. 2011. The Arabic online commentary dataset: an annotated dataset of informal Arabic with high dialectal content. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 37-41, Portland, Oregon, USA. Association for Computational Linguistics.
Omar F. Zaidan and Chris Callison-Burch. 2014. Arabic dialect identification. Computational Linguistics, 40(1):171-202.
Fuzhen Zhuang, Xiaohu Cheng, Ping Luo, Sinno Jialin Pan, and Qing He. 2015. Supervised representation learning: Transfer learning with deep autoencoders. In Proceedings of the 24th International Conference on Artificial Intelligence, IJCAI'15, page 4119-4125. AAAI Press.
Yftah Ziser and Roi Reichart. 2018. Pivot based language modeling for improved neural domain adaptation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1241-1251, New Orleans, Louisiana. Association for Computational Linguistics.
A Experiments settings
1.1 Implementation details
We implement our method as well as the other evaluated methods using PyTorch. For all our experiments, AraBERT is used as the input text representation generator. The sentiment classification model is a fully-connected layer that takes the input text representation generated by AraBERT and outputs label probabilities through the Softmax function. For hyperparameters tuning, we have followed the method of Chen et al. (2020). For tuning the learning rate, we have conducted a random search over the set of values  $\{310^{-5}, 210^{-5}, 10^{-5}, 810^{-6}, 510^{-6}\}$ . We employ Adam optimizer with a learning rate of  $510^{-6}$  for our method and DANN, while a learning rate of  $10^{-5}$  is adopted for the CORAL and MMD and  $210^{-5}$  for ZS-BERT. We use the same discriminator architecture as in (Chen et al., 2020). The learning rate of the discriminator is set to be ten times the value of the generator. During the training, the learning is adjusted at every iteration using  $\eta_{p} = \frac{\eta_{0}}{(1 + \alpha \cdot q)^{\beta}}$  (Chen et al., 2020), where  $q$  is the training progress,  $\eta_{0} = 0.01$ ,  $\alpha = 10$  and  $\beta = 0.75$ . The hyperparameter  $\lambda$  is varied between 0 and 1 during the training using  $\frac{2}{1 + \exp(10 \cdot q)} - 1$  (Chen et al., 2020). The self-labeling threshold (varied between 0.6 and 0.9) is fixed to 0.8, except for the ArSentD-LEV dataset, where it is fixed to 0.6. For all the evaluated methods, we use a batch-size of 16 samples and 20 training epochs, except for ZS-BERT, where the number of epochs is fixed at 5. Table 5 presents the number of trainable parameters for each domain adaptation method.
Method	Genrator	Classifier	Discriminator	Total
ZS-BERT	135197189	3845	--	135201034
CORAL	135197189	3845	--	135201034
MMD	135197189	3845	--	135201034
DANN	135197189	3845	1181953	136382987
Ours	135197189	3845	1185029	136386063
1.2 Computing Infrastructure
We conduct our experiments using an Intel(R) Xeon(R) Gold 6152 CPU @ 2.10GHz working station, having a single Nvidia Tesla P100 with 16GB of RAM.
1.3 Average Runtimes
Table 6 presents the average runtime of a single run of domain adaptation methods and ZS-BERT (for the cross-domain and cross-dialect experiments).
Table 5: The number of trainable parameters for each domain adaptation method in scenario 1 (number of output classes equals to 5).
Method	Average runtime
ZS-BERT	2m30s
MMD	11m26s
CORAL	14m20s
DANN	14m46s
Ours	21m50s
1.4 Evaluation measure
For all our experiments, we have employed the accuracy evaluation measure:
$$
a c c u r a c y = \frac {\text {N u m b e r o f c o r r e c t p r e d i c t i o n s}}{\text {T o t a l n u m b e r o f p r e d i c t i o n s}}
$$
1.5 Details of the used datasets in Scenario 3
Table 7 shows the details of the train/test splits used in scenario 3 (Section 4.1). We use  $20\%$  of the dataset as a test set, except for the ArSenti-Tweet dataset where the train-test is available.
Table 6: Average runtime of each method for cross-domain and cross-dialect experiments (Table 3).
Dataset	Size	Train	Test
ArSAS	11109	80%	20%
MSAC	2000	80%	20%
TSAC	5506	80%	20%
ASTD	1589	80%	20%
AJGT	1800	80%	20%
TweetSYR	1798	80%	20%
ArSenti-Tweet	11112	9750	1362
Table 7: The description of the datasets used in Scenario 3. When the train-test split is not available, we use  $20\%$  of the data for the test set.
1.6 Datasets links
- Scenario 1: ArSentD-LEV
- Scenario 2: BRAD, HARD, TEAD
- Scenario 3: TSAC, ASTD, AJGT, ArSAS, MSAC, TweetSYR, AraSenti-Tweet $^{3}$ .
Table 8 shows a sample of sentences along with their predicted label by our proposed model, ZSBERT, and DANN.
3The AraSenti-Tweet dataset is delivered by the authors (Al-Twairesh et al., 2017)
Sentence	Ground-truth	ZS-BERT	DANN	Ours
satttahia
maa
/AstvnAQy AlnZAfp wAlbhw wAstqbAl DEf AtSAI cbkp mwbAyly/	Positive	Negative	Negative	Positive
dajm aljazm
dajm
/DEyf bwtp AlAfTAr AlrmDAny mmtAz Edm AlAltzAm bmwEd Aldxwl swC mEAamlp mwZfyn AlAstqbAl wqAHp mdyr mwZfyn AlAstqbAl/	Negative	Negative	Positive	Negative
jyd AlmwqE AlnZAfh AlHrAsh AlxSwSyh AlsEr mbAlg fyhAllintrntvlAjh Algrfh IA ywjd bhA cQ mnE dxwl AlmOkwlAt wAlmrcbwAt AsEAr AlmTEM mbAlg fyhAbckl xyAly/	Positive	Negative	Negative	Positive
jyd mwZfy AlAstqbAl bTQ fy AltEAm l wEdm xbrh kAfyh/	Positive	Negative	Negative	Negative
Table 8: Examples of predictions made by our proposed system compared with the Zero-Shot BERT and DANN