SlowGuess/ACL_2020_OCR huggingface dataset:
Amazon Review (French, German, Japanese). This is a multilingual sentiment classification dataset (Duh et al., 2011) in four languages, in
cluding English (en), French (fr), German (de), and Japanese (ja), covering three products (book, DVD, and music). For each product in each language, there are 2000 documents in each of the training and test sets. Each document contains a title, a category label, a review, and a 5-point scale star rating. Following Xu and Yang (2017); Chen and Qian (2019), we convert multi-class ratings to binary ratings by thresholding at 3-point. For each product, since the test set in English is not used, we combine the English training and test sets and randomly sample  $20\%$  (800) documents as the validation set to tune hyper-parameters, and use the rest 3200 samples for training. For each target language, we use the original 2000 test samples for comparison with previous methods. Unlike Chen et al. (2018); Chen and Qian (2019) that used labeled data in the target language for model selection, we only use the labels of reviews in the target language for testing. There are 105k, 58k, 317k, 300k unlabeled reviews for English, French, German and Japanese, respectively, which can be used as monolingual data to train the encoder-decoder of our model.
Yelp and Hotel Review (Chinese). This dataset is from two sources: (i) 700k Yelp reviews in English with five classes from Zhang et al. (2015), and (ii) 170k hotel reviews in Chinese segmented and annotated with five classes from Lin et al. (2015). Following the same setup in Chen et al. (2018), we split all Yelp reviews into a training set with 650k reviews and validation set with 50k reviews. The 650k review contents are also served as the monolingual training data for English. For Chinese hotel review data, we sample 150k reviews as the monolingual training set. The rest 20k reviews are treated as the test set.
Social Media Posts (Arabic). The BBN Arabic Sentiment dataset is from Mohammad et al. (2016). There are 1200 documents from social media posts annotated with three labels (negative, neutral, positive) in the data. The original dataset was split into half as training and the other half as testing. Since we do not need validation data in the target language to tune the model, we randomly sample 1000 documents as test data. For English resource, we still use Yelp reviews and follow the same split as the Chinese case, but convert 5 level reviews into 3 levels<sup>2</sup>. Also, we randomly sample
${}^{2}1,2 \rightarrow$  negative,  $3 \rightarrow$  neutral,  $4,5 \rightarrow$  positive
161k sentences from the United Nations Corpus Arab subset (Ziemski et al., 2016) as unlabeled monolingual data for our model training.
4.2 Experiment Setting
For French, German and Japanese, we perform binary classification. For Chinese and Arabic, we perform multi-class classification.
Data Preprocessing. Following Lample et al. (2018c), we extract and tokenize monolingual data of each language using Moses (Koehn et al., 2007). Then we use the neural machine translation for rare words with subword units, named fastBPE (Sennrich et al., 2016) in three steps. In detail, BPE code is collected from the pretrained XLM-100 models (Conneau and Lample, 2019), then applied to all tokenized data and used to extract the training vocabulary. To constrain our model size, we only keep the top  $60\mathrm{k}$  most frequent subword units in our training set. Finally, we binarize monolingual data and labeled data for model training, validation and testing.
Pretraining Details. As mentioned earlier, our model depends on an initial translation machine to compute reconstruction loss and classifier loss. We leverage pretrained language models (Connau and Lample, 2019) to initialize a transformer-based UMT (Lample et al., 2018a) and train it on Wikipedia text<sup>3</sup>. In particular, we sample 10 million sentences from each language pairs and use the XLM library<sup>4</sup> to train a UMT (Lample et al., 2018a) for 200K steps. The resulting encoder-decoder are used to initialize our model.
Regarding word embedding initialization, we use the embeddings obtained from the 1st layer of pretrained language models (Conneau and Lample, 2019), which has demonstrated better crosslingual performance in a number of evaluation metrics over MUSE (Lample et al., 2018b).
Training Details. In our experiment, both encoder and decoder are 6 layer transformers with 8-head self-attention. We set both subword embedding and hidden state dimension to 1024 and use greedy decoding to generate a sequence of tokens. The encoder-decoder and classifier are trained using Adam optimizer (Kingma and Ba, 2015) with a learning rate of  $10^{-5}$  and a mini-batch size of 32. We set the hidden dimension to 128 for both clas
sifier and discriminator. For parameters of denoising auto-encoder, we set  $p_d = 0.1$ ,  $p_b = 0.2$  and  $k = 3$  following Lample et al. (2018a). Finally, we perform a grid search for hyper-parameters on  $\{0.5,1,2,4,8\}$  and set  $\lambda_{wd}$ ,  $\lambda_{cd}$  to 1 and  $\lambda_{adv}$  to 4. To prevent gradient explosion, we clip the gradient  $L_2$  norm by 5.0. Our approach is implemented in PaddlePaddle<sup>5</sup> and all experiments are conducted on an NVIDIA Tesla M40 (24GB) GPU.
Competing Methods. We have compared our method with several recently published results. Due to the space limit, we briefly introduce several representative baselines: LR+MT translated the bag of words from target language to source language via machine translation and then built a logistic regression model. BWE baselines rely on Bilingual Word Embeddings (BWEs), wherein 1-to-1 indicates that we are only transferring from English, while 3-to-1 means the training data from all other three languages. CLDFA (Xu and Yang, 2017) was built on model distillation on parallel corpora with adversarial feature adaptation technique. PBLM (Ziser and Reichart, 2018) used bilingual word embeddings and pivot-based language modeling for cross-domain & cross-lingual classification. MBERT (Devlin et al., 2019) and XLM-FT (Conneau and Lample, 2019) directly fine-tuned a single layer classifier based on pretrained LM multilingual BERT and XLM.
In Table 1 and Table 2, we compare our method with others based on their published results or our reproduced results from their code. Our results are averaged based on 5 rounds of experiment with the standard deviation around  $1\% - 1.5\%$ . Following previous baselines, we do not report them here.
Our first observation from Table 1 is that our model and the fine-tuned multilingual LM MBERT (Devlin et al., 2019) and XLM-FT (Conneau and Lample, 2019) outperform all previous methods including the methods with cross-lingual resources for 8/9 tasks by a large margin, which indicates the huge benefit from pretrained LMs in the zero-shot setting. Compared with MBERT and XLM-FT, our model obtains better performance when the target language is more similar to the source language, for example, German and French, and one task in Japanese.
<sup>3</sup>http://dumps.wikimedia.org/
$^{4}$ www.github.com/facebookresearch/XLM
5http://www.paddlepaddle.org/
Approach	German (2)	French (2)	Japanese (2)
books	DVD	music	avg	books	DVD	music	avg	books	DVD	music	avg
With cross-lingual resources
LR+MT	79.68	77.92	77.22	78.27	80.76	78.83	75.78	78.46	70.22	71.30	72.02	71.18
CR-RL1	79.89	77.14	77.27	78.10	78.25	74.83	78.71	77.26	71.11	73.12	74.38	72.87
Bi-PV2	79.51	78.60	82.45	80.19	84.25	79.60	80.09	81.31	71.75	75.40	75.45	74.20
CLDFA3	83.95	83.14	79.02	82.04	83.37	82.56	83.31	83.08	77.36	80.52	76.46	78.11
With implicit cross-lingual resources
UMM4	81.65	81.27	81.32	81.41	80.27	80.27	79.41	79.98	71.23	72.55	75.38	73.05
PBLM5	78.65	79.90	80.10	79.50	77.90	75.65	75.95	76.50	-	-	-	-
Without cross-lingual resources
BWE (1-to-1)	76.00	76.30	73.50	75.27	77.80	78.60	78.10	78.17	55.93	57.55	54.35	55.94
BWE (3-to-1)	78.35	77.45	76.70	77.50	77.95	79.25	79.95	79.05	54.78	54.20	51.30	53.43
MAN-MoE6	82.40	78.80	77.15	79.45	81.10	84.25	80.90	82.08	62.78	69.10	72.60	68.16
MBERT7	84.35	82.85	83.85	83.68	84.55	85.85	83.65	84.68	73.35	74.80	76.10	74.75
XLM-FT8	86.85	84.20	85.90	85.65	88.1	86.95	86.20	87.08	80.95	79.20	78.02	79.39
MVEC (Ours)	88.41	87.32	89.97	88.61	89.08	88.28	88.50	88.62	79.15	77.15	79.70	78.67
$^{1}$  Xiao and Guo (2013)
2 Pham et al. (2015)
$^{3}$  Xu and Yang (2017)
$^{4}$  Xu and Wan (2017)
5 Ziser and Reichart (2018)
6 Chen and Qian (2019)
7 Devlin et al. (2019)
8 Conneau and Lample (2019)
Table 1: Prediction accuracy of binary classification in the test set for three language pairs. The highest performance is in bold, while the highest performance within the method group is underlined.
Approach	Chinese (5)	Arabic (3)
LR+MT	34.01	51.67
DAN	29.11	48.00
mSDA	31.44	48.33
ADAN	42.49	52.54
MBERT	38.85	50.40
XLM-FT	42.22	49.50
MVEC (Ours)	43.36	49.70
Table 2: Prediction accuracy of 5-class and 3-class classification tasks on the test set.
In Table 2, we show the comparison between our method and a few other published results, including ADAN (Chen et al., 2018) and mSDA (Chen et al., 2012) for Chinese and Arabic languages in multi-class setting. Similarly, our model obtains slightly better accuracy in Chinese. Overall, built on top of the pretrained LMs and UMT, our full model achieves the state-of-the-art performance on 8/11 sentiment classification tasks, especially when the target language is more similar to the source language.
Moreover, we illustrate the effectiveness of encoder-decoder based regularization in reducing the language shift in the shared latent space. Intuitively, if the fine-tuned latent space is less sensitive to the language shift, the performance on validation sets and test sets should be highly correlated during training. In Figure 2, we report the average accuracy of both validation and test set w.r.t. training epochs over five runs on Amazon book review data in French.
Figure 2: Validation and test accuracy w.r.t. training epochs for Amazon book review in French. Left: our method (MVEC). Right: XLM-FT.
From Figure 2, we observe that even though our model's best validation accuracy is lower than XLM-FT (Conneau and Lample, 2019) in English, it has more correlated accuracy curves than XLM-FT across English and French. For example, the validation accuracy of XLM-FT starts decreasing after epoch 10, while the test accuracy is still increasing. Such an observation shows that the latent representation learned solely from self-supervised objectives (e.g., masked language modeling) may not well capture the semantic similarity among languages. Hence the resulting classifier may work well in the source language but may not generalize to the target language. In contrast, our model sacrifices some accuracy in the source language but can select better models for the target language in a cross-lingual setting.
To understand the effect of different components in our model on the overall performance, we con
5766
German	French	Japanese	Chinese	Arabic
Full model:	88.61	88.62	78.67	43.36	49.70
w/o cross-domain loss:	83.22	82.40	72.05	35.74	42.80
w/o within-domain loss:	82.90	82.15	71.27	37.21	41.60
w/o adversarial training:	84.85	84.58	73.75	39.36	46.37
w/o two-views consensus:	86.21	86.18	75.25	40.95	46.77
Table 3: Ablation study on five language pairs.
duct an ablation study, as reported in Table 3. Clearly, the encoder-decoder trained either by the within-domain objective or cross-domain objective is the most critical. For Amazon data in three languages (German, French, Japanese), the model without cross-domain loss obtains prediction accuracy of  $83.22\%$ ,  $82.40\%$ , and  $72.05\%$ , which gets decreased by  $5\% - 7\%$  compared with the full model. The performance is also significantly degraded when the adversarial training component is removed because the distribution of latent document representations is not similar between two languages. The two-views consensus component also has a significant effect on the performance of our model, with a performance drop up to 5 points for en-jp. Such a result verifies our claim that cross-lingual model benefits from training on multiple views of the input.
To further explore the effectiveness of our approach, we visualize the encoder's output and the last layer before softmax for 10 randomly sampled Amazon reviews in English and their translations in French using Google Translation, as shown in Appendix A.2.
As seen in the lower-left panel of Figure 3, most red circles and black squares with the same indices are very close for our method but are distant for XLM-FT in the top-left. Such an observation implies that our encoder combined UMT and a language discriminator adequately maps the input into a shared language-invariant latent space while preserving semantic similarity. For the last layer before softmax, even though XLM-FT also generates reasonable representations to separate positive and negative reviews, the data points are scattered randomly. On the contrary, our model's output in the lower right panel of Figure 3 shows two more obvious clusters with corresponding labels that can be easily separated. One cluster in the left contains all of the positive documents, while the negative examples only appear on the right side.
Figure 3: t-SNE visualizations of various layers of XLM-FT and MVEC for en-fr. Red circles and black squares indicate documents from English and their corresponding translations in the target language, respectively. Numbers indicate the document index and have a one-to-one mapping. +/- indicates labels and we only annotate English documents for simplicity. Top left: encoder output of XLM-FT. Top right: the last layer before softmax of XLM-FT. Lower left: encoder output of our method. Lower right: the last layer before softmax of our method.
In this paper, we propose a cross-lingual multiview encoder-classifier (MVEC) that requires neither labeled data in the target language nor cross-lingual resources with the source language. Built upon pretrained language models, our method utilizes the encoder-decoder component with a language discriminator from an unsupervised machine translation system to learn a language-invariant feature space. Our approach departs from previous models that could only make use of the shared language-invariant features or depend on parallel resources. By constructing the fine-tuned latent feature space and two views of input from the encoder-decoder of UMT, our model significantly outperforms previous methods for 8/11 zero-shot sentiment classification tasks.
5767
Daniel Andrade, Kunihiko Sadamasa, Akihiro Tamura, and Masaaki Tsuchida. 2015. Cross-lingual text classification using topic-dependent word probabilities. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), pages 1466-1471, Denver, CO.
Núria Bel, Cornelis H. A. Koster, and Marta Villegas. 2003. Cross-lingual text categorization. In Proceedings of the 7th European Conference on Research and Advanced Technology for Digital Libraries (ECDL), pages 126-139, Trondheim, Norway.
Minmin Chen, Zhixiang Eddie Xu, Kilian Q. Weinberger, and Fei Sha. 2012. Marginalized denoising autoencoders for domain adaptation. In Proceedings of the 29th International Conference on Machine Learning (ICML), Edinburgh, UK.
Xilun Chen, Yu Sun, Ben Athiwaratkun, Claire Cardie, and Kilian Q. Weinberger. 2018. Adversarial deep averaging networks for cross-lingual sentiment classification. Trans. Assoc. Comput. Linguistics, 6:557-570.
Zhuang Chen and Tieyun Qian. 2019. Transfer capsule network for aspect level sentiment classification. In Proceedings of the 57th Conference of the Association for Computational Linguistics (ACL), pages 547-556, Florence, Italy.
Alexis Conneau and Guillaume Lample. 2019. Crosslingual language model pretraining. In Advances in Neural Information Processing Systems (NeurIPS), pages 7057-7067, Vancouver, Canada.
Alexis Conneau, Holger Schwenk, Loic Barrault, and Yann LeCun. 2017. Very deep convolutional networks for text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (EACL), pages 1107-1116, Valencia, Spain.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 4171-4186, Minneapolis, MN.
Xin Dong and Gerard de Melo. 2019. A robust self-learning framework for cross-lingual text classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6305–6309, Hong Kong, China.
Kevin Duh, Akinori Fujino, and Masaaki Nagata. 2011. Is machine translation ripe for cross-lingual sentiment classification? In Proceedings of the 49th Annual Meeting of the Association for Computational
Linguistics: Human Language Technologies (ACL-HLT), pages 429-433, Portland, OR.
Akiko Eriguchi, Melvin Johnson, Orhan First, Hideto Kazawa, and Wolfgang Macherey. 2018. Zero-shot cross-lingual classification using multilingual neural machine translation. Technical report, arXiv:1809.04686.
Yanwei Fu, Timothy M. Hospedales, Tao Xiang, and Shaogang Gong. 2015. Transductive multi-view zero-shot learning. IEEE Trans. Pattern Anal. Mach. Intell., 37(11):2332-2345.
Ruidan He, Wee Sun Lee, Hwee Tou Ng, and Daniel Dahlmeier. 2019. An interactive multi-task learning network for end-to-end aspect-based sentiment analysis. In Proceedings of the 57th Conference of the Association for Computational Linguistics (ACL), Florence, Italy.
Phillip Keung, Yichao Lu, and Vikas Bhardwaj. 2019. Adversarial learning with contextual embeddings for zero-resource cross-lingual classification and NER. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1355–1360, Hong Kong, China.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR), San Diego, CA.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, and et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics (ACL), Prague, Czech Republic.
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018a. Unsupervised machine translation using monolingual corpora only. In Proceedings of the 6th International Conference on Learning Representations (ICLR), Vancouver, Canada.
Guillaume Lample, Alexis Conneau, Marc'Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018b. Word translation without parallel data. In Proceedings of the 6th International Conference on Learning Representations (ICLR), Vancouver, Canada.
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018c. Phrase-based & neural unsupervised machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5039-5049, Brussels, Belgium.
Yingming Li, Ming Yang, and Zhongfei Zhang. 2019. A survey of multi-view representation learning. IEEE Trans. Knowl. Data Eng., 31(10):1863-1883.
Yiou Lin, Hang Lei, Jia Wu, and Xiaoyu Li. 2015. An empirical study on sentiment classification of Chinese review using word embedding. In Proceedings of the 29th Pacific Asia Conference on Language, Information and Computation (PACLIC), Shanghai, China.
Saif M. Mohammad, Mohammad Salameh, and Svetlana Kiritchenko. 2016. How translation alters sentiment. J. Artif. Intell. Res., 55:95-130.
Hieu Pham, Thang Luong, and Christopher D. Manning. 2015. Learning distributed representations for multilingual text sequences. In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing (VS@NAACL-HLT), pages 88-94, Denver, Co.
Peter Prettenhofer and Benno Stein. 2010. Cross-language text classification using structural correspondence learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1118-1127, Uppsala, Sweden.
Sara Sabour, Nicholas Frosst, and Geoffrey E. Hinton. 2017. Dynamic routing between capsules. In Advances in Neural Information Processing Systems (NIPS), pages 3856-3866, Long Beach, CA.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), Berlin, Germany.
Samuel L. Smith, David H. P. Turban, Steven Hamblin, and Nils Y. Hammerla. 2017. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. In Proceedings of the 5th International Conference on Learning Representations (ICLR), Toulon, France.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing (NIPS), pages 6000-6010, Long Beach, CA.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. 2008. Extracting and composing robust features with denoising autoencoders. In Proceedings of the Twenty-Fifth International Conference on Machine Learning (ICML), pages 1096-1103, Helsinki, Finland.
Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing
and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 833-844, Hong Kong, China.
Min Xiao and Yuhong Guo. 2013. Semi-supervised representation learning for cross-lingual text classification. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1465-1475, Seattle, WA.
Kui Xu and Xiaojun Wan. 2017. Towards a universal sentiment classifier in multiple languages. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 511-520, Copenhagen, Denmark.
Ruochen Xu and Yiming Yang. 2017. Cross-lingual distillation for text classification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1415–1425, Vancouver, Canada.
Qingheng Zhang, Zequn Sun, Wei Hu, Muhao Chen, Lingbing Guo, and Yuzhong Qu. 2019. Multi-view knowledge graph embedding for entity alignment. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI), pages 5429-5435, Macao, China.
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems (NIPS), pages 649-657, Montreal, Canada.
Jing Zhao, Xijiong Xie, Xin Xu, and Shiliang Sun. 2017. Multi-view learning overview: Recent progress and new challenges. Inf. Fusion, 38:43-54.
Xinjie Zhou, Xiaojun Wan, and Jianguo Xiao. 2016. Cross-lingual sentiment classification with bilingual document representation learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), Berlin, Germany.
Michal Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016. The united nations parallel corpus v1.0. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC), Porto.rož, Slovenia.
Yftah Ziser and Roi Reichart. 2018. Deep pivot-based modeling for cross-language cross-domain transfer with minimal guidance. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 238-249, Brussels, Belgium.
Will Y. Zou, Richard Socher, Daniel M. Cer, and Christopher D. Manning. 2013. Bilingual word embeddings for phrase-based machine translation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1393-1398, Seattle, WA.
5769