SlowGuess/ACL_2020_OCR huggingface dataset:
Hongliang Fei, Ping Li
1195 Bordeaux Dr, Sunnyvale, CA 94089, USA
10900 NE 8th St, Bellevue, WA 98004, USA
{hongliangfei,liping11}@baidu.com
Recent neural network models have achieved impressive performance on sentiment classification in English as well as other languages. Their success heavily depends on the availability of a large amount of labeled data or parallel corpus. In this paper, we investigate an extreme scenario of cross-lingual sentiment classification, in which the low-resource language does not have any labels or parallel corpus. We propose an unsupervised cross-lingual sentiment classification model named multi-view encoder-classifier (MVEC) that leverages an unsupervised machine translation (UMT) system and a language discriminator. Unlike previous language model (LM) based fine-tuning approaches that adjust parameters solely based on the classification error on training data, we employ the encoder-decoder framework of a UMT as a regularization component on the shared network parameters. In particular, the cross-lingual encoder of our model learns a shared representation, which is effective for both reconstructing input sentences of two languages and generating more representative views from the input for classification. Extensive experiments on five language pairs verify that our model significantly outperforms other models for 8/11 sentiment classification tasks.
Recent neural network models have achieved remarkable performance on sentiment classification in English and other languages (Conneau et al., 2017; Chen et al., 2018; He et al., 2019; Chen and Qian, 2019). However, their success heavily depends on the availability of a large amount of labeled data or parallel corpus. In reality, some low-resource languages or applications have limited labeled data or even without any labels or parallel corpus, which may hinder us from training a robust and accurate sentiment classifier.
To build sentiment classification models for low-resource languages, recent researchers developed cross-lingual text classification (CLTC) models (Xu and Yang, 2017; Eriguchi et al., 2018), which transfers knowledge from a resource-rich (source) language to a low-resource (target) language. The core of those models is to learn a shared language-invariant feature space that is indicative of classification for both languages. Therefore a model trained from the source language can be applied to the target language. Based on how the shared feature space is learned, there are three categories, namely word-level alignments (Andrade et al., 2015), sentence-level alignments (Eriguchi et al., 2018) and document level alignments (Zhou et al., 2016). Those models can well capture the semantic similarity between two languages. They, however, require parallel resources such as a bilingual dictionary, parallel sentences, and parallel Wikipedia articles. Such a limitation may prevent these models from being applicable in languages without any parallel resources.
Recently, there have been several attempts at developing "zero-resource" models (Ziser and Reichart, 2018; Chen et al., 2018; Chen and Qian, 2019). Most notably, Ziser and Reichart (2018) proposed a cross-lingual & cross-domain (CLCD) model that builds on pivot based learning and bilingual word embedding. Although CLCD does not directly need labeled data or parallel corpus, it requires bilingual word embeddings (BWEs) (Smith et al., 2017) that requires thousands of translated words as a supervised signal. Chen et al. (2018) developed an adversarial deep averaging network to learn latent sentence representations for classification, but it had an implicit dependency on BWEs (Zou et al., 2013) that requires pretraining on a large bilingual parallel corpus. Chen and Qian (2019) extended the
5759
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5759-5771 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
cross-lingual model in Chen et al. (2018) to multiple source languages by using the unsupervised BWEs (Lample et al., 2018b) and adding individual feature extractor for each source language, which eliminated the dependency on a parallel corpus. Nevertheless, their model is very sensitive to the quality of BWEs and performs poorly on distant language pairs such as English-Japanese, as illustrated in their experimental study.
In parallel, cross-lingual language models (LMs) trained from raw Wikipedia texts, such as multilingual BERT $^{1}$  (Devlin et al., 2019) and XLM (Conneau and Lample, 2019), have been prevalent in solving zero-shot classification problems (Wu and Dredze, 2019). Those models use the BERT-style Transformer (Vaswani et al., 2017) architecture simultaneously trained from multiple languages to construct a sentence encoder, and fine-tune the encoder and a classifier on labeled training data from the source language. Then the fine-tuned model is applied to the target language. The whole process does not require any labeled data or parallel corpus. However, under the "zero parallel resource" setting, the encoder trained from self-supervised masked language modelling within each language may not well capture the semantic similarity among languages, which could harm the generalization performance of fine-tuned models.
In this paper, we propose a sentiment classification model called multi-view encoder-classifier (MVEC) in an unsupervised setting, in which we only have monolingual corpora from two languages and labels in the source language. Different from previous language model (LM) based fine-tuning approaches (Devlin et al., 2019; Conneau and Lample, 2019) that adjust parameters solely based on the classification error of training data, we utilize the encoder-decoder network from unsupervised machine translation (UMT) (Lample et al., 2018a) to regularize and refine the shared latent space. In particular, the transformer-based encoder regularized by a language discriminator learns shared but more refined language-invariant representations, which are effective for both reconstructing sentences from two languages by the decoder and generating multi-view feature representations for classification from input documents. In our model, we construct two views from the en
coder: (i) the encoded sentences in the source language; (ii) the encoded translations of the source sentences in the target language.
Our proposed MVEC is partially initialized by pretrained LMs (Conneau and Lample, 2019) but further fine-tuned to align sentences from two languages better, accurately predict labeled data in the source language and encourage consensus between the predictions from the two views. The full model is trained in an end-to-end manner to update parameters for the encoder-decoder, the language discriminator, and the classifier at each iteration.
- We present an unsupervised sentiment classification model without any labels or parallel resource requirements for the target language. By designing a multi-view classifier and integrating it with pretrained LMs and UMT (Lample et al., 2018a), we build our model (MVEC) on a more refined latent space that is robust to language shift with better model interpretation compared to previous zero-shot classification works (Chen et al., 2018; Conneau and Lample, 2019).
- We extensively evaluate our model in 5 language pairs involving 11 sentiment classification tasks. Our full model outperforms state-of-the-art unsupervised fine-tuning approaches and partially supervised approaches using crosslingual resources in 8/11 tasks. Therefore, our results provide a strong lower bound performance on what future semi-supervised or supervised approaches are expected to produce.
2.1 Cross-Lingual Text Classification (CLTC)
CLTC aims to learn a universal classifier that can be applied to languages with limited labeled data (Bel et al., 2003; Dong and de Melo, 2019; Keung et al., 2019), which is naturally applicable for sentiment analysis. Traditional supervised methods utilize cross-lingual tools such as machine translation systems and train a classifier on the source language (Prettenhofer and Stein, 2010). The latest models used parallel corpus either to learn a bilingual document representation (Zhou et al., 2016) or to conduct cross-lingual model distillation (Xu and Yang, 2017).
In the unsupervised setting, Chen et al. (2018) learned language-invariant latent cross-lingual representations with adversarial training. Ziser
$^{1}$ https://github.com/google-research/BERT/blob/master/multilingual.md
and Reichart (2018) used pivot based learning and structure-aware DNN to transfer knowledge to low-resourced languages. In both papers, however, they have an implicit dependency on BWEs, which requires a bilingual dictionary to train. Chen and Qian (2019) was the first fully unsupervised approach using the unsupervised BWEs (Lample et al., 2018b) and multi-source languages with adversarial training. In contrast, our model is a multi-view classification model that is seamlessly integrated pretrained LMs (Conneau and Lample, 2019) and the encoder-decoder from UMT (Lample et al., 2018a) with adversarial training. Hence we learn a more fine-tuned latent space to better capture document-level semantics and generate multiple views to represent the input.
2.2 Unsupervised Machine Translation
UMT does not rely on any parallel corpus to perform translation, which lays a foundation for our approach. At the word-level, Lample et al. (2018b) built a bilingual dictionary between two languages by aligning monolingual word embeddings in an unsupervised way. At the sentence and document level, Lample et al. (2018a) proposed a UMT model by learning an autoencoder that can reconstruct two languages under both within-domain and cross-domain settings. Lample et al. (2018c) extended Lample et al. (2018a) with a phrase-based approach. Since we aim to learn more refined language-invariant representations for classification, it is natural to employ the encoder from a UMT system to generate multiple views of the input and enable knowledge transfer.
2.3 Multi-View Transfer Learning
The task of multi-view transfer learning is to simultaneously learn multiple representations and transfer the learned knowledge from source domains to target domains, which have fewer training samples. Generally, data from different views contains complementary information and multi-view learning exploits the consistency from multiple views (Li et al., 2019).
Our work is particularly inspired by Fu et al. (2015) and Zhang et al. (2019), both of which exploit the complementarity of multiple semantic representations with semantic space alignment. The difference is that we use an encoder-decoder framework to generate multiple views for input from the source language and enforce a consensus between their predictions. Furthermore,
we introduce a language discriminator (Lample et al., 2018a) to encourage the encoder to generate language-invariant representations from the input.
In this section, we will introduce our model's general workflow, including the details of each component and our training algorithm.
Given monolingual text data  $\{D_{src}, D_{tgt}\}$  from both the source and target language with a subset of labeled samples  $\{D_{src}^{L}, y_{src}^{L}\}$  in the source language where  $y_{src}^{L}$  is a vector of class labels and  $D_{src}^{L} \subset D_{src}$ , the task aims to build a universal classification model  $f(X; \theta) \to y$  parameterized by  $\theta$  that can be directly applicable to unlabeled data in the target language, where  $X$  is an input document from any language and  $y$  is its class label. Note that in this paper we assume two languages share the same class types.
Our proposed approach multi-view encoder classifier (MVEC) is composed of three components: an encoder-decoder, a language discriminator, and a classifier. Motivated by the success of unsupervised machine translation (UMT) in Lample et al. (2018a) and reconstruction regularization by an autoencoder in Sabour et al. (2017), we adopt the encoder-decoder framework from UMT (Lample et al., 2018a) and introduce self-reconstruction loss within one language and back-translation reconstruction loss across languages together with the normal loss from classification. For simplicity, we denote self-reconstruction loss as "within-domain loss" and back-translation reconstruction loss as "cross-domain loss" throughout the paper.
Although the encoder from UMT can generate a latent representation for input sentences/documents, there is still a semantic gap between the source and target language. Following Lample et al. (2018a); Chen et al. (2018), we enrich the encoder-decoder framework with a language discriminator that can produce fine-tuned latent representations to align latent representations from two languages better. Such representations are necessary to train a language-invariant classifier that is robust to the shift in languages.
In particular, as illustrated in Figure 1, the encoder is used to encode source and target docu
5761
Figure 1: Multi-view encoder classifier (MVEC) architecture. Blue (red) lines indicate the message flow within the source/target language (across languages), respectively. Green lines indicate the message flow from the encoder to the text classifier. The encoder and decoder share the same parameters.
ments (a sequence of sentences) into a shared latent space, while the decoder is responsible for decoding the documents from the latent space to the source or the target language. Following Lample et al. (2018a), the encoder-decoder is shared for both languages (domains) and trained within-domain and cross-domain. The language discriminator aims to predict the language source for each document, and the classifier is trained to classify each document into predefined class labels.
Under the unsupervised setting, MVEC only observes unlabeled monolingual corpora from two languages and some labeled documents in the source language. The unlabeled monolingual data is normally sampled from the application domain, i.e., unlabeled product reviews or social media posts, which is used in both adopting pretrained LMs in the target domain and training UMT. As shown in Figure 1, unlabeled source and target data only pass through encoder-decoder and language discriminator, while the labeled source data pass all components in the system, including the sentiment classifier. For evaluation purposes, we may have labeled documents in the target language. However, they are only used during the test period. In the following subsections, we introduce each component of MVEC in detail.
3.3 Encoder-Decoder
Let  $x^{(l)} = (x_1^{(l)}, x_2^{(l)}, \dots, x_n^{(l)})$  denote the input document of  $n$  words from a particular lan
guage  $l$ , where  $l \in \{src, tgt\}$ . The encoder is a neural network  $e_{\theta_{enc}}(x^{(l)})$  parameterized by  $\theta_{enc}$  that produces a sequence of  $n$  hidden states  $\mathbf{Z}^{(l)} = (\mathbf{z}_1^{(l)}, \mathbf{z}_2^{(l)}, \dots, \mathbf{z}_n^{(l)})$  by using the corresponding word embedding for  $x_i^{(l)}$ , where  $\mathbf{z}_i^{(l)}$  is the latent representation of  $x_i^{(l)}$  in the shared latent space and  $\theta_{enc}$  are parameters of the encoder shared between two languages. The encoder could be a BiLSTM or a transformer (Vaswani et al., 2017). In this paper, we adopt the transformer, which has achieved enormous success in (e.g.,) recent text representation learning tasks (Devlin et al., 2019; Conneau and Lample, 2019).
Given  $\mathbf{Z}^{(l)}$  as the input, the decoder  $d_{\theta_{dec}}(\mathbf{Z}^{(l)})$  generates the output sequence  $y^{(l)} = (y_1^{(l)}, y_2^{(l)}, \dots, y_k^{(l)})$ . We use the same transformer based decoder as in Conneau and Lample (2019), parameterized by  $\theta_{dec}$ . For simplicity, we will denote the encoder and decoder by  $e(x^{(l)})$  and  $d(\mathbf{Z}^{(l)})$  respectively instead of  $e_{\theta_{enc}}(x^{(l)})$  and  $d_{\theta_{dec}}(\mathbf{Z}^{(l)})$ .
It is more likely for the encoder-decoder to merely memorize every input word one by one if there are no imposed constraints. To improve the robustness of encoder-decoder, we follow Lample et al. (2018a) to adopt the Denoising Autoencoders (DAE) (Vincent et al., 2008), which recovers input from its corrupted version.
There are three ways to inject noise into the document including shuffle, dropout, and replacement by special words. In our model, we drop and replace every word with probabilities of  $p_d$  and  $p_b$ , respectively, and we slightly shuffle the input document by implementing random permutation  $\sigma$  on the input document, where  $p_d$  and  $p_b$  can be viewed as hyper-parameters for controlling noise levels. In our design, the permutation  $\sigma$  satisfies the condition  $|\sigma(i) - i| \leq k$ ,  $\forall i \in \{1, \dots, n\}$ , where  $n$  is the length of input document and  $k$  is another hyper-parameter.
Note that the noise model is only applied to unlabeled data used for training the encoder-decoder and the discriminator, while labeled data will keep its originality for all components training. We use  $G(.)$  to denote a stochastic noise model, which takes input document  $x^{(l)}$  and generates  $G(x^{(l)})$  as a randomly sampled noisy version of  $x^{(l)}$ .
To incorporate the encoder-decoder as regularization components, we follow Lample et al. (2018a) to consider both within-domain and cross-domain objective functions. The first objective
5762
function aims to reconstruct a document from a noisy version of itself within a language, whereas the second (cross-domain) objective function targets to teach the model to translate an input document across languages. Specifically, given a language  $l \in \{src, tgt\}$ , the within-domain objective function can be written as:
$$
R _ {w d} (\pmb {\theta} _ {e d}, l) = \mathbb {E} _ {x \sim \pmb {D} _ {l}, \hat {x} \sim d (e (G (x)))} [ \Delta (x, \hat {x}) ] \quad (1)
$$
where  $\pmb{\theta}_{ed} = [\pmb{\theta}_{enc},\pmb{\theta}_{dec}],\hat{x}\sim d(e(G(x)))$  is a reconstruction of the corrupted version of  $x$  sampled from the monolingual dataset  $D_{l}$ , and  $\Delta$  is the sum of token-level cross-entropy loss to measure discrepancy between two sequences.
Similarly, we consider teaching the encoder-decoder to reconstruct  $x$  in one language from a translation of  $x$  in the other language, leading to the following cross-domain objective function:
$$
R _ {c d} \left(\boldsymbol {\theta} _ {e d}, l _ {1}, l _ {2}\right) = \mathbb {E} _ {x \sim \boldsymbol {D} _ {l _ {1}}, \hat {x} \sim d (e (T (x)))} [ \Delta (x, \hat {x}) ] \tag {2}
$$
where  $(l_1, l_2) \in \{(src, tgt), (tgt, src)\}$  and  $T(\cdot)$  is the current UMT model applied to input document  $x$  from language  $l_1$  to language  $l_2$ .
3.4 Language Discriminator
Cross-lingual classifiers work well when their input produced by the encoder is language-invariant, as studied in Chen et al. (2018). Thus, we prefer our encoder to map input documents from both languages into a shared feature space independent of languages. To achieve this goal, we follow Chen et al. (2018); Lample et al. (2018a) and introduce a language discriminator into our model, which is a feed-forward neural network with two hidden layers and one softmax layer to identify the language source from the encoder's output. In particular, we minimize the following cross-entropy loss function:
$$
L _ {D} \left(\boldsymbol {\theta} _ {D} \mid \boldsymbol {\theta} _ {e n c}\right) = - \mathbb {E} _ {(l, x ^ {(l)})} \left[ \log P _ {D} (l \mid e \left(x ^ {(l)}\right) \right] \tag {3}
$$
where  $\theta_{D}$  denotes parameters of the discriminator,  $(l,x^{(l)})$  corresponds to language and document pairs uniformly sampled from monolingual datasets, and  $P_{D}(.)$  is the output from the softmax layer. Meanwhile, the encoder is trained to "fool" the discriminator:
$$
L _ {a d v} \left(\boldsymbol {\theta} _ {e n c} \mid \boldsymbol {\theta} _ {D}\right) = - \mathbb {E} _ {x ^ {\left(l _ {i}\right)} \sim D _ {l _ {i}}} [ \log P _ {D} \left(l _ {j} \mid e \left(x ^ {\left(l _ {i}\right)}\right)\right) ] \tag {4}
$$
with  $l_{j} = l_{1}$  if  $l_{i} = l_{2}$ , and vice versa.
3.5 Multi-view Classifier
Thus far, we have described how we obtain a language-invariant latent space to encode two languages, which may not be sufficient to generalize well across languages if we simply train a classifier on the encoder's output for the source language (Chen et al., 2018). One key difference between Chen et al. (2018) and our work is that we use UMT (Lample et al., 2018a), which can generate multiple views for the input labeled documents from the source language. We can thereby benefit from multi-view learning's superior generalization capability over single-view learning (Zhao et al., 2017).
Particularly, we consider two views of input: (i) the encoded labeled documents from the source language; (ii) the encoded back-translations of the source documents from the target language. Our learning objective is to train the classifier to match predicted document labels with ground truth from the source language and to encourage two predictive distributions on the two views to be as similar as possible. We consider the following objective function:
$$
\begin{array}{l} L _ {C} \left(\boldsymbol {\theta} _ {C}, \boldsymbol {\theta} _ {e d}\right) = \mathbb {E} _ {(x, y)} \left[ \Delta \left(y, P _ {\boldsymbol {\theta} _ {c}} (e (x))\right) \right. \\ + \underbrace {D _ {K L} \left(P _ {\boldsymbol {\theta} _ {c}} (e (x)) \mid \mid P _ {\boldsymbol {\theta} _ {c}} (e (T (x))) \right]} _ {\text {T w o v i e w s ＂ c o n s e n s u s}} \tag {5} \\ \end{array}
$$
where  $(x,y)\sim \{D_{src}^{L},\mathbf{y}_{src}^{L}\}$ ,  $D_{KL}(.||.)$  is KL Divergence to measure the difference between two distributions,  $y$  is the class label of input document  $x$  and  $\theta_c$  are parameters of classifier. Following previous studies in text classification (Devlin et al., 2019), we use the first token's representation in the last hidden layer from the transformer encoder as the document representation vector. The classifier is a feed-forward neural network with two hidden layers and a softmax layer.
The final objective function at one iteration of our learning algorithm is to minimize the following loss function:
$$
\begin{array}{l} L _ {a l l} = L _ {C} + \lambda_ {w d} \times \left(R _ {w d \_ s r c} + R _ {w d \_ t g t}\right) \tag {6} \\ + \lambda_ {c d} \times \left(R _ {c d - s r c} + R _ {c d - t g t}\right) + \lambda_ {a d v} \times L _ {a d v} \\ \end{array}
$$
where  $\lambda_{wd},\lambda_{cd},\lambda_{adv}$  are hyper-parameters to trade-off among within-domain loss, the crossdomain loss and the adversarial loss, respectively.
3.6 Training Algorithm
Our model relies on an initial translation machine  $T^{(0)}$ , which provides a translation from one lan
5763
guage to another for calculating the cross-domain loss in Eq. (2) and classifier loss in Eq. (5).
To accelerate the training, we initialize  $T^{(0)}$  by pretraining a transformer-based UMT (Connieu and Lample, 2019) for certain steps with the same encoder-decoder architecture as our model on monolingual Wikipedia text. After pretraining, we use the pretrained encoder-decoder network to initialize our model and start training the classifier and the discriminator. Meanwhile, we refine the encoder and the decoder on monolingual data and labeled data from the source language.
During each training step, the optimization iterates from updating  $\theta_{D}$  in Eq. (3) to updating  $\theta_{ed}$  and  $\theta_{C}$  in Eq. (6). Note that if a batch of documents drawn from monolingual data are all unlabeled, then we suspend updating classifier parameters and only update the parameters of the language discriminator and encoder-decoder. In Algorithm 1, we provide a detailed procedure.
We conduct experiments on cross-lingual multiclass and binary sentiment classification using five language pairs involving 11 tasks. More specifically, English is always the source language, and the target languages are French, German, Japanese, Chinese, and Arabic, respectively.