SlowGuess/ACL_2020_OCR huggingface dataset:
Grandee Lee
grandee.lee@u.nus.edu
Haizhou Li
Kriston AI Lab, China
haizhou.li@nus.edu.sg
Language modeling is the technique to estimate the probability of a sequence of words. A bilingual language model is expected to model the sequential dependency for words across languages, which is difficult due to the inherent lack of suitable training data as well as diverse syntactic structure across languages. We propose a bilingual attention language model (BALM) that simultaneously performs language modeling objective with a quasi-translation objective to model both the monolingual as well as the cross-lingual sequential dependency. The attention mechanism learns the bilingual context from a parallel corpus. BALM achieves state-of-the-art performance on the SEAME code-switch database by reducing the perplexity of  $20.5\%$  over the best-reported result. We also apply BALM in bilingual lexicon induction, and language normalization tasks to validate the idea.
Monolingual language modeling has enabled many NLP tasks (Devlin et al., 2019; Dai et al., 2019; Radford et al., 2019). However, the bilingual language model was not well studied. The recent advances in cross-lingual word embedding (CLWE) (Ruder et al., 2019), which projects word of different languages into a shared embedding space for cross-lingual representations (Devlin et al., 2019; Lample and Conneau, 2019), make possible some cross-lingual applications. Unfortunately, they are not optimized to model the sequential dependency for word prediction in a bilingual text.
In this paper, we would like to propose a bilingual language model that can learn word embeddings to represent the equivalent words between two languages, and more importantly, to model the sequential dependency for words across languages at the same time. For instance, the model should be able to predict the appropriate word to fill in the blank, given the bilingual context:
昨晚的movie( ).1
The above sentence is an example of code-switching or code-mixing (henceforth, CS), where a bilingual speaker alternates words of two or more languages within a single sentence. The switches could happen at sentence boundaries or word boundaries and for some agglutinative languages even within words. Code-switching is common in both spoken and, to some extent, written communication in many multilingual societies, such as Southeast Asia. Hence, the study of code-switch in linguistics and bilingual language modeling is becoming imperative, especially for NLP tasks such as code-switching automatic speech recognition (ASR) (Adel et al., 2013b; Li and Fung, 2013; Lee et al., 2019), cross-lingual language normalization.
It is tempting to think that, given enough of code-switching text data, bilingual language modeling could be approached in the same way as that for monolingual data. The main challenge is the lack of such CS data. We note that CS mainly occurs in the spoken form, and CS does not occur in every sentence. Therefore, collecting enough pure CS data is just not practical or even feasible (Lee et al., 2017; Pratapa et al., 2018).
The problem is further exacerbated by the syntactic constraints of the two diverse languages, such as Chinese and English. Three dominant theories seek to explain the syntactic formation of CS sentences. They are the Matrix Language Frame theory (Myers-Scotton, 1997), which shows that individual monolingual sentences will conform to the grammar of the matrix language. The Equivalence Constraint theory (Poplack, 2000; Sankoff, 1998), which further constrains the intra-sentential CS points to the syntactic boundaries shared by both languages, and the Functional Head Constraint theory (Di Sciullo et al., 1986; Belazi et al., 1994) that imposes constraints on the functional head and its
1English:Themovie last night
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 860-870 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
complements.
A bilingual language model should be able to predict a word, either in the matrix language or otherwise, given either a bilingual or monolingual context. Therefore, it has to respect the respective monolingual word sequential dependency, the cross-lingual word correspondence, as well as the switching rules between languages. The contributions of this paper are summarized as follows:
1. We propose an attention-based, auto-regressive model, bilingual attention language model (BALM), that not only learns the latent alignment from a parallel corpus for cross-lingual word embedding but also captures the word sequential dependency.
2. Adhering to the Matrix Language Frame theory (Myers-Scotton, 1997) and Equivalence Constraint theory (Poplack, 2000; Sankoff, 1998), we implement an objective function by jointly optimizing the cross-entropy loss as the monolingual constraint and the quasi-translation loss as the cross-lingual constraint.
3. We show that BALM can learn from bilingual parallel data without the need for CS data. When adapted on CS data, it outperforms the best reported result on the SEAME dataset in the perplexity test. We also successfully apply BALM in bilingual lexicon induction, and language normalization tasks to validate the idea.
Several prior studies related to bilingual language modeling are the inspiration for this work.
Cross-lingual correspondence: Several studies are focused on projecting words of different languages onto the common embedding space to establish cross-lingual correspondence. One idea is to train a model using bilingual information from corpora aligned at the sentence level (Zou et al., 2013; Hermann and Blunsom, 2014; Luong et al., 2015) and document level (Vulic and Moens, 2016; Levy et al., 2017). Another is to exploit the isomorphic structure (Conneau et al., 2017; Artetxe et al., 2018), dictionary (Mikolov et al., 2013; Faruqui and Dyer, 2014; Huang et al., 2015; Zhang et al., 2016), shared cognate, vocab (Hauer et al., 2017; Smith et al., 2017), numeral (Artetxe et al., 2017) through ad-hoc projection.
As the above approaches do not explicitly consider the sequential dependency of words, the embedding doesn't encode the word ordering information. The multilingual techniques, such as MBERT (Devlin et al., 2019) and XLM (Lample and Conneau, 2019), do not explicitly model the syntactic constraints for CS as formulated in the Equivalence Constraint theory, thus not making full use of the information which could potentially improve their performance.
Code-switching modeling: Another school of thoughts is to extend the monolingual language modeling technique to accommodate code-switch content. Adel et al. (2013b, 2014) use factored language models and recurrent neural network (RNN) language model to improve the bilingual language model for CS ASR rescoring. They include additional linguistic information such as Part-of-Speech, language identifier to improve model generalization. Inversion constraints (Li and Fung, 2013) and Functional Head constraints (Li and Fung, 2014) are also used in language models for the ASR decoding process. Lee and Li (2019) use cross-lingual embedding to tie the input and output layer, and incorporate classes in the RNN language model. While these models are effective, they rely on the availability of CS training data. Therefore, they are not easily scalable. To address this, we propose a way to make use of the existing abundant parallel corpora. The method will be explained in Section 3.3.
Code-switching text generation: Closer to our line of research, Pratapa et al. (2018) propose to use synthetic data following the Equivalence Constraint theory, while Lee et al. (2019) apply the Matrix Language Frame theory. In their works, a parser or an aligner is required to process the parallel corpus, which is followed by the standard monolingual language modeling process. Such techniques suffer from inaccurate alignment or parsing errors. These errors will be carried forward when training the language model. More recently, Winata et al. (2019) propose a technique to generate neural-based synthetic data using parallel sentences, in which a Point-Gen network is used to synthesize CS data without external aligner or parser. In this paper, we propose to learn the bilingual context and the CS language model jointly by attending to the parallel sentences directly without the need for an external aligner, parser or explicitly generating the synthetic data.
3 Bilingual Attention Language Model
Next, we discuss the motivation and the theoretical formulation of the proposed Bilingual Attention Language Model (BALM). In a bilingual text, we could encounter a sequence of word,  $\mathbf{w} = w_{1}^{l_{1}}, w_{2}^{l_{2}}, \ldots, w_{t}^{l_{2}}, \ldots, w_{T}^{l_{1}}$ , code mixed between languages  $l_{1}$  and  $l_{2}$ . However, such code mixed training data are not easily available. Let us assume that only parallel corpus at sentence level between  $l_{1}$  and  $l_{2}$  languages is available to us.
Assuming the validity of the Matrix Frame theory, and Equivalence Constraint theory, the above code-switch sentence,  $\mathbf{w}$ , can be constructed from two parallel sentences,  $\mathbf{w}^{l_1} = w_1^{l_1}, w_2^{l_1}, \ldots, w_{T_1}^{l_1}, \mathbf{w}^{l_2} = w_1^{l_2}, w_2^{l_2}, \ldots, w_{T_2}^{l_2}$ . For a monolingual case, the language model maximizes the log-likelihood of  $p(w_t | \mathbf{w}_{<t})$  which effectively captures the monolingual word sequential dependency. For a CS case, we would like to maximize  $p(w_t | \mathbf{w}_{<t})$ , whereby the bilingual context,  $\mathbf{w}_{<t}$ , is non-existent during training. In the subsequent section, we will explain the idea to encode the bilingual context using an attention mechanism.
A bilingual language model has to be built on a common word representation. The continuous space word embedding is an effective solution. We first draw some principled insights from the crosslingual word embedding (CLWE) study, which motivates this work.
Building on the idea of CLWE, we refer to the general form of the loss function,  $J$ , summarized by Ruder et al. (2019) as follows,
$$
J = \mathcal {L} \left(\mathbf {X} ^ {l _ {1}}\right) + \mathcal {L} \left(\mathbf {X} ^ {l _ {2}}\right) + \Omega \left(\mathbf {X} ^ {l _ {1}}, \mathbf {X} ^ {l _ {2}}, \mathbf {A}\right). \tag {1}
$$
The monolingual language constraint  $\mathcal{L}$ , which could be implemented with negative sampling, preserves the monolingual integrity. Importantly, there has to be a cross-lingual constraint, which could be the mean squared error (MSE) between the  $l_{2}$  embedding space  $\mathbf{X}^{l_2} = \{x_i^{l_2}\}$ , and the transformed  $l_{1}$  embedding space,  $\mathbf{X}^{l_1} = \{x_i^{l_1}\}$ . We use  $x_{i}$  to denote the embedding of a word  $w_{i}$ , which is also referred to as a token. The vocabulary size is  $v$ . The cross-lingual language constraint  $\Omega$  maps the two monolingual embeddings into a common space using the transformation matrix  $\mathbf{A}$ ,
$$
\Omega_ {M S E} = \sum_ {i = 1} ^ {v} | | \mathbf {A} x _ {i} ^ {l _ {1}} - x _ {i} ^ {l _ {2}} | |. \tag {2}
$$
The CLWE network can also be jointly learned (Luong et al., 2015) with the alignment information as the regularization loss,  $\Omega$ . While CLWE lays the foundation for many cross-lingual applications, it is not designed to model word sequential dependency.
3.2 Bilingual Objective
We draw inspiration from the CLWE loss function and extend the objective function to the modeling of word sequential dependency while preserving its general form.
The monolingual objective,  $\mathcal{L}(\mathbf{X}^l)$  as formulated in Equation 3, is set to be the cross entropy loss between the target distribution,  $y^{l}$  and the predicted distribution  $\log p(w_t^l |\mathbf{w}_{< t}^l)$ , for the respective language, which preserves the monolingual word sequential order.
$$
\mathcal {L} \left(\mathbf {X} ^ {l}\right) = y ^ {l} \log p \left(w _ {t} ^ {l} \mid \mathbf {w} _ {<   t} ^ {l}\right), l \in \left\{l _ {1}, l _ {2} \right\} \tag {3}
$$
This allows the bilingual language model to adhere to the monolingual syntactic rules of the Matrix Language Frame and the Equivalent Constraint theory during word prediction, that the dominant language still abide by its own syntactic principle.
We also define a quasi-translation loss,  $\Omega$ , that optimizes the model to learn the correspondence of tokens between languages as well as the dependencies between the current token in  $l_{1}$  and the preceding context in  $l_{2}$ . The quasi-translation loss can be interpreted as satisfying the requirement of the code-switching principle as described by the two theories.
$$
\Omega_ {l _ {1} l _ {2} \rightarrow l _ {1}} = y ^ {l _ {1}} \log p \left(w _ {t} ^ {l _ {1}} \mid \mathbf {w} ^ {l _ {2}}, \mathbf {w} _ {<   t} ^ {l _ {1}}\right) \tag {4}
$$
Equation 4 is the quasi-translation loss,  $\Omega_{l_1l_2\to l_1}$  when predicting a word in  $l_{1}$  given a bilingual context. Similarly, we have  $\Omega_{l_1l_2\rightarrow l_2}$  to predict a word in  $l_{2}$ .
3.3 Bilingual Attention
Motivated by the self-attention model (Vaswani et al., 2017), we hypothesize that an auto-regressive translation-cum-language modeling objective could leverage on parallel sentences to learn the bilingual context.
To start with, let us consider a monolingual case that deals with  $l_{1}$ . We define a transformer language model,  $f$ , using a causal mask (Radford et al., 2019), which can be further broken down
(a) Train on  $l_{1}l_{2}\rightarrow l_{2}$  data.
(b) Estimate  $p(\text{"you"}|\mathbf{w}_{< 5})$
(c)Normalize  $l_{1}l_{2}\to l_{1}$
Figure 1: (a) Trained on a parallel sentence pair  $l_{1}l_{2}$ , "i like you" and "我喜欢你", BALM learns to predict the next  $l_{2}$  word, "你", given its context  $\mathbf{x}_{<3}^{l_{2}}$ , "我喜欢", and its whole sentence translation  $\mathbf{x}_{<5}^{l_{1}}$ , "i like you". (b) During perplexity evaluation, BALM estimates the probability of  $p(\text{"you"}| \mathbf{w}_{<5})$ , given a bilingual context  $\mathbf{w}_{<5}$ , "他也是 like". (c) Normalizing a  $l_{1}l_{2}$  code-switch sentence to  $l_{1}$  with BALM by generating the  $l_{1}$  sentence sequentially in an auto-regressive manner.  $x = \text{embed}(w)$  is the cross-lingual word embedding layer and the transpose of the embed weight is used for the output projection layer to decode the word distribution.
into individual layer  $n$  in a total of  $N$  layers,
$$
f _ {1} ^ {n} = A t t e n t i o n (\mathbf {x} _ {<   t} ^ {l _ {1}}))
$$
$$
f _ {2} ^ {n} = \text {F e e d F o r w a r d} \left(f _ {1} ^ {n}\right)
$$
$$
f ^ {n} = f _ {2} ^ {n} \circ f _ {1} ^ {n}
$$
The model will take in the embedding,  $x_{t}^{l_{1}} = \text{embed}(w_{t}^{l_{1}})$  of each word,  $w_{t}^{l_{1}}$ , in  $l_{1}$  at the first layer,  $f_{1}^{1}$ , and the output will encode the contextual information that is a weighted sum of its preceding context,  $f^{1} = f_{2}^{1}(\text{Attention}(\mathbf{x}_{< t}^{l_{1}}))$ . In this way, the output of the last layer  $f_{2}^{N}$  contains the information, that is necessary for decoding  $p(w_{t}^{l_{1}}|\mathbf{w}_{< t}^{l_{1}})$ . This process is carried out on the monolingual side of the parallel data respectively for  $l_{1}$  and  $l_{2}$  to minimize the loss function in Equation 3.
Extending the context of  $l_{1}$  to include words in  $l_{2}$ , we enable the model to learn from a bilingual context, as shown in Figure 1a. The question is how to find the appropriate context in both  $l_{1}$  and  $l_{2}$  to predict a word in  $l_{2}$ . The attention mechanism with the quasi-translation loss provides a solution. Figure 1a is an illustration for  $l_{1}l_{2} \rightarrow l_{2}$  training case.
At the last layer, the encoded output for the time step  $t$  in  $l_{2}$  will be,  $f_{2}^{N}(\text{Attention}(\mathbf{x}^{l_{1}}, \mathbf{x}_{\leq t}^{l_{2}}))$ . It is important to note that the model architecture allows learnable alignment between current word  $x_{t}$  with its preceding context in its own language  $l_{2}$  as well as the whole sentence translation  $\mathbf{x}^{l_{1}}$  in  $l_{1}$ . The use of preceding context can be seen as an autoregressive process over the words in a sentence.
As the predicted word always follows its preceding context sequentially, the word order in the matrix language matters in BALM. However, the attention mechanism does not attempt to distinguish word order within the encoded context, which is a weighted sum of the bilingual context (see discussions in Section 3.5). This can be observed in the quasi-translation loss, as formulated in Equation 4.
During training, we use the two sides of the parallel corpus independently as two monolingual corpora and both sides together as the bilingual constraint. When presented with monolingual text in  $l_{1}$  or  $l_{2}$ , the network learns to attend to the words in either  $l_{1}$  or  $l_{2}$  using a causal mask for monolingual word prediction. When presented with  $l_{1}l_{2}$  parallel sentences, and predicting a word in  $l_{1}$  or  $l_{2}$ , the network learns to attend to the bilingual context for word prediction.
To summarize, given a parallel corpus, BALM is trained with 4 input  $\rightarrow$  output pairs,  $l_{1}\rightarrow l_{1}$ $l_{2}\to l_{2}$ $l_{1}l_{2}\rightarrow l_{1}$  , and  $l_{1}l_{2}\rightarrow l_{2}$  . The bilingual attention in theory allows BALM to take any of  $l_{1}$ $l_{2}$  or  $l_{1}l_{2}$  as input, and generate any of  $l_{1},l_{2}$  or  $l_{1}l_{2}$  as output in 6 possible combinations.  $l_{1}l_{2}\rightarrow l_{1},l_{2}$  represents the code-switch language modeling task of our interest. For brevity, we only illustrate the case of  $l_{1}l_{2}\rightarrow l_{2}$  in Figure 1a.
At run time inference, we do not have the two parallel sentences, but rather a code-switch sentence that consists of a mixture of words  $\mathbf{w}_{< t}$  from
the two languages, as in Figure 1b. To predict  $p(w_{t}^{l_{2}}|\mathbf{w}_{< t})$  for a code-switch sentence at run time, we assume that the model would have encountered some variants of the bilingual context through (Attention( $\mathbf{x}_{< t}^{l_{1}},\mathbf{x}_{< t}^{l_{2}}$ )). In this way, the model can estimate the run time probability according to the similarity between the encoding of the code-switch sequence,  $\mathbf{w}_{< t}$ , and the learned bilingual representation. The attention-based alignment is expected to find the appropriate bilingual context that was trained under the objective function to maximize  $p(w_{t}^{l_{2}}|\mathbf{w}_{< t}^{l_{1}},\mathbf{w}_{< t}^{l_{2}})$ .
3.5 Positional Embedding
In stark contrast to the masked language model (MLM), which employs positional embedding on top of its sequence ordering invariant setup, BALM does not use positional embedding. We argue that under the auto-regressive objective, positional embedding is not necessary.
In BALM, the amount of information in an auto-regressive setup is strictly increasing. Taking one of its intermediate layers as an example, the hidden representation for the current token  $h_t$  is the weighted sum of the previous tokens, and the weights are computed through the learned query and key matrix,  $\mathbf{A}_Q, \mathbf{A}_K$ .
$$
h _ {t} = a _ {1, t} x _ {1} + a _ {2, t} x _ {2} + \dots + a _ {t, t} x _ {t}
$$
$$
a _ {n, m} = \mathbf {A} _ {K} x _ {n} \cdot \mathbf {A} _ {Q} x _ {m}
$$
In comparison with a RNN layer, whereby the hidden state is a gated sum of the previous hidden states, i.e.  $h_t = \tanh(\mathbf{W}_h h_{t-1} + \mathbf{W}_x x_t)$ , the difference is that the weight matrix,  $\mathbf{W}_h$ , for RNN is applied on the gated sum,  $h_{t-1}$ , at each time step while the weight for the attention model,  $a_{n,m}$ , is a similarity comparison of the current token's query with the previous tokens' keys.
The two networks are similar in the sense that they both compute the weights and incorporate the past information. They only differ in their implementation. We argue that the sequential information is already included in the attention model under an auto-regressive setup. Thus the positional encoding is not necessary. This is corroborated by Irie et al. (2019), which shows that the removal of positional encoding slightly improves the language model performance. By dropping the positional embedding, we can mix the bilingual context, as discussed in Section 3.3.
We evaluate the language models on the text transcripts of the South East Asia Mandarin-English (SEAME) corpus (LDC2015S04) (Lee et al., 2017), a well-documented database for spontaneous conversational speech code-switching between Chinese Mandarin (ZH) and English (EN). A large number of CS studies were reported on SEAME.
We adopt a slightly different setup as we focus on how BALM is able to learn from a parallel corpus alone without the need of CS training data. We use SEAME data mainly for adaptation and evaluation. We split the SEAME Phase II text transcripts equally into three portions, labeled as Adapt, Valid and Test respectively in Table 1. Such split also ensures that the individual component within the Test data, e.g. Test EN, is of sufficient size.
Additionally, we also split the dataset following approximately the same proportion as in the previous works (Winata et al., 2019; Lee et al., 2019) for a fair benchmarking, labeled as Train, Dev, and Eval respectively. We use a random split of  $1.1M / 60.8K / 60.3K$  for the number of tokens in Train/Dev/Eval as compared to  $1.2M / 65K / 60K$  in the previous works.
We use a bilingual parallel corpus from Ted and OpenSubtitle (Tiedemann, 2012; Lison and Tiedemann, 2016) for BALM training because they are text transcripts of spontaneous speech similar to SEAME. The English text is tokenized using NLTK tokenizer (Bird et al., 2009) while the Chinese text is tokenized using Stanford Word Segmenter (Chang et al., 2008). We also develop a test set of 200 sentences for language normalization experiments, labeled as SEAME Norm.
We conduct a series of experiments, namely BALM, Synthetic CS, CS-Only, and Mono, using the same BALM network architecture to evaluate different modeling strategies.
During training, we construct a 50K vocabulary consisting of the most frequent words in the combined SEAME and parallel dataset, of which there are  $17.7K$  and  $32.3K$  unique Chinese and English words, respectively. Only for the benchmarking in Table 3, we use the SEAME vocabulary, a subset of the 50K vocabulary, for the perplexity evaluation to meaningfully compare the perplexity with the prior work on SEAME corpus.
Dataset	#Lines	#Tokens	#Vocab	SPF
Ted+OpenSubtitle*	3.6M	234.4M	366.7K	0
SEAME Adapt	30.9K	398.4K	14.1K	0.17
SEAME Valid	30.9K	399.1K	14.1K	0.17
SEAME Test	30.9K	400.8K	14.0K	0.17
-Test CS	18.9K	284.8K	11.9K	0.23
-Test EN	5.8K	58.5K	4.3K	0
-Test ZH	6.2K	57.5K	3.3K	0
SEAME Norm	200	1.8K	650	0.26
SEAME Train	82.3K	1.1M	20.7K	0.17
SEAME Dev	4.6K	60.8K	5.7K	0.16
SEAME Eval	4.6K	60.3K	5.9K	0.17
Table 1: Test CS, Test EN, and Test ZH represent code-switching, pure English, and pure Chinese partition of SEAME Test respectively. SPF refers to Switching Point Fraction Pratapa et al. (2018). Ted+OpenSubtitle* is a bilingual parallel corpus.
Unless otherwise stated, we train for 60 epochs with 100K lines per epoch and adapt for 17 epochs with the full Adapt dataset. We use Adam optimizer (Kingma and Ba, 2014) for all the experiments.
BALM The attention mechanism follows largely the implementation of GPT (Radford et al., 2019), with 384-dimension hidden states, 12 layers and 12 heads. While Dai et al. (2019) reports state-of-the-art results using the recurrence mechanism within the attention, we exclude this in our experiment for two reasons. Firstly, the context beyond the given parallel sentence is not meaningful after shuffling the sentences. Furthermore, attending target sequence to context beyond the source sequence may introduce noise and depart from the theoretical motivation of the experiment. Secondly, for many downstream tasks like ASR, the decoding remains at the utterance level.