SlowGuess/ACL_2020_OCR huggingface dataset:
We first train the BALM on the parallel corpus as described in Section 3.4. The trained network is then adapted with SEAME Adapt to bridge the domain gap, namely from  $l_{1}l_{2} \rightarrow l_{1}$  and  $l_{1}l_{2} \rightarrow l_{2}$  towards  $l_{1}l_{2} \rightarrow l_{1}l_{2}$ .
Synthetic CS In this contrastive experiment, we remove the bilingual constraint, i.e. equation 4, from BALM, and use offline synthetic CS text outlined in Lee et al. (2019) in the training. The idea of synthetic CS is motivated by the Matrix Language Frame theory. The phrase alignment is performed on the same parallel dataset in Table 1, using Giza++ (Och and Ney, 2003). The aligned parallel sentences are then used to randomly switch phrases between the languages according to an empirical probability of 0.7. At the same time the phrase table is used to inhibit switch within
frequently occurring phrases. We train the same BALM network with both the synthetic CS data and the monolingual side of the parallel data. The model is finally adapted with SEAME Adapt.
Mono & CS-Only In the Mono setting, we simply use parallel corpus as two independent monolingual corpora without any form of bilingual constraint. The monolingual sentences are passed alternating between the two languages to ensure a balanced training curriculum. The model is finally adapted with SEAME Adapt. This is similar to the Multilingual BERT pre-training under causal masking and subsequently fine-tune on the task dataset. The CS-Only model is trained only on the SEAME Adapt data without involving the parallel data.
Positional Embedding We also implement the sinusoidal encoding matrix (Vaswani et al., 2017) and the learned weight matrix for the positional embedding in model PE-S and PE-L respectively. Both models are implemented on top of the BALM model using the same training data. The positional embedding is an element-wise addition to the word embedding layer. For the learned matrix in PE-L, we treat it as another lookup table. We simply extend the embedding matrix with the additional entries for each pos. In the case of sinusoidal encoding, the extended matrix is fixed to be,
$$
\begin{array}{l} P E _ {(p o s, 2 i)} = s i n (p o s / 1 0 0 0 0 ^ {2 i / 3 8 4}) \\ P E _ {(p o s, 2 i + 1)} = c o s (p o s / 1 0 0 0 0 ^ {2 i / 3 8 4}). \\ \end{array}
$$
4.3 CS Point Perplexity
While the perplexity test on SEAME Test CS describes the overall performance of the model on CS sentences. As shown in Table 1, CS only takes place at an average occurrence (SPF) of  $23\%$  in the CS sentences. We would like to take a closer look at how the model performs only at those CS points, which is the main focus of this work. A lower perplexity suggests a better word prediction ability. The perplexity is evaluated on SEAME Test CS, in which we only include perplexity for the word that is preceded by a different language.
4.4 Bilingual Lexicon Induction
While BALM is mainly optimized for word prediction, it also establishes cross-lingual word correspondence through word embedding. To examine the quality of cross-lingual embedding, we conduct bilingual lexicon induction (BLI) experiments, and compare with other major cross-lingual pretraining models. The same parallel corpus in Ta
Models	Training Data	PPL (SEAME Test)	PPL (Test EN/ZH)	PPL (Test CS)	PPL (CS Points)	WER
CS only	SEAME Adapt	180.09	147.42/139.96	198.09	650.82	28.02%
Mono	Monolingual+SEAME Adapt	131.54	96.33/99.99	146.37	554.71	27.62%
Synthetic CS	Parallel+SEAME Adapt	124.65	95.13/99.91	139.17	506.81	26.42%
BALM	Parallel+SEAME Adapt	118.25	91.74/94.41	130.49	477.78	19.73%
+ PE-S	Parallel+SEAME Adapt	135.22	101.78/106.12	151.05	561.11	26.24%
+ PE-L	Parallel+SEAME Adapt	143.29	107.34/109.54	161.12	578.02	27.16%
Table 2: Perplexity is reported on different test subsets, and at CS Points of Test CS. Word Error Rate (WER) for language normalization is reported for experiments in Section 4.5.
Model	SEAME Dev	SEAME Eval
RNNLM* (Adel et al., 2013a)	246.60	287.88
FL + OF* (Adel et al., 2013a)	219.85	239.21
FLM* (Adel et al., 2013b)	177.79	192.08
LSTM (Winata et al., 2018)	150.65	153.06
Multi-task (Winata et al., 2018)	141.86	141.71
Synthetic CS (Lee et al., 2019)	142.41	142.53
CSLM (Lee and Li, 2019)	128.12	129.85
BALM	102.79	103.20
Table 3: Code-switch language models trained on SEAME Train (see Table 1). The models with \*\* are trained and tested on SEAME Phase I, which is approximately  $60\%$  smaller than SEAME Phase II.
ble 1 is used for training and the same dictionary2 is used for testing for all models.
VecMap<sup>3</sup> (Artetxe et al., 2018) is a projection based CLWE alignment method which gives robust results using a unsupervised strategy (Glavaš et al., 2019). The respective monolingual embeddings are trained using fastText<sup>4</sup> (Bojanowski et al., 2017) with the default setup and 384 dimensions. The two monolingual embedding space are then mapped using the VecMap. BiSkip<sup>5</sup> (Luong et al., 2015) is jointly trained with word alignment constraint. We prepare the alignment using fast_align<sup>6</sup> (Dyer et al., 2013) following the similar procedure outlined in the paper.
For the BALM model, we use the embedding from the model without the SEAME adaptation phase for a fair comparison. These three models represent three distinct categories in CLWE implementation, i.e. projection-based, jointly learned, and deep learning based embedding for VecMap, BiSkip and BALM, respectively.
4.5 Language Normalization
Suppose that  $l_{1}$  is the matrix language in a code-switch sentence  $\mathbf{w}$ . We would like to replace all  $l_{2}$  tokens in  $\mathbf{w}$  with their  $l_{1}$  equivalent tokens, that is referred to as  $l_{1}l_{2} \rightarrow l_{1}$ . The normalized sentence  $\hat{\mathbf{w}}^{l_{1}}$  can be expressed as,  $\hat{\mathbf{w}}^{l_{1}} = \arg \max_{\mathbf{w}^{l_{1}}} p(\mathbf{w}^{l_{1}}|\mathbf{w})$ .
In practice, when  $\mathbf{w}$  is presented to BALM, as illustrated in Figure 1c, the network predicts a sequence of tokens one by one in the matrix language as follows,
$$
\hat {\mathbf {w}} ^ {l _ {1}} = \arg \max  _ {\{w _ {t} ^ {l _ {1}} \}} \prod_ {i = 1} ^ {t} p (w _ {t} ^ {l _ {1}} | \mathbf {w}, \mathbf {w} _ {i <   t} ^ {l _ {1}}), \quad (5)
$$
The generated tokens  $\mathbf{w}_{i < t}^{l_1}$  becomes the context for the next token  $w_{t}^{l_{1}}$  in an auto-regressive manner. The sequence with the highest probability is simply computed using beam search, which is performed when the  $\in o s$  token is observed.
5.1 Perplexity Evaluation
We conduct two perplexity (PPL) test experiments, one for comparing the variations of BALM, another for benchmarking against the state-of-the-art.
Comparing the variations of BALM, we report the overall test PPL as well as the PPL of each components, i.e. Test EN/ZH and Test CS for each model discussed in Section 4.2. It is observed in Table 2 that BALM outperforms all other variations, with a PPL of 118.25 on SEAME Test. Mono, Synthetic CS and BALM all benefit from the use of data beyond SEAME Adapt. BALM represents the most effective use of the bilingual parallel corpus. All the results are reported according to the best performing model on SEAME Valid dataset.
Benchmarking against the state-of-the-art, we show in Table 3 that BALM achieves a PPL of 103.20 on SEAME Eval, which is a  $20.52\%$  reduc
$^{2}$ https://github.com/facebookresearch/MUSE#ground-truth-bilingual-dictionaries
<sup>3</sup>https://github.com/artetxem/vecmap
$^{4}$ https://github.com/facebookresearch/fastText
$^{5}$ https://github.com/lmthang/bivec
No.	Code-switch sentence	Normalized	Reference
1	when there is still test then 他们会练 like once a week or once in two weeks	when there is still test then they will practise like once a week or once in two weeks	when there is still test then they will practise like once a week or once in two weeks
2	i have a high chance of being 拒绝 by her because obviously 我跟她很 不熟	i have a high chance of being re-jected by her because obviously i am not very familiar with	i have a high chance of being re-jected by her because obviously i am not very familiar with her
3	but comparative to last year i think 已经蛮不错了	but comparative to last year i think is quite good lah	but comparative to last year i think is quite good already
4	开学之前 i have already secured a job	开学之前我已经有了一个工作	开学之前我已经找到了一个工作
5	这种活动 is a bit challenging	这种活动是有点困难的	这种活动是有点困难的
6	星期六我就要 hand in 我的 assign-ment 了	星期六我就要去做我的任务了	星期六我就要交我的功课了
tion over the best reported result of 129.85 (Lee and Li, 2019) on the same test data in the literature.
5.1.1 CS point perplexity
Let us examine the perplexity only at CS points. In Table 2, from CS-Only to Mono, we observe a  $14.8\%$  PPL reduction, from 650.82 to 554.71, as a result of the additional monolingual data. We have seen similar results in Lee et al. (2019); Gonen and Goldberg (2019). Our observation is also very similar to M-Bert and corroborates with the findings of Pires et al. (2019). The monolingual data contribute to a better word embedding, which is an integral part of the BALM. As the quality of the word embedding improves, so does the word prediction at the CS points.
We also observe that Synthetic CS shows a  $8.6\%$  PPL reduction, from 554.71 to 506.81 with the inclusion of the synthetic CS data. This is consistent with the observations in Lee et al. (2019) and Pratapa et al. (2018).
We further observe that BALM, which is trained on exactly the same parallel data as in Synthetic CS, but with a different objective function, outperforms Synthetic CS by  $5.73\%$ . This suggests that the quasi-translation loss function is an effective regularizer to enforce the linguistic constraint governing CS. We also confirm our aforementioned hypothesis that self-attention mechanism is able to attend to the appropriate bilingual context for word prediction without violating the grammar of the matrix language by qualitatively analysing the generated sentences from the model not yet adapted with CS adapt.
5.1.2 Positional embedding
Both the sinusoidal encoding and the learned encoding matrix degrade the model performance by  $14.4\%$  and  $21.2\%$  respectively. This result con
Table 4: Samples of language normalized CS text and the reference
Method	EN-ZH	ZH-EN
VecMap (Artetxe et al., 2018)	57.13%	48.46%
BiSkip (Luong et al., 2015)	35.54%	33.39%
BALM (our work)	56.24%	55.87%
Vocabulary Coverage	38.84%	31.72%
Table 5: BLI accuracy  $(\%)$  for different methods on the same parallel corpus in Table 1 for training and the same dictionary2 for testing.
firms our hypothesis that the attention mechanism is able to encode the mixed context well without positional embedding. The improvement of BALM over BALM+PE in the monolingual PPL also demonstrates that dropping the positional embedding is in fact beneficial.
5.2 Bilingual Lexicon Induction
The comparable performance justifies the premise that the model is able to find word-level correspondence, which enables the subsequent bilingual context encoding. As shown in Table 5, when inferring ZH (Chinese) words from EN (English), BALM  $(56.24\%)$  shows comparable performance with VecMap  $(57.13\%)$ , that reported the state-of-the-art results in CLWE. However, BALM significantly outperforms VecMap in the inverse pair ZH-EN with an absolute  $7.41\%$  improvement  $(48.46\% \rightarrow 55.87\%)$ .
Two points to take note of, firstly, Glavaš et al. (2019) point out that BLI cannot be used as the only metric to assess the word embedding quality and we do not intend to do so. Secondly, while it is true that VecMap does not need the corpus to be parallel and ours does, so the comparison did not showcase the best ability of VecMap. However, the focus of this paper is not on comparing the best cross-lingual word embedding methods. We use
BLI performance as evidence to support our claim that BALM does not compromise on its CLWE while focusing on sequential modeling.
5.3 Language Normalization
As the code-switch sentence follows the syntactic structure of the matrix language, we assume that the matrix language is known in advance, for example, English for sentences 1-3, and Chinese for sentences 4-6 in Table 4. We observe that sometimes, mistakes can take the form of bad translation, however the normalized sentence still maintains an appropriate structure of the matrix language. The  $6^{th}$  sentence of Table 4 is an example, which is wrongly normalized to "to do my assignment (in the sense of task)" instead of "hand in my assignment (in the sense of homework)". We report the WER on SEAME Norm between the normalized text and the reference. We observe in Table 2 that, with a WER of  $19.73\%$ , BALM outperforms other models in the same way as in the perplexity tests.
We note that BALM is an implementation of  $l_{1}l_{2} \rightarrow l_{1}l_{2}$ . The experiments show that it outperforms all state-of-the-art models in the literature for similar tasks. The results validate the idea of bilingual attention. The same BALM can be used in  $l_{1}l_{2} \rightarrow l_{1}$  or  $l_{2}$  for language normalization. It can be further extended for  $l_{1} \rightarrow l_{1}l_{2}$ , or  $l_{2} \rightarrow l_{1}l_{2}$  for code switch sentence generation, and  $l_{1} \rightarrow l_{2}$ , or  $l_{2} \rightarrow l_{1}$  for machine translation.
We thank the reviewers for their insightful feedback. This research is supported by the National Research Foundation Singapore under its AI Singapore Programme (Award No. AISG-GC-2019-002); the Programmatic Grant No. A18A2b0046 from the Singapore Government's Research, Innovation and Enterprise 2020 plan (Advanced Manufacturing and Engineering domain), Project title: Human Robot Collaborative AI for AME; and the National Research Foundation, Prime Minister's Office, Singapore under the National Robotics Programme, Project title: Human-Robot Interaction Phase 1 (Grant No. 192 25 00054).
Heike Adel, Dominic Telaar, Ngoc Thang Vu, Katrin Kirchhoff, and Tanja Schultz. 2014. Combining re
current neural networks and factored language models during decoding of code-switching speech. In NTERSPEECH-2014, pages 1415-1419.
Heike Adel, Ngoc Thang Vu, Franziska Kraus, Tim Schlippe, Haizhou Li, and Tanja Schultz. 2013a. Recurrent neural network language modeling for code switching conversational speech. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8411-8415.
Heike Adel, Ngoc Thang Vu, and Tanja Schultz. 2013b. Combination of recurrent neural networks and factored language models for code-switching language modeling. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 206-211, Sofia, Bulgaria. Association for Computational Linguistics.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2017. Learning bilingual word embeddings with (almost) no bilingual data. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 451-462, Vancouver, Canada. Association for Computational Linguistics.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 789-798, Melbourne, Australia. Association for Computational Linguistics.
Hedi M Belazi, Edward J Rubin, and Almeida Jacqueline Toribio. 1994. Code switching and x-bar theory: The functional head constraint. Linguistic inquiry, pages 221-237.
Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural Language Processing with Python, 1st edition. O'Reilly Media, Inc.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146.
Pi-Chuan Chang, Michel Galley, and Christopher D. Manning. 2008. Optimizing Chinese word segmentation for machine translation performance. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 224-232, Columbus, Ohio. Association for Computational Linguistics.
Alexis Conneau, Guillaume Lample, Marc'Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2017. Word translation without parallel data. CoRR, abs/1710.04087.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th
Annual Meeting of the Association for Computational Linguistics, pages 2978-2988, Florence, Italy. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Anne-Marie Di Sciullo, Pieter Muysken, and Rajendra Singh. 1986. Government and code-mixing. Journal of linguistics, 22(1):1-24.
Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of IBM model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644-648, Atlanta, Georgia. Association for Computational Linguistics.
Manaal Faruqui and Chris Dyer. 2014. Improving vector space word representations using multilingual correlation. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 462-471, Gothenburg, Sweden. Association for Computational Linguistics.
Goran Glavaš, Robert Litschko, Sebastian Ruder, and Ivan Vulić. 2019. How to (properly) evaluate cross-lingual word embeddings: On strong baselines, comparative analyses, and some misconceptions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 710–721, Florence, Italy. Association for Computational Linguistics.
Hila Gonen and Yoav Goldberg. 2019. Language modeling for code-switching: Evaluation, integration of monolingual data, and discriminative training. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4173-4183, Hong Kong, China. Association for Computational Linguistics.
Bradley Hauer, Garrett Nicolai, and Grzegorz Kondrack. 2017. Bootstrapping unsupervised bilingual lexicon induction. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 619-624, Valencia, Spain. Association for Computational Linguistics.
Karl Moritz Hermann and Phil Blunsom. 2014. Multilingual models for compositional distributed semantics. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics
(Volume 1: Long Papers), pages 58-68, Baltimore, Maryland. Association for Computational Linguistics.
Kejun Huang, Matt Gardner, Evangelos Papalexakis, Christos Faloutsos, Nikos Sidiropoulos, Tom Mitchell, Partha P. Talukdar, and Xiao Fu. 2015. Translation invariant word embeddings. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1084-1088, Lisbon, Portugal. Association for Computational Linguistics.
Kazuki Irie, Albert Zeyer, Ralf Schlüter, and Hermann Ney. 2019. Language Modeling with Deep Transformers. In Proc. Interspeech 2019, pages 3905-3909.
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization.
Guillaume Lample and Alexis Conneau. 2019. Crosslingual language model pretraining.
Grandee Lee, Thi-Nga Ho, Eng-Siong Chng, and Haizhou Li. 2017. A review of the Mandarin-English code-switching corpus: SEAME. In Asian Language Processing (IALP), 2017 International Conference on, pages 210-213. IEEE.
Grandee Lee and Haizhou Li. 2019. Word and class common space embedding for code-switch language modelling. In ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6086-6090.
Grandee Lee, Xianghu Yue, and Haizhou Li. 2019. Linguistically Motivated Parallel Data Augmentation for Code-Switch Language Modeling. In Proc. Interspeech 2019, pages 3730-3734.
Omer Levy, Anders Søgaard, and Yoav Goldberg. 2017. A strong baseline for learning cross-lingual word embeddings from sentence alignments. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 765–774, Valencia, Spain. Association for Computational Linguistics.
Ying Li and Pascale Fung. 2013. Improved mixed language speech recognition using asymmetric acoustic model and language model with code-switch inversion constraints. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 7368-7372.
Ying Li and Pascale Fung. 2014. Language modeling with functional head constraint for code switching speech recognition. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 907-916, Doha, Qatar. Association for Computational Linguistics.
Pierre Lison and Jörg Tiedemann. 2016. OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles. In Proceedings of the Tenth
International Conference on Language Resources and Evaluation (LREC), pages 923-929.
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Bilingual word representations with monolingual quality in mind. In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing, pages 151-159, Denver, Colorado. Association for Computational Linguistics.
Tomas Mikolov, Quoc V. Le, and Ilya Sutskever. 2013. Exploiting similarities among languages for machine translation. CoRR, abs/1309.4168.
Carol Myers-Scotton. 1997. Duelling languages: Grammatical structure in codeswitching. Oxford University Press.
Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19-51.
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual bert? CoRR, abs/1906.01502.
Shana Poplack. 2000. Sometimes i'll start a sentence in spanish yerno en elspanol: Toward a typology of code-switching. The bilingualism reader, 18(2):221-256.
Adithya Pratapa, Gayatri Bhat, Monjit Choudhury, Sunayana Sitaram, Sandipan Dandapat, and Kalika Bali. 2018. Language modeling for code-mixing: The role of linguistic theory based synthetic data. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1543-1553, Melbourne, Australia. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog.
Sebastian Ruder, Ivan Vulic, and Anders Søgaard. 2019. A survey of cross-lingual word embedding models. Journal of Artificial Intelligence Research, 65:569-631.
David Sankoff. 1998. The production of code-mixed discourse. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics-Volume 1, pages 8-21. Association for Computational Linguistics.
Samuel L. Smith, David H. P. Turban, Steven Hamblin, and Nils Y. Hammerla. 2017. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. CoRR, abs/1702.03859.
Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC'12), pages 2214-2218.