Learn to Code-Switch: Data Augmentation using Copy Mechanism on Language Modeling

Abstract: Building large-scale datasets for training code-switching language models is challenging and very expensive. To alleviate this problem using parallel corpus has been a major workaround. However, existing solutions use linguistic constraints which may not capture the real data distribution. In this work, we propose a novel method for learning how to generate code-switching sentences from parallel corpora. Our model uses a Seq2Seq model in combination with pointer networks to align and choose words from the monolingual sentences and form a grammatical code-switching sentence. In our experiment, we show that by training a language model using the augmented sentences we improve the perplexity score by 10% compared to the LSTM baseline.

Language mixing has been a common phenomenon in multilingual communities. It is motivated in response to social factors as a way of communication in a multicultural society. From a sociolinguistic perspective, individuals do code-switching in order to construct an optimal interaction by accomplishing the conceptual, relational-interpersonal, and discourse-presentational meaning of conversation BIBREF0 . In its practice, the variation of code-switching will vary due to the traditions, beliefs, and normative values in the respective communities. A number of studies BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 found that code-switching is not produced indiscriminately, but follows syntactic constraints. Many linguists formulated various constraints to define a general rule for code-switching BIBREF1 , BIBREF3 , BIBREF4 . However, the constraints are not enough to make a good generalization of real code-switching constraints, and they have not been tested in large-scale corpora for many language pairs.
One of the biggest problem in code-switching is collecting large scale corpora. Speech data have to be collected from a spontaneous speech by bilingual speakers and the code-switching has to be triggered naturally during the conversation. In order to solve the data scarcity issue, code-switching data generation is useful to increase the volume and variance. A linguistics constraint-driven generation approach such as equivalent constraint BIBREF5 , BIBREF6 is not restrictive to languages with distinctive grammar structure.
In this paper, we propose a novel language-agnostic method to learn how to generate code-switching sentences by using a pointer-generator network BIBREF7 . The model is trained from concatenated sequences of parallel sentences to generate code-switching sentences, constrained by code-switching texts. The pointer network copies words from both languages and pastes them into the output, generating code switching sentences in matrix language to embedded language and vice versa. The attention mechanism helps the decoder to generate meaningful and grammatical sentences without needing any sequence alignment. This idea is also in line with code-mixing by borrowing words from the embedded language BIBREF8 and intuitively, the copying mechanism can be seen as an end-to-end approach to translate, align, and reorder the given words into a grammatical code-switching sentence. This approach is the unification of all components in the work of BIBREF5 into a single computational model. A code-switching language model learned in this way is able to capture the patterns and constraints of the switches and mitigate the out-of-vocabulary (OOV) issue during sequence generation. By adding the generated sentences and incorporating syntactic information to the training data, we achieve better performance by INLINEFORM0 compared to an LSTM baseline BIBREF9 and INLINEFORM1 to the equivalent constraint.
The synthetic code-switching generation approach was introduced by adapting equivalence constraint on monolingual sentence pairs during the decoding step on an automatic speech recognition (ASR) model BIBREF5 . BIBREF10 explored Functional Head Constraint, which was found to be more restrictive than the Equivalence Constraint, but complex to be implemented, by using a lattice parser with a weighted finite-state transducer. BIBREF11 extended the RNN by adding POS information to the input layer and factorized output layer with a language identifier. Then, Factorized RNN networks were combined with an n-gram backoff model using linear interpolation BIBREF12 . BIBREF13 added syntactic and semantic features to the Factorized RNN networks. BIBREF14 adapted an effective curriculum learning by training a network with monolingual corpora of both languages, and subsequently train on code-switched data. A further investigation of Equivalence Constraint and Curriculum Learning showed an improvement in language modeling BIBREF6 . A multi-task learning approach was introduced to train the syntax representation of languages by constraining the language generator BIBREF9 .
A copy mechanism was proposed to copy words directly from the input to the output using an attention mechanism BIBREF15 . This mechanism has proven to be effective in several NLP tasks including text summarization BIBREF7 , and dialog systems BIBREF16 . The common characteristic of these tasks is parts of the output are exactly the same as the input source. For example, in dialog systems the responses most of the time have appeared in the previous dialog steps.
We use a sequence to sequence (Seq2Seq) model in combination with pointer and copy networks BIBREF7 to align and choose words from the monolingual sentences and generate a code-switching sentence. The models' input is the concatenation of the two monolingual sentences, denoted as INLINEFORM0 , and the output is a code-switched sentence, denoted as INLINEFORM1 . The main assumption is that almost all, the token present in the code-switching sentence are also present in the source monolingual sentences. Our model leverages this property by copying input tokens, instead of generating vocabulary words. This approach has two major advantages: (1) the learning complexity decreases since it relies on copying instead of generating; (2) improvement in generalization, the copy mechanism could produce words from the input that are not present in the vocabulary.
Instead of generating words from a large vocabulary space using a Seq2Seq model with attention BIBREF17 , pointer-generator network BIBREF7 is proposed to copy words from the input to the output using an attention mechanism and generate the output sequence using decoders. The network is depicted in Figure FIGREF1 . For each decoder step, a generation probability INLINEFORM0 INLINEFORM1 [0,1] is calculated, which weights the probability of generating words from the vocabulary, and copying words from the source text. INLINEFORM2 is a soft gating probability to decide whether generating the next token from the decoder or copying the word from the input instead. The attention distribution INLINEFORM3 is a standard attention with general scoring BIBREF17 . It considers all encoder hidden states to derive the context vector. The vocabulary distribution INLINEFORM4 is calculated by concatenating the decoder state INLINEFORM5 and the context vector INLINEFORM6 . DISPLAYFORM0 
where INLINEFORM0 are trainable parameters and INLINEFORM1 is the scalar bias. The vocabulary distribution INLINEFORM2 and the attention distribution INLINEFORM3 are weighted and summed to obtain the final distribution INLINEFORM4 . The final distribution is calculated as follows: DISPLAYFORM0 
We use a beam search to select INLINEFORM0 -best code-switching sentences and concatenate the generated sentence with the training set to form a larger dataset. The result of the generated code-switching sentences is showed in Table TABREF6 . As our baseline, we compare our proposed method with three other models: (1) We use Seq2Seq with attention; (2) We generate sequences that satisfy Equivalence Constraint BIBREF5 . The constraint doesn't allow any switch within a crossing of two word alignments. We use FastAlign BIBREF18 as the word aligner; (3) We also form sentences using the alignments without any constraint. The number of the generated sentences are equivalent to 3-best data from the pointer-generator model. To increase the generation variance, we randomly permute each alignment to form a new sequence.
The quality of the generated code-switching sentences is evaluated using a language modeling task. Indeed, if the perplexity in this task drops consistently we can assume that the generated sentences are well-formed. Hence, we use an LSTM language model with weight tying BIBREF19 that can capture an unbounded number of context words to approximate the probability of the next word. Syntactic information such as Part-of-speech (POS) INLINEFORM0 is added to further improve the performance. The POS tags are generated phrase-wise using pretrained English and Chinese Stanford POS Tagger BIBREF20 by adding a word at a time in a unidirectional way to avoid any intervention from future information. The word and syntax unit are represented as a vector INLINEFORM1 and INLINEFORM2 respectively. Next, we concatenate both vectors and use it as an input INLINEFORM3 to an LSTM layer similar to BIBREF9 .
In our experiment, we use a conversational Mandarin-English code-switching speech corpus called SEAME Phase II (South East Asia Mandarin-English). The data are collected from spontaneously spoken interviews and conversations in Singapore and Malaysia by bilinguals BIBREF21 . As the data preprocessing, words are tokenized using Stanford NLP toolkit BIBREF22 and all hesitations and punctuations were removed except apostrophe. The split of the dataset is identical to BIBREF9 and it is showed in Table TABREF6 .
In this section, we present the experimental settings for pointer-generator network and language model. Our experiment, our pointer-generator model has 500-dimensional hidden states and word embeddings. We use 50k words as our vocabulary for source and target. We evaluate our pointer-generator performance using BLEU score. We take the best model as our generator and during the decoding stage, we generate 1-best and 3-best using beam search with a beam size of 5. For the input, we build a parallel monolingual corpus by translating the mixed language sequence using Google NMT to English ( INLINEFORM0 ) and Mandarin ( INLINEFORM1 ) sequences. Then, we concatenate the translated English and Mandarin sequences and assign code-switching sequences as the labels ( INLINEFORM2 ).
The baseline language model is trained using RNNLM BIBREF23 . Then, we train our 2-layer LSTM models with a hidden size of 500 and unrolled for 35 steps. The embedding size is equal to the LSTM hidden size for weight tying. We optimize our model using SGD with initial learning rates of INLINEFORM1 . If there is no improvement during the evaluation, we reduce the learning rate by a factor of 0.75. In each time step, we apply dropout to both embedding layer and recurrent network. The gradient is clipped to a maximum of 0.25. Perplexity measure is used in the evaluation.
UTF8gbsn The pointer-generator significantly outperforms the Seq2Seq with attention model by 3.58 BLEU points on the test set as shown in Table TABREF8 . Our language modeling result is given in Table TABREF9 . Based on the empirical result, adding generated samples consistently improve the performance of all models with a moderate margin around 10% in perplexity. After all, our proposed method still slightly outperforms the heuristic from linguistic constraint. In addition, we get a crucial gain on performance by adding syntax representation of the sequences.
Change in data distribution: To further analyze the generated result, we observed the distribution of real code-switching data and the generated code-switching data. From Figure FIGREF15 , we can see that 1-best and real code-switching data have almost identical distributions. The distributions are left-skewed where the overall mean is less than the median. Interestingly, the distribution of the 3-best data is less skewed and generates a new set of n-grams such as “那个(that) proposal" which was learned from other code-switching sequences. As a result, generating more samples effects the performance positively.
Importance of Linguistic Constraint: The result in Table TABREF9 emphasizes that linguistic constraints have some significance in replicating the real code-switching patterns, specifically the equivalence constraint. There is a slight reduction in perplexity around 6 points on the test set. In addition, when we ignore the constraint, we lose performance because it still allows switches in the inversion grammar cases.
Does the pointer-generator learn how to switch? We found that our pointer-generator model generates sentences that have not been seen before. The example in Figure FIGREF1 shows that our model is able to construct a new well-formed sentence such as “我们要去(We want to) check". It is also shown that the pointer-generator model has the capability to learn the characteristics of the linguistic constraints from data without any word alignment between the matrix and embedded languages. On the other hand, training using 3-best data obtains better performance compared to 1-best data. We found a positive correlation from Table TABREF6 , where 3-best data is more similar to the test set in terms of segment length and number of switches compared to 1-best data. Adding more samples INLINEFORM0 may improve the performance, but it will be saturated at a certain point. One way to solve this is by using more parallel samples.
We introduce a new learning method for code-switching sentence generation using a parallel monolingual corpus that is applicable to any language pair. Our experimental result shows that adding generated sentences to the training data, effectively improves our model performance. Combining the generated samples with code-switching dataset reduces perplexity. We get further performance gain after using syntactic information of the input. In future work, we plan to explore reinforcement learning for sequence generation and employ more parallel corpora.

Figure and table captions:
Fig. 1. Pointer Generator Networks [8]. The figure shows the example of input and 3-best generated sentences.
Table 3. Language Modeling Results (in perplexity).
Table 1. Data Statistics of SEAME Phase II and Generated Sequences using Pointer-generator Network [10].
Table 2. Code-Switching Sentence Generation Results. Higher BLEU and lower perplexity (PPL) is better.
Fig. 2. Univariate data distribution for unigram (topleft), bigram (top-right), trigram (bottom-left), and fourgram (bottom-right). The showed n-grams are sampled from 3-best data pointer-generator model.

Questions about this paper:
What was their perplexity score?
What parallel corpus did they use?

Answers:
Perplexity score 142.84 on dev and 138.91 on test
Parallel monolingual corpus in English and Mandarin