CoMix: Guide Transformers to Code-Mix using POS structure and Phonetics
Year: 2023

Abstract:
Code-mixing is ubiquitous in multilingual societies, which makes it vital to build models for code-mixed data to power human language interfaces. Existing multilingual transformer models trained on pure corpora lack the ability to intermix words of one language into the structure of another. These models are also not robust to orthographic variations. We propose CoMix1, a pretraining approach to improve representation of code-mixed data in transformer models by incorporating phonetic signals, a modified attention mechanism, and weak supervision guided generation by partsof-speech constraints. We show that CoMix improves performance across four code-mixed tasks: machine translation, sequence classification, named entity recognition (NER), and abstractive summarization. It also achieves new SOTA performance for English-Hinglish translation and NER on LINCE Leaderboard and provides better generalization on out-ofdomain translation. Motivated by variations in human annotations, we also propose a new family of metrics based on phonetics and demonstrate that the phonetic variant of BLEU correlates better with human judgement than BLEU on code-mixed text.

heading: 1 Introduction
text: Code-mixing, i.e., embedding linguistic units of one language (embedded language LE) into a sentence grammatically structured as per another language (matrix language LM ), is common in multilingual communities. Growing mobile penetration coupled with the increased adoption of informal conversational interfaces is leading to further rise in such communication. Currently, over 20% of user generated content from South Asia and parts of Europe is code-mixed (Choudhury et al., 2019). Hinglish (code-mixed Hindi-English) has nearly
1CoMix is not a trademark and only used to refer to our models for code-mixed data for presentational brevity.
350 million speakers (GTS, 2019) making it one of the most widely spoken languages. Recent literature suggests that multilingual users associate codemixing with cultural affinity and prefer chatbots that can code-mix (Bawa et al., 2020). Code-mixed modeling is, thus, a foundational prerequisite for linguistic systems targeted towards such users.
Transformer models such as BART (Lewis et al., 2020) and BERT (Devlin et al., 2018) have been successful across various NLP tasks. These models can readily capture code-mixing semantics if a large corpus was available for training. Unfortunately, that is not true for most code-mixed languages. Existing approaches rely on learning from a parallel corpus of embedded and matrix languages (e.g., English and Hindi for Hinglish). Recent work (Chen et al., 2022), however, shows that multilingual models such as mBERT trained on monolingual sources fail to effectively interleave words from topologically diverse languages.
Adapting transformers to code-mixed data requires addressing the following challenges: 1. Divergent grammatical structure. For codemixed languages such as Hinglish, where LE and LM have different Parts-of-Speech (POS) patterns, models trained on monolingual corpora do not yield similar representations for equivalent words across languages, which is needed to facilitate interleaving of LE and LM words. Linguistic theories propose certain syntactic constraints for code-mixed generation (Poplack, 1980), but these are not usually incorporated into the modeling. 2. Code-mixing diversity. Code-mixed languages also exhibit a wide diversity in the degree of code-mixing (e.g., ratio of LE to LM words). Fig 1 shows multiple Hinglish constructions for a given sentence in English. Accounting for this variation in code-mixing is necessary for high fidelity modeling. 3. Orthographic variations. The informal nature of code-mixed interactions and lack of standardized transliteration rules leads to users employing adhoc
7985
phonological rules while writing code-mixed content. Fig 1 shows Hinglish sentences with similar sounding words and their variations (“kis”, “kys”). Contributions. In this paper, we adapt transformer models for code-mixed data by addressing the above challenges. To ensure applicability to multiple downstream tasks, we focus on pretraining. 1. We propose CoMix, a set of generic pretraining methods to improve code-mixed data representations that can be applied to any transformer model assuming the availability of POS-tagger and phonetic transcription tools. These include: (a) Domain Knowledge-based Guided Attention (DKGA) mechanism that facilitates intermixing of linguistic units of LE into the structure of LM through a modified attention function, (b) Weakly Supervised Generation (WSG) that generates code-mixed data for training in a controllable fashion driven by linguistic constraints, and (c) inclusion of phonetic signals to align embeddings of similar sounding with different orthographic representation. 2. We instantiate CoMix pretraining for BART and BERT and demonstrate efficacy on multiple downstream NLP tasks, namely Machine Translation, NER, Sequence Classification, and Abstractive Summarization with relative improvements of up to 22%. CoMixBART and CoMixBERT achieve new state-of-the-art (SOTA) results for EnglishHinglish translation and Hinglish NER tasks on LINCE Leaderboard (Aguilar et al., 2020), beating previous best mT5 (Jawahar et al., 2021) and XLMR (Winata et al., 2021) models, despite having less than 0.5x and 0.1x model size respectively. 3. We evaluate out-of-domain code-mixed translation performance on two test sets, one created in-house and other one adapted from GupShup corpus (Mehnaz et al., 2021), and show that CoMix generalizes better than other models. To the best of our knowledge, this is the first such evaluation for English-Hinglish translation. We hope our benchmark will assist the community to improve out-ofdomain generalization of code-mixed translation, a critical need for low-resource regimes. 4. To address the limitations of existing metrics in handling orthographic variations in code-mixed data, we propose a new family of natural language generation (NLG) metrics based on phonetic adaptation of existing metrics. We observe that PhoBLEU, the phonetic variant BLEU, is better aligned to human judgement (+0.10 - 0.15 on Pearson correlation) than BLEU on Hinglish.

heading: 2 Related Work
text: Multilingual and Code-Mixed NLP. Recent advances in large multilingual pre-trained models such as mBERT (Devlin et al., 2018) and mBART (Liu et al., 2020) have led to significant gains on many multilingual NLP tasks. However, evaluation of these models on code-mixed content for machine translation (Chen et al., 2022), sequence classification (Patwa et al., 2020), summarization (Mehnaz et al., 2021) and other tasks (Aguilar et al., 2020) points to their inability to intermix words from two languages since these are pretrained on monolingual text without any language alternation. Our CoMix approach encourages the model to learn representations that allows appropriate embedding of words from one language into structure of another via domain knowledge guided attention and through weakly supervised code-mixed generation. Prior work (Sanad Zaki Rizvi et al., 2021) focuses on generating synthetic code-mixed data using constraints from linguistic theories followed by learning. We perform joint generation and learning using pretrained models that has dual benefit of data generation and improving model representations, and has been shown to be effective for anomaly detection in medical images (Li et al., 2019). Incorporating Phonetics in Language Modeling. Combined modeling of phonemes and text has been a topic of recent interest and has contributed in improving robustness to ASR errors (Sundararaman et al., 2021). In code-mixed domain, Soto et al. (Soto and Hirschberg, 2019) engineered spelling and pronunciation features by calculating distance between pairs of cognate words to improve perplex-
ity of English-Spanish models. We also incorporate phonetic signals to learn robust representations. Sentence Evaluation Metrics. Automated sentence evaluation metrics such as BLEU (Papineni et al., 2002), ROUGE (Lin, 2004) for comparison of unstructured sentences led to rapid innovation in NLP by facilitating ready evaluation of NLP systems against ground truth without additional human annotations. However, these metrics are unreliable for code-mixed content as they disregard widely prevalent orthographic variations. We propose a new family of metrics to address this gap.

heading: 3 CoMix Approach
text: Given a corpus of sentence pairs from LM and LE , our goal is to adapt transformer models such as BART and BERT to overcome the key challenges in modeling code-mixed data. To ensure applicability to multiple downstream tasks, we focus on the pretraining phase. We assume access to POS tagging and phonetic transcription tools 2 which is true for many languages (see Section 7). Below we summarize our approach for each of the challenges. P1 - Divergence in POS structure of LE and LM : To enable transformer models to extrapolate from LE and LM to code-mixed data, we rely on linguistic constraints. We observe that coarse groups of POS labels of concepts are preserved across translation (see Section 7) and that code-mixed sequences often retain the POS structure of LM sequence. Assuming access to POS labels the above constraints provide token-level correspondence for parallel training sentences, which can be used to augment the transformer attention mechanism and lead to representations that facilitate accurate interleaving of LE and LM words. [Section 3.1] P2 - Variations in the level of code-mixing: To accurately model variations in code-mixed data such as the mixing propensity, we propose a weakly supervised approach that combines POS constraints with a control on the code-mixing probability to generate code-mixed sequences from parallel monolingual corpora3 for training. [Section 3.2] P3 - Orthographic variations: To align similar sounding words with orthographic variations, we incorporate phonetic signal as an additional input channel. We modify the transformer architecture
2We use Stanza for POS-tagging and Refined Soundex implementation Pyphonetics for phonetic transcription.
3By parallel monolingual corpora we mean parallel corpora wherein each one of the parallel sentences are in pure/monolingual form of their corresponding language.
to include two multi-head self-attention layers, one each for text and phoneme channels. [Section 3.3]

heading: 3.1 Domain Knowledge Guided Attention
text: Attention (Vaswani et al., 2017) is an essential mechanism of transformer architecture that converts an input sequence into a latent encoding using representational vectors formed from the input, i.e., queries, keys and values to determine the importance of each portion of input while decoding. Let X and Z denote the sequence of input tokens and the associated representations. Further, let Q, K, V denote the sequences of query, key and value vectors derived from appropriate projections of Z. In this case, attention is typically defined in terms of the scaled dot-product of Q and K.
To incorporate domain knowledge, we propose augmenting attention with an additional independent term fDKGA(X) defined on the input:
Attention(Q,K, V ) = softmax(QK T+fDKGA(X)√
dk )V,
(1) where dk is the dimension of query and key vectors. While the notion of DKGA is general, to aid with code-mixing, we focus on linguistic constraints. We construct three groups of POS labels (see A.1) that are preserved during translation (see 7). Let X denote the concatenation of the parallel monolingual sentences, i.e., X = XM∥XE , where XM and XE are sentences in LM and LE respectively. Let POSGP(x) denote the group of the POS label of a token x. The linguistic constraints require that aligned token pairs from XM and XE belong to the same POS label group. Hence, for matrix tokens, we restrict attention to compatible embedded
words by choosing fDKGA(X) = [fDKGAij ], where
fDKGAij =    0 if POSGP(xi) = POSGP(xj) and xi ∈ XM , xj ∈ XE or i = j for xi ∈ XE
−∞ otherwise. (2)
Note that the above asymmetric choice is motivated by the fact that code-mixed sentences retain the POS structure of LM . Fig 2 shows how tokens from XE are selected using the above strategy which coupled with self-attention ensures learning representations that facilitate better intermixing of LE tokens into LM structure. See A.4.9 for example. Instead of a hard constraint on POS-label preservation, the DKGA function can also be modified to incorporate soft transition probabilities of POS labels during an LM to LE translation, which could be learned from parallel sentence pairs with token level alignment. We can also extend DKGA to include other sources for attention guidance, e.g., domain ontologies, word-alignment and also for cross-attention. Pretraining with DKGA. We modify all selfattention blocks in the encoder with DKGA and pretrain CoMixBERT with masked language modeling (MLM) objective (Devlin et al., 2018) and CoMixBART with denoising objective (text infilling with span=1). We mask the tokens of XM for which we want DKGA to guide attention to embedded words (e.g., in Fig 2, "kapde" will be masked).

heading: 3.2 Weakly Supervised Generation (WSG)
text: Lack of large code-mixed corpora poses a big challenge for code-mixed modeling. Hence, to facilitate direct training and allow control over desired properties such as the level of code mixing, we propose a weakly supervised mechanism for code-mixed generation using any transformer-based encoderdecoder model. The key idea is to nudge a pretrained multilingual model to code-mix by restricting the search space in the autoregressive step to a small suitable set of LE tokens exploiting the the fact that tokens with similar meaning and POS labels in LE are likely to replace the LM token.
Fig 3 shows the generative mechanism (Equation 3 shows the corresponding equations). At each auto-regressive step, we first determine the choice to code-mix, denoted by Mi, sampled based on the mixing probability pMix of the POS label of the token xi ∈ XM and an overall code-mixing
level τ . The vocabulary search space denoted by Vi is chosen as the POS-compatible words (same POS group as that of xi) from XE for code-mixing, and the entire vocabulary V all otherwise. The next token is generated via greedy search with teacher forcing. In case of code-mixing, the target yi is set to the predicted value ŷi and xi otherwise. We train the model with negative log-likelihood loss using XM as the input, Y = [yi]Ni=1 as the target, Ŷ = [ŷi]Ni=1 as the prediction. Due to the self-dependency in WSG, the efficacy depends on whether the underlying model can correctly order the tokens in Vi, which is a reasonable expectation from SOTA pretrained multilingual models.
ŷi = argmaxy∈ViP (y|y1, y2, ..., yi−1, XM ),
yi = { ŷi if Mi = 1 xi if Mi = 0 ,
Vi =    {xj |xj ∈ XE , POSGP(xj) = POS
GP(xi)} if Mi = 1 V all if Mi = 0 ,
Mi ∼ Bernoulli(τpMixPOS(xi)). (3)
In our experiments, we set τ = 1 and pMix
to 1 for POS groups {NOUN, PROPN,ADJ, ADV, V ERB} where code-mixing is frequent and 0 for the rest but can learn it from a small codemixed corpus in future. The proposed WSG mechanism can also be applied to encoder-only models such as BERT by considering a similar restriction of the vocabulary set Vi at the last layer.

heading: 3.3 Mixing Phonetic Signal
text: Given a text sequence X , let XPh denote the corresponding phonetic sequence. To incorporate both the signals, we replace the multi-head self attention layer in the transformer encoder layer with two multi-head self attention layers, one each for
text and phoneme channel. Text sequence shares feed-forward layers with the phonetic sequence as shown in Fig 4 since we want phonetic representations to be in the same space as text representations . To keep the number of parameters in check, we add phonetics part of the encoder to only alternate encoder layers. Our decoder uses the concatenated sequence of contextual embeddings from X and XPh as keys and values for cross attention. Pretraining with Phonetics. We pretrain CoMixBERT for phonetics with MLM objective (as in BERT) and CoMixBART with denoising objective (text infilling with span length 1).

heading: 4 Phonetic Sentence Comparison Metrics
text: Lack of standardized transliteration rules is a key challenge not only for modeling but also for evaluating code-mixed or multi-lingual NLG tasks, e.g., English to Hinglish translation Human annotators employ orthographic variations and are also inconsistent in the use of punctuation and upper-lower casing as shown in Fig 1. Most NLG evaluation metrics such as BLEU, do not account for these variations, which leads to pessimistic and inaccurate estimates of the performance of NLG systems. To address this gap, we propose a new family of metrics based on the phonetic representation. Let s(·, ·) be any metric such as BLEU and ROUGE (Banerjee and Lavie, 2005) that facilitates comparison of a word sequence against a reference one. Given a pair of sentences (X,Y ), we define the phonetic metric as Pho-s(X,Y ) = s(XPh, Y Ph), where XPh, Y Ph are the phonetic sequences. In this paper, we limit our focus to PhoBLEU and
present observations in Sec 6.6.

heading: 5 Experiments

heading: 5.1 Downstream Tasks, Baselines and Metrics
text: Table 1 lists the four downstream tasks, baselines, SOTA models and metrics used in the evaluation4. For translation on HooD dataset (Sec 5.2), we also include Echo baseline which just passes the input sequence as output and helps measure the contribution of input-output overlap to the final performance.
Previous studies (Dabre et al., 2021) (Kakwani et al., 2020) indicate that IndicBART (Dabre et al., 2021) and IndicBERT (Kakwani et al., 2020) are competitive relative to mBART and mBERT respectively on Indic languages. Further, since we initialize our models CoMixBART and CoMixBERT with weights from IndicBART and IndicBERT, we consider these as strong baselines for our evaluation of generative and classification tasks respectively.

heading: 5.2 Datasets for Downstream Tasks
text: We evaluate on LINCE Eng-Hinglish dataset for translation (Chen et al., 2022), SemEval-2020 Hinglish Sentimix dataset for sequence classification (Patwa et al., 2020), GupShup Hinglish chats to summaries (GupShup H2H) dataset (Mehnaz et al., 2021) for summarization and LINCE Hinglish (Singh et al., 2018) dataset for the NER task. Table 9 in Appendix A.3.2 lists data statistics. Hinglish Out-of-Domain Translation Dataset (HooD). We introduce two out-of-domain translation test-sets for Hinglish. Of these, the first one from shopping domain was prepared by inhouse human experts who translated English sentences generated by humans and models like GPT3, following the guidelines in Appendix A.3.1. The second test set was prepared from GupShup corpus (Mehnaz et al., 2021) from parallel EnglishHinglish summaries of conversations created by linguists (Gliwa et al., 2019). These datasets can help
4Formulations of all tasks are in A.2.
assess the zero-shot transfer capabilities of models from movies to shopping and open-domain for models trained on LINCE English-Hinglish dataset. Table 2 shows statistics of the HooD dataset.

heading: 5.3 Experimental Setup

heading: 5.3.1 Pretraining
text: Initialization. Training large transformer models from scratch requires thousands of GPU hours, which can be prohibitive. To ensure broader accessibility and best utilise existing models, we initialize CoMixBART decoder and encoder’s nonphonetic weights (NPW) from IndicBART and CoMixBERT’s NPW from IndicBERT. These are pretrained using Samanantar English-Hindi parallel corpus (Ramesh et al., 2021). CoMixBART. We pretrain CoMixBART with DKGA on 1M sentences from Samanantar for 36k steps (∼ 2 epochs) on three 24GB GPUs with batch size of 2816 tokens, linear learning rate warmup and decay with 16k warmup steps. We use Adam optimizer with max learning rate of 1e-3, label smoothing of 0.1, dropout of 0.1 and token masking probability of 0.4. For WSG, we pretrain the DKGA model for additional 2k steps with the same setup except label smoothening and masking probability of 0. Learning curve for pretraining with DKGA and WSG is shown in Appendix A.4.2. Since pretraining CoMixBART for phonetics from scratch is computationally prohibitive because of its size, we devise a way to obtain reasonable weights for downstream training. We initialize embeddings of phonetic tokens with the mean of embeddings of the text tokens that map to it. We also initialize phonetic self-attention layer parameters with the same weights as that of corresponding text channel’s self-attention layer. CoMixBERT. We pretrain CoMixBERT with DKGA, WSG and Phonetics on 100k sentences from Samanantar on six 32GB GPUs with batch size of 20 per GPU, starting with a learning rate of 5e-5, linear learning rate warmup and AdamW optimizer. We pretrain DKGA and WSG for 1k steps and Phonetics for 3k steps. We are able to pretrain CoMixBERT with Phonetics because it has 7x less
parameters than CoMixBART.

heading: 5.3.2 Downstream Fine-tuning
text: CoMixBART. The pretrained model is fine-tuned for downstream tasks in two stages. First, we attach a custom task-specific head to the decoder and train its weights, CoMixBART’s NPW (encoder and decoder) for 5k steps on three 24GB GPUs with batch size of 2048 tokens, linear learning rate warmup, and decay with 2k warmup steps and max. learning rate of 5e-4 using Adam optimizer. In the second stage, phonetic weights of CoMixTransformer encoder are initialized as per section 5.3.1. Then, in downstream training of complete model, the weights from previous step are optimised with smaller learning rate than CoMixBART encoder’s phonetic weights for additional 5k steps. We use beam search (size 4) for decoding. We train baseline IndicBART model for all tasks using YANMTT (Dabre, 2022), as prescribed in IndicBART repository (IndicBART, 2022) with the same setup as CoMix models. In all cases, we pick the model with best validation score after 5k training steps. CoMixBERT. We attach a custom task-specific head to the model and train using standard finetuning procedure. For NER, we also have CRF layer attached after all models including baseline. Since its possible to combine encoder only models without sequential training, we report ensemble results obtained by averaging logits for DKGA+WSG and DKGA+WSG+Phonetic variants as they were better than sequential training. We use grid search to find the right set of hyperparameters for all models including baseline and pick the model with best validation score. We custom-build CoMixBART and CoMixBERT implementation using transformers (Wolf et al., 2020), YANMTT (Dabre, 2022), and PyTorch (Paszke et al., 2019).

heading: 6 Results and Analysis

heading: 6.1 Machine Translation
text: Table 5 shows the results for the LINCE Leaderboard English-Hinglish translation task. CoMix
5We show punctuation-less metrics for Echo baseline for HooD in brackets to correct for the inconsistent punctuation.
achieves the new SOTA result with 12.98 BLEU points beating previous best mT5 based model that is more than double the size. Validation set scores in Table 4 show that CoMix beats IndicBART by over 2 BLEU and 1.7 PhoBLEU points besides yielding faster convergence (see Appendix A.4.3). To test the generalization capabilities, we also evaluate the above models on out-of-domain HooD data. Table 3 shows the results with CoMix improving over IndicBART on both the HooD datasets in terms of both BLEU and PhoBLEU metrics, pointing towards better generalization capabilities of CoMix. HooD Open-Domain dataset has higher Code-Mixing Index (CMI) (shown in section 5.2) than HooD Shopping dataset, which is where DKGA+WSG model improves over DKGA model owing to its pretraining procedure which encourages code-mixing (see Appendix A.4.7). To reduce overfitting for phonetic weights on the downstream dataset, here we train DKGA+Phonetic model on actual training data and 40k unlabelled English sentences and DKGA model’s predictions. Fig 12 in Appendix A.4.4 compares few sample generated translations from IndicBART and CoMix.

heading: 6.2 Named Entity Recognition
text: Table 6 shows weighted F1-score from LINCE leaderboard for Hinglish NER task. All the CoMixBERT components and their combinations beat baseline IndicBERT by 0.77-2.42 points and
SOTA XLM-R large model, which has 10x more parameters, by 0.72-2.37 points. Since combinations yield upto 1.65 points boost, it is likely they capture different facets of code-mixing.

heading: 6.3 Sequence Classification
text: Table 6 shows micro-F1 score for Hinglish sentiment classification task where individual CoMix components beat IndicBERT model by 0.36-1.86 points and DKGA+WSG+Phonetics model beats it by over 3 points. Similar to the mBERT training in (Patwa et al., 2020), we train our model in a minimalistic fashion without any data-augmentation or weighted adversarial loss or token ids that can improve performance. Hence, we do not compare our results against other solutions in Semeval-2020 task and only compare against mBERT and IndicBERT.

heading: 6.4 Abstractive Summarization
text: On Gupshup H2H summarization dataset, CoMix beats IndicBART on all metrics (BLEU, PhoBLEU, R1, R2 and RL) by margin of 0.8 to 2 points as shown in Table 7. CoMix even beats previously published best BLEU results obtained from PEGASUS model (Zhang et al., 2019) but is worse on R1 and R2 metrics. CoMix is worse on recallbased metrics (R1, R2) and better on precision based metrics (BLEU) than PEGASUS and BART likely because of their ability to recall Englishbased words in the Hinglish summaries as they were pretrained only on English and the Gupshup dataset has been adapted from English conversa-
tional summarization corpus due to which it contains a lot of English named entities and words. We believe CoMixBART performance can further improve if we do pretraining with phonetics in future.

heading: 6.5 Qualitative Analysis
text: We examine how well the models in Section 6.1 separate 3655 pairs of words (178 similar, 3477 dissimilar) from 20 sentences in Appendix A.4.6. Figure 5 shows the distribution of cosine similarity of contextual embeddings (phonetic and textual for CoMix, textual for IndicBART) for similar (green) and dissimilar (red) pairs . We note that CoMix text embeddings separate the similar and dissimilar pairs better relative to IndicBART. Note that the scores for phonetic embeddings are on the higher side most likely due to initialisation choice (mean of all text tokens mapped to a phonetic token) and the smaller (0.25x of text) vocab size for phonetics.

heading: 6.6 Efficacy of PhoBLEU on code-mixed data
text: On English-Hinglish translation for the LINCE dataset, we observe that annotations from human experts fluent in both Hindi and English achieve a BLEU score of 10.43 BLEU, which is lower than most MT models. Further analysis revealed that BLEU is unable to account for valid variations in
spellings, pronouns, and language switching (LE vs. LM ) as shown in Fig 1. To address these gaps, we consider PhoBLEU [as defined in Section 4] and evaluate its correlation with human judgements. We randomly selected 200 English-Hinglish sentence pairs and their system-generated translations to be rated by professional on a scale of 1 to 5, with 1 as poor and 5 as perfect. Completeness (no information lost) and fluency (grammatical correctness) were considered as the rating criteria. Results in Table 8 and Figure 6 show that PhoBLEU is significantly more correlated with human judgement and that its distribution is better aligned with human ratings than other BLEU variants.

heading: 7 Extensibility of CoMix
text: The proposed ideas of domain-knowledge guided attention, weak supervision, and phonetic representations are not specific to Hinglish and readily generalize to any language pair where we have a parallel corpus of embedded and matrix language content and tools for POS tagging and phonetic transcription. Below we discuss these requirements
along with other assumptions on the POS structure that permits extensions of our methodology to most common code-mixed language pairs.
Assumption 1: Availability of parallel corpora. Most common code-mixed languages happen to include English among the language pair which is typically transcribed in Latin script that permits easy phonetic transcription through tools such as Pyphonetics. Currently, there also exist multiple large parallel corpora (e.g. Flores, CCMatrix, Samanantar) where sentences in English are paired with that of multiple other languages. There are also many ongoing initiatives for creating such parallel corpora even for low-resource languages. Hence, the requirement of a large parallel corpus of matrix and embedded language content is satisfied by most common code-mixed pairs.
Assumption 2: Availability of pretrained multilingual models. With the proliferation of massively multilingual foundational models (e.g., mBART (50 languages), mBERT (104 languages), T5 (101 languages)) including advances in synthetic data augmentation, our assumption on the availability of pretrained LLMs or datasets to pretrain those models is also a reasonable one. We choose to work with IndicBART and IndicBERT which support 11 and 12 Indic languages respectively because they provide stronger baselines for Indic Languages and are faster to experiment with because of their smaller size, but the proposed ideas can be readily applied with any pre-trained transformer model.
Assumption 3: Languages of LM and LE share the same POS set and access to POS tagging utilities. (Petrov et al., 2012) proposed Universal POS tagset comprising 12 categories that exist across languages and had developed a mapping from 25 language-specific tagsets to this universal set. They demonstrated empirically that the universal POS categories generalize well across language boundaries and led to a open community initiative by universaldependencies.org on creating Universal POS tags (Nivre et al., 2020) for 90 languages. In our work, we use these universal POS tags to build three coarse groups (nouns-pronouns, adjectivesverbs-adverbs, rest) of POS tags (see Fig 7). Note that even though we utilize POS-tagging, the structural constraints are imposed with respect to these three coarse groups. Fig 7 in A.1 lists the POS tags from the universal POS tags website, which we use in our work. Further, Stanza provides Universal POS-tagging utilities for around 66 languages.
Assumption 4: Equivalent word pair from LM and LE share the same coarse POS group. We assume that equivalent words in an LM and LE pair share the same coarse POS group (from Fig 7 ) and not necessarily the same POS tag. A small-scale empirical analysis of 50 Hindi-English-Hinglish sentences from the HooD dataset (Sec 5.2) indicates this assumption is true in 88.6% of the cases. POS-tags provide complementary (weak) supervision for intermixing (in DKGA) and generation (in WSG) in addition to word semantics already captured in embeddings. Further, even though our current guiding function fDKGA assumes a hard constraint on the word pairs to be in the same coarse POS group, our methodology is general and can be extended to the case where the two languages have different POS tag sets. In particular, given empirical probabilities that a matrix token with POS tag A maps to an embedded token with POS tag B for all possible pairs of POS tags (A,B), we can define the guiding function’s value fDKGAij associated with matrix token xi and embedded token xj as the log of the empirical transition probability of the POS tags of the matrix token xi and embedded token xj . The current choice is the special case where transition probability is uniform for all POS tag pairs within a coarse group and 0 for the rest.

heading: 8 Conclusion
text: We presented CoMix, a pretraining approach for code-mixed data that combines (a) domain knowledge guided attention (DKGA), (b) weakly supervised code-mixed generation based on POSstructure constraints, and (c) transformer encoder modification to include phonetic signal. We showed that CoMix yields improvements across multiple code-mixed tasks, achieving new SOTA result for Eng-Hinglish translation and Hinglish NER on LINCE Leaderboard with superior performance on out-of-domain translation. Our approach is applicable to code-mixing with all languages where POS tagging and phonetic transcription is possible. Motivated by gaps in current NLG evaluation metrics for code-mixed data, we proposed a new family of metrics based on phonetic representation and show that PhoBLEU is better correlated with human judgement than BLEU on Hinglish. In future, we plan to extend the applicability of DKGA and WSG to other settings that can benefit from domain knowledge, and explore new metrics for code-mixed NLG with a large scale evaluation.
Limitations
Our CoMix approach assumes availability of parallel bilingual (embedded and matrix language) corpora and mature tools for POS tagging and phonetic transcription for both the embedded and matrix languages which does not hold true for every language. But these assumptions are reasonable for a large number of languages as shown in Appendix 7. Second, our current choice of guiding function for attention fDKGA and mixing probability pMix are based on limited knowledge of the linguistic structure specific to English and Indic languages, and might need to be adapted for other language families. Additionally, as discussed in Section 4, due to multiple variations in code-mixed generation, current automated metrics that compare system generated text with reference text do not provide a true reflection of a system’s ability to generate code-mixed text. Lastly, as with large language models, our CoMix models are also vulnerable to biases inherent in the training corpus.
Ethics Statement
Our research motivation is to address the inequities in language resources and AI systems for multilingual societies such as India. The primary contribution of our work is a new modeling approach CoMix, which is especially designed to leverage existing pretrained models with moderate computation so that it is accessible to a wider community and does not create an adverse environmental impact. We also created two new Hinglish datasets for out-of-domain evaluation (HooD), which we described in detail in Section 5.2. There are no privacy or intellectual property rights associated with either of these datasets. We will open-source HooD, our models and code in future post organizational approval. Human translations and evaluations reported in the paper have been done by professional annotation teams and are reflective of typical performance. Similar to other large language models, our CoMix model also encodes biases in the original training corpus and the domain constraints used as supervision. While the performance might be acceptable for natural language understanding, it is important to have guardrails while using the models directly for natural language generation.

heading: A Appendix
text: A.1 Classes of POS tags
Figure 7 shows the POS tag groups used by DKGA. We built these groups using information from Universal Dependencies (Dependencies, 2014).
A.2 Formulations of Downstream tasks
A.2.1 Machine Translation Given a source sequence X = [x1, x2, ...xS ] and target sequence Y = [y1, y2, ...yT ], autoregressive neural machine translation system learns to model the following distribution
P (Y |X) = T∏
t=1
P (yt|y0, y1, ..., yt−1, x1, x2, ..., xS)
(4) Given a training set D = {⟨X(i), Y (i)⟩}Mi=0 with M data points, we aim to maximize Lθ =∑M i=0 logP (Y
(i)|X(i); θ) where θ is set of model parameters.
A.2.2 Sequence Classification In sequence classification task, we are given a sequence X = [x1, x2, ...xS ] and corresponding label y ∈ {y1, y2, ..., yk} from fixed set of k classes. Given a training set D = {⟨X(i), y(i)⟩}Mi=0 with M data points, we aim to maximize Lθ =∑M
i=0 logP (y (i)|X(i); θ)
A.2.3 Abstractive Summarization Mathematical formulation for summarization is same as translation so we avoid repeating it here for brevity. In abstractive summarization, unlike translation, target sequence Y is a concise summary of source sequence X , usually much shorter in length than X .
A.2.4 Token Classification In token classification task, we are given a sequence X = [x1, x2, ...xS ] and corresponding label ys ∈ {y1, y2, ..., yk}∀s ∈ {1, S} for every input token, where {y1, y2, ..., yk} is the fixed set of k classes. Given a training set D = {⟨X(i), Y (i)⟩}Mi=0 with M data points, we aim to maximize Lθ = ∑M i=0 logP (Y (i)|X(i); θ)
A.3 More details about datasets
A.3.1 Guidelines for preparing HooD Shopping dataset
Figure 8 shows the guidelines given to human annotators for translating English sentences to Hinglish for HooD Shopping dataset.
A.3.2 Data statistics of public datasets Table 9 shows statistics for public datasets which we have used for downstream tasks.
A.4 Additional experimental details and results
A.4.1 Details about tokenization For Phonetics data, we train our own sub-word tokenizer using sentencepiece6. For text data we use pretrained IndicBART’s tokenizer for CoMixBART and IndicBERT’s tokenizer for CoMixBERT. We consider sub-words POS to be same as the POS of the word from which sub-words have been created.
A.4.2 More details on pretraining with DKGA and WSG
Figure 9 shows the learning curve for pretraining CoMixBART. As you can see from the curve, loss stabilizes after 25k steps and does not change much. Figure 10 shows the learning curve for pretraining ComixBART with WSG.
Table 10 shows few sample inputs which went into the model during WSG training and corresponding targets constructed by the model. These generated sentences can be used for data-
6https://github.com/google/sentencepiece
augmentation which we plan to explore in the future.
A.4.3 Convergence for IndicBART vs CoMix on LINCE Leaderboard translation task
Figure 11 shows the convergence speed for IndicBART and CoMix models for LINCE EnglishHinglish translation task. As you can see from the curve, CoMix is better than baseline IndicBART throughout training.
A.4.4 Qualitative Analysis on few sample predictions for Code-Mixed Translation
Figure 12 shows few example translations generated by IndicBART and CoMix and their qualitative analysis.
A.4.5 Set of sentences for cosine similarity distribution
Figure 13 shows the 20 sentences from which every pair of word was labelled similar/dissimilar manually and then used to create Figure 5 that shows cosine similarity score distribution of contextual embeddings obtained from the encoder of CoMix and IndicBART.
A.4.6 CoMix vs IndicBART Cosine Similarity Distribution of Contextual Embeddings
Figure 11 shows the mean and variance of the cosine similarity distribution of 3655 word pairs constructed from the 20 sentences in Figure 13 along different subsets of positive pairs and close negative pairs. We observe that the cosine score for positive pairs based on CoMix text embeddings has a bimodal distribution with high scores for those with same language and spelling, but relative low scores when that is not the case. However, even these low scoring positive pairs are comparable or score higher than close negatives. In the case of IndicBART, we again observe a bimodal distribution for the negative pairs with high scores for pairs that have different semantics but share either the spelling or phonetic representation, which makes it difficult to separate it from the positive pairs. CoMix phonetic embeddings by itself does not seem to be very discriminatory but it is helpful
in making up for the shortcomings of CoMix text embeddings for handling phonetic variations.
A.4.7 DKGA vs WSG. Who’s code-mixing more?
Since in WSG we’re nudging the model to codemix, that behaviour is visible in the generated translations by the two models as well. Figure 14 shows few randomly sampled translations generated by DKGA and DKGA+WSG models. It is visible from the translations that DKGA+WSG model is switching between matrix and embedded language more often, because of its pretraining.
A.4.8 NER and Classification Results Table 12 shows results on validation and test sets for NER and Sequence Classification tasks.
A.4.9 Example DKGA attention matrix Fig 15 shows DKGA attention matrix for an example sentence.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: A1. Did you describe the limitations of your work?
Left blank.
A2. Did you discuss any potential risks of your work? Left blank.
A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
A4. Have you used AI writing assistants when working on this paper? Left blank.

heading: B Did you use or create scientific artifacts?
text: Left blank.
B1. Did you cite the creators of artifacts you used? Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.

heading: C Did you run computational experiments?
text: Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Left blank.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.

heading: D Did you use human annotators (e.g., crowdworkers) or research with human participants?
text: Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Left blank.

heading: Future Work
text: In future, we plan to extend the applicability of DKGA and WSG to other settings that can benefit from domain knowledge, and explore new metrics for code-mixed NLG with a large scale evaluation. Limitations
Our CoMix approach assumes availability of parallel bilingual (embedded and matrix language) corpora and mature tools for POS tagging and phonetic transcription for both the embedded and matrix languages which does not hold true for every language. But these assumptions are reasonable for a large number of languages as shown in Appendix 7. Second, our current choice of guiding function for attention fDKGA and mixing probability pMix are based on limited knowledge of the linguistic structure specific to English and Indic languages, and might need to be adapted for other language families. Additionally, as discussed in Section 4, due to multiple variations in code-mixed generation, current automated metrics that compare system generated text with reference text do not provide a true reflection of a system’s ability to generate code-mixed text. Lastly, as with large language models, our CoMix models are also vulnerable to biases inherent in the training corpus