Venue: NAACL 2021

# Code-Mixing on Sesame Street: Dawn of the Adversarial Polyglots

Samson Tan\({}^{\lx@sectionsign}\)   Shafiq Joty\({}^{\lx@sectionsign}\)

\({}^{\lx@sectionsign}\)Salesforce Research Asia \({}^{\natural}\)National University of Singapore \({}^{\ddagger}\)Nanyang Technological University {samson.tan,sjoty}@salesforce.com

###### Abstract

Multilingual models have demonstrated impressive cross-lingual transfer performance. However, test sets like XNLI are monolingual at the example level. In multilingual communities, it is common for polyglots to code-mix when conversing with each other. Inspired by this phenomenon, we present two strong blackbox adversarial attacks (one word-level, one phrase-level) for multilingual models that push their ability to handle code-mixed sentences to the limit. The former uses bilingual dictionaries to propose perturbations and translations of the clean example for sense disambiguation. The latter directly aligns the clean example with its translations before extracting phrases as perturbations. Our phrase-level attack has a success rate of 89.75% against XLM-Rlarge, bringing its average accuracy of 79.85 down to 8.18 on XNLI. Finally, we propose an efficient adversarial training scheme that trains in the same number of steps as the original model and show that it improves model accuracy.1

Footnote 1: Code: github.com/salesforce/adversarial-polyplots

## 1 Introduction

The past year has seen incredible breakthroughs in cross-lingual generalization with the advent of massive multilingual models that aim to learn universal language representations Pires et al. (2019); Wu and Dredze (2019); Conneau et al. (2020). These models have demonstrated impressive cross-lingual transfer abilities: simply fine-tuning them on task data from a high resource language such as English after pretraining on monolingual corpora was sufficient to manifest such abilities. This was observed even for languages with different scripts and no vocabulary overlap K et al. (2020).

However, transferring from one language to another is insufficient for NLP systems to understand multilingual speakers in an increasingly multilingual world Aronin and Singleton (2008). In many multilingual societies (e.g., Singapore, Papua New Guinea, etc.), it is common for multilingual interlocutors to produce sentences by mixing words, phrases, and even grammatical structures from the languages in their repertoires Matras and Sakel (2007). This is known as _code-mixing_Poplack et al. (1988), a phenomenon common in casual conversational environments such as social media and text messages.2 Hence, it is crucial for NLP systems serving multilingual communities to be robust to code-mixing if they are to understand and establish rapport with their users Tay (1989); Bawa et al. (2020) or defend against adversarial polysldots.

Footnote 2: Examples of real code-mixing in Appendix A.

Although gold standard data Bali et al. (2014); Patwa et al. (2020) is important for definitively evaluating code-mixed text processing ability, such datasets are expensive to collect and annotate. The

Figure 1: Bumblebee’s three key stages of adversary generation: (a) Align words in the matrix (English) and embedded sentences (top: Indonesian, bottom: Chinese); (b) Extract candidate perturbations from embedded sentences; (c) Construct final adversary by maximizing the target model’s loss.



dizzying range of potential language combinations further compounds the immensity of such an effort.

We posit that performance on appropriately crafted adversaries could act as a lower bound of a model's ability to generalize to the distribution simulated by said adversaries, an idea akin to _worstcase analysis_(Divekar, 1984). For example, Tan et al. (2020b) showed that an NLP system that was robust to morphological adversaries was less perplexed by dialectal text exhibiting morphological variation. Likewise, if a system is robust to code-mixed adversaries constructed from some set of languages, it is reasonable to expect it to also perform better on real code-mixed text in those languages. While they may not fully model the intricacies of real code-mixing (Sridhar and Sridhar, 1980), we believe that they can be useful in the absence of appropriate evaluation data. Hence, we:

* Propose two strong black-box adversarial attacks targeting the cross-lingual generalization ability of massive multilingual representations (Fig. 1), demonstrating their effectiveness on state-of-the-art models for natural language inference and question answering. To our knowledge, these are the first two multilingual adversarial attacks.
* Propose an efficient adversarial training scheme that takes the same number of steps as standard supervised training and show that it creates more language-invariant representations, improving accuracy in the absence of lexical overlap.

## 2 Related Work

Multilingual classifiers.Low resource languages often lack support due to the high cost of annotating data for supervised learning. An approach to tackle this challenge is to build cross-lingual representations that only need to be trained on task data from a high resource language to perform well on another under-resourced language Klementiev et al. (2012). Artetxe and Schwenk (2019) presented the first general purpose multilingual representation using a BiLSTM encoder. Following the success of Transformer models Vaswani et al. (2017), recent multilingual models like mBERT Devlin et al. (2019), Unicoder Huang et al. (2019), and XLM-R Conneau et al. (2020a) take the pretraining\(\rightarrow\)fine-tuning paradigm into the multilingual realm by pretraining Transformer encoders on unlabeled monolingual corpora with various language modeling objectives before fine-tuning them on task data from a high-resource language such as English. This is known as cross-lingual transfer.

Code-mixed text processing.Previous research on code-mixed text processing focused on constructing formal grammars Joshi (1982) and token-level language identification Bali et al. (2014); Solorio et al. (2014); Barman et al. (2014), before progressing to named entity recognition and part-of-speech tagging Ball and Garrette (2018); AlGhamdi and Diab (2019); Aguilar and Solorio (2020). Recent work explores code-mixing in higher-level tasks such as question answering and task-oriented dialogue Chandu et al. (2019); Ahn et al. (2020). Muller et al. (2020) demonstrate mBERT's ability to transfer to an unseen dialect by exploiting its speakers' tendency to code-mix.

A key challenge of developing models that are robust to code-mixing is the availability of code-mixed datasets. Hence, Winata et al. (2019) use a pointer-generator network to generate synthetically code-mixed sentences while Pratapa et al. (2018) explore the use of parse trees for the same purpose.

Yang et al. (2020) propose to improve machine translation with "code-switching pretraining", replacing words with their translations in a similar manner to masked language modeling Devlin et al. (2019). These word pairs are constructed from monolingual corpora using cosine similarity. sitaram et al. (2019) provide a comprehensive survey of code-mixed language processing.

Word-level adversaries.Modified inputs aimed at disrupting a model's predictions are known as adversarial examples Szegedy et al. (2014). In NLP, perturbations can be applied at the character, subword, word, phrase, or sentence levels.

Early word-level adversarial attacks Ebrahimi et al. (2018); Blohm et al. (2018) made use of the target model's gradients to flip individual words to trick the model into making the wrong prediction. However, while the perturbations were adversarial for the target model, perturbed word's original semantics was often not preserved. This could result in the expected prediction changing and making the model appear more brittle than it actually is.

Later research addressed this by searching for adversarial rules Ribeiro et al. (2018) or by constraining the candidate perturbations to the \(k\) nearest neighbors in the embedding space Alzantot et al. (2018); Michel et al. (2019); Ren et al. (2019); Zhang et al. (2019); Li et al. (2019); Jin et al. (2020). Zang 

et al. (2020) take another approach by making use of a annotated sememes to disambiguate polysemous words, while Tan et al. (2020) perturb only the words' morphology and encourage semantic preservation via a part-of-speech constraint. Other approaches make use of language models to generate candidate perturbations Garg and Ramakrishnan (2020); Han et al. (2020). Wallace et al. (2019) find phrases that act as universally adversarial perturbations when prepended to clean inputs. Zhang et al. (2020) provide a comprehensive survey.

Summary.Existing work on pretrained multilingual models has highlighted their impressive zero-shot cross-lingual transfer ability, though some analyses K et al. (2020) indicate this could be a result of exploiting lexical overlaps rather than an indication of true cross-lingual understanding. Although language-agnosticity is commonly measured via cross-lingual retrieval tasks such as LAReQA Roy et al. (2020) and similarity search Artetxe and Schwenk (2019), we offer a different perspective in this paper by operationalizing it as a model's ability to handle code-mixing. Existing evaluations for code-mixed text processing focus on gold annotated data, but such datasets are (relatively) expensive to compile and face similar scarcity challenges as those for low-resource languages. Existing word-/phrase-level adversarial attacks probing the limits of model robustness have largely focused on monolingual (English) inputs. In contrast, our adversarial attacks are designed to test the robustness of multilingual models to adversarial code-mixers. Finally, we propose an efficient adversarial training scheme to improve the robustness of said models to code-mixed adversaries.

## 3 Generating Multilingual Adversaries

Code-mixing is a phenomenon where a multilingual speaker mixes words, and even grammatical rules, from different languages in a single sentence. This is distinguished from code-switching, which occurs at the inter-sentential level Kachru (1978).

Extreme code-mixing.Inspired by the proliferation of real-life code-mixing and polyglots, we propose PolyGloss and Bumblebee, two multilingual adversarial attacks that adopt the persona of an adversarial code-mixer. We focus on the lexical component of code-mixing, where some words in a sentence are substituted with their equivalents from another language in the interlocutor's repertoire. Borrowed words fall into two categories, nonce borrowing and loanwords, though distinguishing between them is beyond the scope of this work.

Since most code-mixers are bilinguals, natural code-mixed sentences tend to be constructed from two languages, with one language determining the syntax of the overall sentence Poplack et al. (1988). However, in a world with an increasing number of multilingual societies, it is conceivable for code-mixing to occur between more than two languages Tan (1988). We take this idea to the extreme to test multilingual representations for their robustness to such cross-lingual lexical variation.

Problem formulation.Given a target multilingual model \(\mathcal{M}\), a clean example \(x\) with the label \(y\), and a set of embedded languages \(\mathbb{L}\) from which to borrow words, we aim to generate the adversarial example \(x^{\prime}\) that maximizes \(\mathcal{M}\)'s loss. Formally,

\[x^{\prime}=\operatorname*{arg\,max}_{x_{c}\in X}\mathcal{L}(y,\mathcal{M}(x_{ c})), \tag{1}\]

where \(x_{c}\in X\) is a candidate adversary generated by perturbing \(x\), \(\mathcal{M}\) is a task-specific neural model, and \(\mathcal{L}(\cdot)\) is the model's loss function.

### PolyGloss: Word-Level Adversaries

To obtain a code-mixed adversary, we first generate candidate adversaries by substituting words in the

\begin{table}
\begin{tabular}{p{45.5pt} p{260.2pt}} \hline \hline Original & **P:** The girl that can help me is all the way across town. **H:** There is no one who can help me. \\ Adversary & **P:** olan girl that can help me is all the way across town. **H:** one who can help me. \\ Prediction & **Before:** Contradiction **After:** Entailment \\ \hline Original & **P:** We didn’t know where they were going. **H:** We didn’t know where the people were traveling to. \\ Adversary & **P:** We didn’t know where they were going. **H:** We didn’t know where les gens allaient. \\ Prediction & **Before:** Entailment **After:** Neutral \\ \hline Original & **P:** Well it got to where there’s two or three aircraft arrive in a week and I didn’t know where they’re flying to. \\  & **H:** There are never any aircraft arriving. \\ Adversary & **P:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Fernández:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez: Gómez:** Gómez:** Gómez:** Gómez:** Gómez:** Gómez: Gómez:** Gómez:** Gómez:** Gómez:** Gómez: Gómez:** Gómez:** Gómez: Gómez:** Gómez:** Gómez:** Gómez: Gómez:** Gómez:** Gómez:** Gómez: Gómez:** Gómez:** Gómez: Gómez:** Gómez:** Gómez: Gómez:** Gómez:** Gómez: Gómez:** Gómez: Gómez:** Gómez:** Gómez: Gómez:** Gómez: Gómez:** Gómez: Gómez:** Gómez:** Gómez: Gómez:** Gómez: Gómez:** Gómez:** Gómez: Gómez:** Gómez: Gómez:** Gómez: Gómez:** Gómez: Gómez:** Gómez: Gómez:** Gómez: Gómez:** Gómez: Gómez:** Gómez: Gómez:** Gómez: Gómez:** Gómez: Gómez:** Gómez: Gómez:** Gómez: Gómez:** Gómez: Gómez:** Gómez: Gómez:** Gómez: Gómez:** Gómez: Gómez:** Gómez: Gómez:** Gómez: Gómez:** Gómez: Gómez: Gómez:** Gómez: Gómez:** Gómez: Gómez:** Gómez: Gómez: Gómez:** Gómez: Gómez:** Gómez: Gómez: Gómez:** Gómez: Gómez:** Gómez: Gómez: Gómez:** Gómez: Gómez: Gómez:** Gómez: Gómez: Gómez:** Gómez: Gómez: Gómez:** Gómez: Gómez: Gómez:** Gómez: Gómez: Gómez: Gómez:** Gómez: Gómez: Gómez:** Gómez: Gómez: Gómez:** Gómez: Gómez: Gómez: Gómez:** Gómez: Gómez: Gómez:** Gómez: Gómez: Gómez: Gómez: Gómez:** Gómez: Gómez: Gómez: Gómez: Gómez:** Gómez: Gómez: Gómez: Gómez: Gómez: Gómez:** Gómez: Gómez: Gómez: Gómez: Gómez: Gómez: Gómez: Gómez:** Gómez 

clean example with their equivalents from another language. These substitutions/perturbations can be generated by via machine translation or mined from bilingual dictionaries. Following Myers-Scotton (1997), we will refer to the original example's language as the matrix language and the perturbation's language as the embedded language.

Next, we perform beam search on the candidates to find the adversary that maximizes the target model's loss in a black-box manner (Alg. 2 in Appendix B.1). In our implementation, we also keep track of successful adversaries and return the ones with the highest and lowest losses. The former is a stronger adversary, while the latter often has fewer perturbations. More details are in Appendix B.1.

Orthographic preservation.When the embedded language uses a different script from the matrix language, code-mixers tend to transliterate borrowed words into the same script Abuhakema (2013); Bali et al. (2014). This still poses a significant challenge to multilingual models Khanuja et al. (2020). We generally preserve the embedded language's script where possible to avoid unfairly penalizing the target model since there is often no standard way of transliterating words.

Scalable sense disambiguation.Due to the polysemous nature of many words, translating the right sense is crucial to preserving the word's (and sentence's) semantics. Common word sense disambiguation methods Agirre and Edmonds (2007) use a sense tagger trained on an annotated sense inventory such as WordNet Miller (1995). However, this approach requires individual taggers and sense inventories for each matrix and embedded language, making it a serious challenge to extend PolyGloss to low-resource languages.

Instead, we propose to filter candidate perturbations using the embedded language translation of the clean example. This is easily done by checking if the candidate perturbation exists in the translation. Since our examples tend to be single sentences, the probability of different senses of the same word occurring in a single sentence is generally low Conneau et al. (2018); Popel et al. (2020). This approach only requires a machine translation (MT) system and no extra linguistic information, making it highly scalable as long as a supervised (or unsupervised) machine translation system is available. By using gold translations instead of machine translations, it is even possible to mostly guarantee semantic preservation at the word-level.

### Bumblebee: Phrase-Level Adversaries

Although using bilingual dictionaries with our filtering method ensures that the semantics of a borrowed word matches the original, the dictionary's comprehensiveness determines the presence of sufficient candidate adversaries. In addition, PolyGloss swaps words at the word level, which may hurt the naturalness of the resulting sentence since it is more common for code-mixers to borrow phrases than individual words Abuhakema (2013).

A solution to these issues is to replace phrases in the matrix sentence with their equivalents from the reference translations instead of using a dictionary lookup (Alg. 1). A key advantage of this approach is its flexibility and scalability to more languages since it only requires parallel bitexts from the matrix and embedded languages. With the advent of neural sequence-to-sequence models, such bitexts can be easily generated using publicly available MT models. However, a key challenge for this approach is extracting the matrix-embedded phrase pairs from the clean example and its translation. We follow common phrase-based machine translation methods and accomplish this by aligning the matrix and embedded sentences Koehn (2010). Implementation details can be found in Appendix B.2.

Syntactic preservation.To improve the adversaries' naturalness, we impose an _equivalence constraint_Poplack (1980), preventing a perturbation from being applied if it is from the same language as the previous word _and_ will disrupt the syntax of the current phrase if applied Winata et al. (2019). Such disruptions usually occur when borrowing words from languages with a different word order.

 



## 4 Experiments

We first evaluate PolyGloss and Bumblebee on XNLI Conneau et al. (2018), then evaluate the stronger attack on XQuAD Artetxe et al. (2020). XNLI is a multilingual dataset for natural language inference (NLI) with parallel translations for each example in fifteen languages. Each example comprises a premise, hypothesis, and a label with three possible classes: {contradiction, neutral, entailment}. We construct two more datasets from XNLI: XNLI-13 and XNLI-32. XNLI-13 comprises all XNLI languages except Swahili and Urdu due to the lack of suitable dictionaries for PolyGloss. We then translate the English test set into eighteen other languages with MT systems to form XNLI-31, increasing the number of embedded languages PolyGloss can draw from. XQuAD is a multilingual dataset for extractive question answering (QA) with parallel translations in eleven languages. In the cross-lingual transfer setting, the models are trained on English data, MNLI Williams et al. (2018) and SQuAD 1.1 Rajpurkar et al. (2016), and tested on mulitlingual data, XNLI and XQuAD, respectively. We perturb the premise and hypothesis for NLI and only the question for QA. More experimental details can be found in Appendix D.

Matrix language.Although our attacks work with any language as the matrix language, we use English as the matrix language in our experiments due to the availability of English\(\rightarrow\)T translation models and the prevalence of English as the matrix language in many code-mixing societies.

Models.We conduct our experiments on three state-of-the-art massive multilingual encoder models: XLM-RoBERTa, mBERT, and Unicoder, each pretrained on more than 100 languages.

### Results

From Tables 2 and 3, we observe that all the models are significantly challenged by adversarial codecel mixing, though XLM-Rlarge is the most robust to both attacks, likely due to having more parameters. However, even after filtering PolyGloss's candidate perturbations by the gold translations in XNLI-13, we observe an average drop in accuracy of 80.01%, relative to the models' accuracy on the clean XNLI-13. Bumblebee induces even greater performance drops (average relative decrease of 90.96% on XNLI-13), likely due to its word aligner yielding more candidates than PolyGloss's dictionary lookup. Increasing the number of embedded languages PolyGloss can draw upon results in greater drops in model performance (average relative decrease in accuracy of 93.66% on XNLI-31).

BERT- vs. XLM-based.We notice that mBERT is more sensitive to intra-phrasal syntactic disruption than the XLM-based models. mBERT is the most robust to Bumblebee out of all the base models when the equivalence constraint is in place, yet is the least robust to PolyGloss. However, the latter trend is replicated for Bumblebee if we remove this constraint (Table 16 in Appendix G). A _possible_ explanation is that XLM-R and Unicoder were trained on monolingual CommonCrawl (CC) data, while mBERT was trained on multilingual Wikipedia, which could be considered as aligned at the article level since there are articles on the same topic in different languages. Hence, it is possible that this helped to align the languages more accurately in the feature space but made it more sensitive to syntactic disruptions. However, many other hyperparameters differ between the two that could have also influenced their robustness. Hence, we leave a rigorous study of these factors to future

\begin{table}
\begin{tabular}{l|c c} \hline \hline
**Model** & **Clean** & **Bumblebee** \\ \hline XLM-Rlarge & 75.64 / 61.39 & **35.32 / 22.52** \\ XLM-Rbase & 68.90 / 53.50 & **17.95 / 10.33** \\ mBERTbase & 64.66 / 49.47 & **20.66 / 11.68** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Bumblebee results on XQuAD (F\({}_{1}\)/EM).

\begin{table}
\begin{tabular}{l|c c c|c c} \hline \hline
**Model** & \multicolumn{3}{c}{**XNLI-13**} & \multicolumn{3}{c}{**XNLI-31**} \\  & Clean & PG\({}_{\text{off}}\) & PG\({}_{\text{intL}}\) & Clean & PG\({}_{\text{infL}}\) \\ \hline XLM-Rlarge & 81.10 & **6.06** & 28.28 & 80.60 & **8.76** \\ XLM-Rbase & 75.42 & **2.17** & 12.27 & 74.75 & **3.57** \\ mBERTbase & 67.54 & **2.15** & 9.24 & 66.56 & **3.11** \\ Unicoderbase & 74.98 & **1.99** & 11.33 & 74.28 & **3.73** \\ \hline \hline \end{tabular}
\end{table}
Table 2: PolyGloss (PG) results (accuracy) on XNLI-13 and -31 test sets with beam width = 1. PG\({}_{\text{fit.}}\), \({}_{\text{tf.}}\) indicates whether the candidate perturbations were filtered using reference translations. Clean accuracy scores are the averages across all languages in the test set. Lower is better.

\begin{table}
\begin{tabular}{l|r r|r r r} \hline \hline
**Model** & \multicolumn{3}{c}{**XNLI-13**} & \multicolumn{3}{c}{**Standard XNLI**} \\  & Clean & B & Clean & Rand. & B \\ \hline XLM-Rlarge & 81.10 & **11.31** & 79.85 & 75.04 & **8.18** \\ XLM-Rbase & 75.42 & **5.08** & 74.06 & 65.19 & **3.53** \\ mBERTestbase & 67.54 & **6.10** & 65.66 & 59.17 & **4.45** \\ Unicoderbase & 74.98 & **4.81** & 73.69 & 65.55 & **3.61** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Bumblebee (Bb) results on XNLI with beam width = 1. _Clean_ accuracies are the averages across all languages in each test set. We include a _random_ (Random.) baseline by randomly (rather than adversarially perturbing sentences and report the average across 5 seeds. Lower is better.



work. The higher performance of the XLM-based models on clean data can likely be attributed to the CC corpus being an order of magnitude larger than multilingual Wikipedia (Lauscher et al., 2020).

Candidate filtering.In the _unfiltered_ setting, it is impossible for PolyGloss to discriminate between valid and invalid senses for a given context. Hence, a potential criticism is that the large difference in PolyGloss's success rate between the filtered and unfiltered settings could be attributed to the inappropriate senses of polysemous words being chosen and disrupting the semantics of the sentence. On the other hand, filtering perturbations with reference translations of the sentence shrinks the space of perturbations to \({\sim}1\) per language. Due to the dictionaries' non-exhaustive nature, not every word in the matrix sentence has an entry in the dictionary to begin with, making this filtering step a significant reduction of the space of candidates.

To determine the likely cause of the accuracy difference between the filtered and unfiltered settings in XNLI-13, we increase the number of languages available to PolyGloss to thirty-one. If the difference between the filtered and unfiltered settings were _not_ due to a lack of sufficient candidates, we should observe only a minor difference between the filtered settings for both XNLI-13 and -31. However, we observe a 69% drop for XLM-Rlarge, indicating that the former accuracy difference is likely due to the reduced number of valid candidates.

Phrase-level adversaries.In addition to generating more fluent sentences (Table 1), extracting the candidate perturbations directly from the translations does away with the need for sense disambiguation and increases the number of perturbations per example since it is not limited to a static dictionary. The increased effectiveness of Bumblebee compared to PolyGLoss (1.13x) is further evidence that a key factor to the success of such adversarial attacks is the availability of sufficient candidates; increasing the dimensionality of the search space increases the probability that an adversarial example for the model exists (Goodfellow et al., 2015). We also include a non-adversarial baseline (Rand.) by sampling candidates from a uniform distribution instead of searching for the worst-case perturbations. Our results in Table 3 indicate that the worst-case performance of multilingual models on code-mixed data may be much lower than the scores reported on human-produced test sets since they were not created in a targeted, adversarial fashion. Experiments on beam width and a proof of concept for fully unsupervised adversaries are in Appendix E.

Transliteration.Since real-life code-mixers often use a single script for the entire sentence, we now test the effect of transliteration on Bumblebee's success rate for the English + Hindi language pair. We accomplish this by transliterating all candidates from Devanagari into Latin using the dictionaries released by Roark et al. (2020). From Table 5, we see that transliteration significantly affects the robustness of all models, even the XLM-based ones which were pretrained on similar data.

XQuAD.We observe that both XLM-R and mBERT are significantly challenged by Bumblebee even though only the question was modified (Table 4). We did not experiment on Unicoder to reduce carbon costs since its performance was almost identical to XLM-Rbase in our XNLI experiments.

PolyGloss or Bumblebee?As expected, inspection of individual adversarial examples revealed that Bumblebee generated more natural sentences than PolyGLoss since the languages used within phrases were more consistent (Table 1). However, incorrect alignments due to the word aligner's probabilistic nature could introduce occasional noise into the adversarial examples. For example, we found "the" (en) to be often aligned with "[f]" (zh) even though the former is an article and the latter a possessive. We observe that the aligner performs better when the sentences have similar word orders (e.g., English-French vs. English-Chinese) and we can expect the adversaries generated in these settings to be more natural. Hence, we recommend PolyGloss when greater preservation of word-level semantics is desired, and Bumblebee when phrase-level perturbations are desired or bilingual dictionaries are unavailable.

Discussion.K et al. (2020) noted significant performance drops in XNLI accuracy for mBERT when the premise and hypothesis were in differ

\begin{table}
\begin{tabular}{l|c|c} \hline \hline
**Model** & **Devanagari** & **Transliterated (Latin)** \\ \hline XLM-Rlarge & 61.35 & **41.97** \\ XLM-Rbase & 48.62 & **30.01** \\ mBERTbase & 37.70 & **23.41** \\ Unicoderbase & 49.34 & **30.00** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Bumblebee results on XNLIenhi using both Devanagari and Latin scripts. Lower is better.

 

ent languages (Fake English vs. {Hindi, Russian, Spanish}), theorizing this to be an effect of disrupting the model's reliance on lexical overlap. Our experiments in SS4 and SS5 lend support to this hypothesis. In Table 1, we see multiple examples where the prediction was flipped from "contradiction" to "entailment" simply by perturbing a few words. If the models did not rely on lexical overlap but performed comparisons at the semantic level, such perturbations should not have severely impacted their performance. Our results on QA also corroborate Lee et al. (2019)'s finding that models trained on SQuAD-style datasets exploit lexical overlap between the question and context.

## 5 Code-Mixed Adversarial Training

Finally, we propose code-mixed adversarial training (CAT), an extension of the standard adversarial training paradigm Goodfellow et al. (2015), to improve the robustness of multilingual models to adversarial polyglots. In standard adversarial training, adversarial attacks are run on the training set to generate adversaries for training. However, this makes adversarial training computationally expensive. Hence, we take inspiration from Tan et al. (2020a)'s method of randomly sampling perturbations from an adversarial distribution and generate code-mixed perturbations using word alignment.

To generate the code-mixed adversarial training set \(X^{\prime}\), we first compute the adversarial distribution \(\mathcal{P}_{adv}\) by enumerating the perturbations per embedded language in all successful adversaries (SS4). Formally, \(\mathcal{P}_{adv}=\{f_{i}\}_{i=1...\mathbb{L}|},\) where \(f_{i}=\frac{l_{i}}{\sum_{j=1}^{i+1}l_{j}}\) and \(\mathbb{L}\) is the set of embedded languages.

Next, for each clean example \(x\), we sample \(n\) languages from \(\mathcal{P}_{adv}\) before translating the example into the \(n\) languages and aligning the translations with \(x\). For sentence-pair classification tasks like NLI, we use a per-sentence \(n\) to further increase variation. Intuitively, limiting \(n\) improves the example's naturalness and the algorithm's efficiency (the alignment is the most costly step). We then extract phrases from the aligned sentences, yielding our candidate perturbations \(\mathbb{P}\). Next, we sample a perturbation with probability \(\rho\) from \(\mathbb{P}\) for each phrase in \(x\). Reducing \(\rho\) yields more natural sentences since they will be less perturbed. Finally, we apply these perturbations to \(x\), obtaining a CAT example \(x^{\prime}\). Doing this \(k\) times for all \(x\) in \(X\) and adding the result to \(X\) yields \(X^{\prime}\) (Alg. 3 in Appendix G).

In contrast to running the adversarial attack on the training set, sampling perturbations from a distribution does not guarantee that the resulting example will be adversarial to the model. This issue can be mitigated by increasing the number of CAT examples observed during training. However, this would increase the computational cost if we were to train the model for the same number of epochs. Hence, we set \(k\) to one less than the number of epochs XLM-Rbase was fine-tuned for in SS4 and train the model for one epoch on the adversarial training set. This exposes the model to more variation in the same number of training steps.

Setting.We conduct our experiments on NLI with XLM-Rbase with no loss of generality. In SS4, the model was trained for ten epochs. Hence, we set \(k=9,n=2,\rho=0.5\) for CAT and train all models for a similar number of steps (60k) with the same hyperparameters as SS4. We first test the models on the Bumblebee adversaries generated in SS4 before directly attacking the model. Next, we construct more realistic settings by running Bumblebee with only 1-2 embedded languages from standard XNLI, Swahili (sw), Hindi (hi), and Urdu (ur). These languages were the lowest resourced in the pretraining data Conneau et al. (2020a).

We also construct another _non-adversarial_ test set from XNLI by randomly choosing hypotheses and premises from different languages K et al. (2020). Since the original examples are individually monolingual, this test set will reveal if a model is simply exploiting lexical overlap rather than comparing the underlying concepts.

Finally, we run Bumblebee with embedded languages not seen during task-specific training _and_ from a different family (Austronesian) from the XNLI languages, Filipino (tl) and Indonesian (id). This zero-shot defense setting will reveal if CAT encourages the learning of more language-invariant representations, or is simply allowing the model to adapt to the adversarial distribution.

Baselines.Since training on languages in the test set takes us out of the cross-lingual transfer setting, we train a _translate-train-\(n\)_ baseline for a fair comparison. In this setting, we train on every \(x\) and its translations in the \(n\) languages sampled in CAT, regardless of whether they contributed words to the final CAT examples. We also include Ganin et al. (2016)'s domain adversarial neural network (DANN), which has been used for cross-lingual adaptation Joty et al. (2017); Chen et al. (2018) 



### Results

From Table 6, we observe that both training on fully translated data and on CAT examples improved accuracy on the non-adversarial test sets and robustness to code-mixed adversaries, compared to the cross-lingual transfer model that was only trained on English data. Similar to K et al. (2020), we found that disrupting the models' reliance on lexical overlap (\(\text{Clean}_{\text{DL}}\)) hurt performance. The drop was particularly significant for the cross-lingual transfer (8 points) and _translate-train-n_ models (5.24 points). On the other hand, our CAT model only suffered a 1.5-point drop, indicating that the former two models likely rely heavily on lexical overlap to make predictions, while our CAT model may be using "deeper", more language-agnostic features. Crucially, our CAT model achieves similar to better clean accuracy than the baselines, contrasting with prior work showing that adversarial training hurts clean accuracy Tsipras et al. (2019). Finally, our CAT model is >1.7x more robust to adversaries constructed from all fifteen XNLI languages than the _translate-train-n_ model. Although DANN-type training improved robustness to the previous Bumblebee adversaries, clean performance was significantly degraded and Bumblebee was able to find even more damaging adversaries upon attacking the model directly.

When attacked with 1-2 embedded languages that were seen during training, CAT also yields significant improvements in robustness over the baselines: a \(>7\) point increase compared to _translate-train-n_ and a >19 point gain over the zero-shot transfer setting. In the zero-shot defense setting, CAT shows a >12-point gain over the zero-shot transfer model and a >4.7-point gain over the _translate-train-n_ model. We believe these results to be due to CAT encouraging the learning of language-invariant representations by exposing the model to cross-lingual lexical variation and preventing the model from exploiting lexical overlaps.

## 6 Seeing is Believing

To further understand the effect of various fine-tuning methods on XLM-Rbase, we visualize the 
 vector from the layer before the classification head using t-SNE Linderman et al. (2019). Here, all sentences from XNLI are passed through the representations individually. If a representation were 100% language-invariant, we should expect t-SNE to be unable to separate individual languages into their own clusters. Hence, the extent to which t-SNE is able to do so would indicate the amount of language-specific information in this last layer.

From Fig. 1(a), we observe that for the cross-lingual transfer model (SS4), t-SNE managed to organize the sentences from several languages Chinese, Hindi, Thai, Urdu) into distinct clusters. This indicates that a significant amount of language-specific information remains in the vector representations of sentences from these languages. Visualizing the sequence-averaged embeddings makes this even clearer (Fig. 3 in Appendix G). Hence, while XLM-R may be multilingual, it appears to be structured as a space of individual language subspaces as opposed to a mixed, or language-invariant space. On the other hand, t-SNE was much less successful when given the representation trained with CAT (Fig. 1(b)). Mixing multiple languages in the same sentence and showing the model multiple variants of the same sentence likely encourages the model to refine its representation such that all variants of the same sentence are represented similarly, re

\begin{table}
\begin{tabular}{l|c c c|c c c|c c} \hline \hline
**Condition/Method** & **Clean** & **Clean\({}_{\text{DL}}\)** & **Adv\({}_{\text{sq}}\)** & **Adv\({}_{\text{sq}}\)** & **Adv\({}_{\text{bi-inter}}\)** & **Adv\({}_{\text{NNL}}\)** & **Adv\({}_{\text{bl+tl}}\)** \\ \hline Cross-lingual transfer (from §4) & 74.06 & 66.09 & 3.53 & 38.54 & 29.12 & 3.53 & 36.96 & 24.83 \\ Translate-train-\(n\) & **77.25** & 72.01 & 29.44 & 50.53 & 40.63 & 7.04 & 44.37 & 33.23 \\ DANN Ganin et al. (2016) & 51.86 & 35.10 & 33.45 & 16.02 & 17.52 & 6.54 & 12.05 & 7.06 \\ Code-mixed adv. training (CAT) & 77.10 & **75.46** & **50.21** & **58.58** & **48.20** & **12.63** & **49.14** & **38.16** \\ \hline \hline \end{tabular}
\end{table}
Table 6: Results on standard XNLI with XLM-Rbase. Clean refers to the combined test set of all languages, \(\text{Clean}_{\text{DL}}\) to the variant where the hypothesis and premise of each example are from different languages, Adv\({}_{\text{st}}\) to the Bumblebee adversaries from §4, and Adv\({}_{\text{(lg)}}\) to new adversaries from English + the subscripted languages. Higher is better.

Figure 2: t-SNE visualizations of XLM-Rbase representations fine-tuned using different methods.



sulting in a more language-invariant representation. T-SNE plots of the other models are in Appendix G.

## 7 Limitations and Future Work

We acknowledge that our methods do not fully model real code-mixing since we do not learn the mixing patterns from real data and there are subtleties in real code-mixing we ignore for simplicity, e.g., accounting for the prestige of participating languages Bhatia (2011). In addition, it is impossible to _guarantee_ the semantic preservation of a sentence generated by Bumblebee due to the word aligner's statistical nature, though we can expect more accurate alignments to improve semantic preservation. Finally, while CAT improves robustness, there remains a significant gap between the robust and clean accuracies. In line with recent work challenging the Anglocentricity of cross-lingual models Anastasopoulos Neubig (2020); Liang et al. (2020), a promising direction of future work lies in investigating how the choice of matrix language affects model robustness.

## 8 Conclusion

Ensuring that multilingual models are robust to both natural and adversarial code-mixing is important in today's increasingly multilingual world if they are to allow their target users to fully express themselves in human-machine conversations and to defend against adversarial users attempting to evade toxicity/misinformation detection systems.

To approximate a lower bound for model performance on lexical code-mixing, we propose two strong black-box multilingual adversarial attacks and demonstrate their effectiveness on state-of-the-art cross-lingual NLI and QA models. The former generates perturbations from bilingual dictionaries and disambiguates between senses using sentence translations, while the latter generates perturbations by aligning sentences from different languages.

Next, we show that training on code-mixed data synthesized via word alignment improves clean and robust accuracy when models are prevented from exploiting lexical overlap _without_ hurting clean accuracy. Crucially, we achieve this in the same number of steps as standard supervised training.

Finally, we use t-SNE visualizations to show that multilingual models are not necessarily language-invariant and that our code-mixed adversarial training scheme encourages language-invariance.

## 9 Broader Impact / Ethical Considerations

Adversarial attacks and defenses are double-edged swords. On one hand, adversarial examples expose the gaps in existing models and help to focus the research community's attention on flaws that need to be addressed before these models can be used reliably in noisy, real-world environments. On the other, the same adversarial attacks can be used by malicious actors to bypass toxicity/misinformation detection systems. Similarly, methods for improving adversarial robustness can be used to defend against malicious actors and improve robustness to natural noise or linguistic variation, yet they can also be used to strengthen automated censorship systems and limit freedom of speech. For example, our adversarial attacks could be used both as a lower bound for model performance on naturally occurring code-mixed text _and_ to bypass misinformation detection systems while preserving the message's intelligibility for multilingual speakers. Our adversarial training method could be used to both improve machine understanding of code-mixers by making multilingual representations more language-invariant _and_ suppress the freedom of speech of polyglots who could have been using code-mixing to evade censorship.

At the same time, technology strongly shapes our behavior Reeves et al. (2019). Consequently, given the centrality of code-switching/mixing to many polyglots' lived experiences Duff (2015) and the positive correlations between multilingual, code-switching, and creativity Leikin (2013); Kharkhurin and Wei (2015); Furst and Grin (2018), we should ensure that the natural language technologies we build do not inhibit multilingual speakers from fully expressing themselves, e.g., by discouraging code-mixing due to non-understanding. In addition, studies have found that aphasic polyglots code-mix more frequently than neurotypical polyglots to cope with word-retrieval difficulties Goral et al. (2019), making it important for natural language technologies to be robust to code-mixing if they are to be inclusive. Therefore, we include both adversary generation and defense methods to avoid tipping the balance too far in either direction.

## Acknowledgments

We would like to thank Cynthia Siew (NUS Psychology), Greg Bennett and Kathy Baxter Salesforce, Lav Varshney (UIUC Electrical and Computer Engineering), Min-Yen Kan (NUS Com 

puter Science), and our anonymous reviewers for their invaluable feedback. We are also grateful to Guangsen Wang, Mathieu Ravaut, Soujanya Lanka, and Tuyen Hoang for contributing manually code-mixed sentences and Bari M Saiful for pointers on replicating the XLM-R results. Samson is supported by Salesforce and Singapore's Economic Development Board under its Industrial Postgraduate Programme.

## References

* G. M. Abuhakema (2013)Code switching and code mixing in arabic written advertisements: Patterns, aspects, and the question of prestige and standardisation. The Internet Journal Language, Culture and Society. Cited by: SS4.
* E. Agirre and P. Edmonds (2007)Word sense disambiguation: algorithms and applications. Note: [http://www.cs.ucsdavis.edu/~gawan/Cited](http://www.cs.ucsdavis.edu/~gawan/Cited) by: SS5.
* Gustavo Aguilar and T. Solorio (2020)From English to code-switching: transfer learning with strong morphological clues. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 8033-8044. External Links: Link, Document Cited by: SS6.
* Emily Ahn et al. (2020)What code-switching strategies are effective in dialogue systems?. Proceedings of the Society for Computation in Linguistics3 (1), pp. 308-318. Cited by: SS6.
* F. AlGhamdi and M. Diab (2019)Leveraging pretrained word embeddings for part-of-speech tagging of code switching data. In Proceedings of the Sixth Workshop on NLP for Similar Languages, Varieties and Dialects, pp. 99-109. Cited by: SS6.
* Moustafa Alzantot et al. (2018)Generating natural language adversarial examples. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, pp. 2890-2896. External Links: Link, Document Cited by: SS6.
* A. Anastasopoulos and G. Neubig (2020)Should all cross-lingual embeddings speak english?. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 8658-8679. External Links: Link, Document Cited by: SS6.
* L. Aronin and D. Singleton (2008)Multilingual as a new linguistic dispensation. International Journal of Multilingualism5 (1), pp. 1-16. Cited by: SS6.
* M. Artetxe, S. Ruder, and D. Yogatama (2020)On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 4623-4637. External Links: Link, Document Cited by: SS6.
* M. Artetxe and H. Schwenk (2019)Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics7, pp. 597-610. Cited by: SS6.
* K. Bali, J. Sharma, M. Choudhury, and Y. Vyas (2014)"i am borrowing ya mixing?" an analysis of english-hindi code mixing in facebook. In Proceedings of the First Workshop on Computational Approaches to Code Switching, pp. 116-126. Cited by: SS6.
* K. Ball and D. Garrette (2018)Part-of-speech tagging for code-switched, transliterated texts without explicit language identification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, pp. 3084-3089. External Links: Link, Document Cited by: SS6.
* M. Safil R. Bari, T. Mohiuddin, and S. Joty (2020)Multimix: a robust data augmentation framework for cross-lingual nlp. arXiv preprint arXiv:2004.13240. Cited by: SS6.
* U. Barman, A. Das, J. Wagner, and J. Foster (2014)Code mixing: a challenge for language identification in the language of social media. In Proceedings of the first workshop on computational approaches to code switching, pp. 13-23. Cited by: SS6.
* Anshul Basua et al. (2020)Do multilingual users prefer chat-botts that code-mix? let's nudge and find out!. Proceedings of the ACM on Human-Computer Interaction4 (CCSW1), pp. 1-23. Cited by: SS6.
* Tej K Bhatia (2011)The multilingual mind, optimization theory, and hinglish. Chutneying English: The Phenomenon of Hanglish, pp. 37-52. Cited by: SS6.
* Matthias Blohm et al. (2018)Comparing attention-based convolutional and recurrent neural networks: success and limitations in machine reading comprehension. In Proceedings of the 22nd Conference on Computational Natural Language Learning, pp. 108-118. Cited by: SS6.
* K. Chandu, E. Loginova, V. Gupta, J. van Genabith, G. Neumann, M. Chinnakotla, E. Nyberg, and A. W. Black (2019)Code-mixed question answering challenge: crowdsourcing data and techniques. In Third Workshop on Computational Approaches to Linguistic CodesWitching, pp. 29-38. Cited by: SS6.

 

Xilun Chen, Yu Sun, Ben Athiwaratkun, Claire Cardie, and Kilian Weinberger. 2018. Adversarial deep averaging networks for cross-lingual sentiment classification. _Transactions of the Association for Computational Linguistics_, 6:557-570.
* Conneau et al. (2020) Alexis Conneau, Kartiaky Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzman, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020a. Unsupervised cross-lingual representation learning at scale. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 8440-8451, Online. Association for Computational Linguistics.
* Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2475-2485, Brussels, Belgium. Association for Computational Linguistics.
* Conneau et al. (2020b) Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2020b. Emerging cross-lingual structure in pretrained language models. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 6022-6034, Online. Association for Computational Linguistics.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Divekar (1984) Dileep A Divsekar. 1984. DC statistical circuit analysis for bipolar ic's using parameter correlations-an experimental example. _IEEE transactions on computer-aided design of integrated circuits and systems_, 3(1):101-103.
* Duff (2015) Patricia A Duff. 2015. Transnationalism, multilingual, and identity. _Annual Review of Applied Linguistics_, 35:57.
* Ebrahimi et al. (2018) Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018. HotFlip: White-box adversarial examples for text classification. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 31-36, Melbourne, Australia. Association for Computational Linguistics.
* Furst and Grin (2018) Guillaume Furst and Francois Grin. 2018. Multilingualism and creativity: a multivariate approach. _Journal of Multilingual and Multicultural Development_, 39(4):341-355.
* Ganin et al. (2016) Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Francois Laviolette, Mario Marchand, and Victor Lempitsky. 2016. Domain-adversarial training of neural networks. _The Journal of Machine Learning Research_, 17(1):2096-2030.
* Garg and Ramakrishnan (2020) Siddhant Garg and Gottham Ramakrishnan. 2020. BAE: BERT-based adversarial examples for text classification. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_, Online. Association for Computational Linguistics.
* Goodfellow et al. (2015) Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. In _3rd International Conference on Learning Representations_, San Diego, California.
* Moral et al. (2019) Mira Goral, Monica Norvik, and Bard Uri Jensen. 2019. Variation in language mixing in multilingual aphasia. _Clinical linguistics & phonetics_, 33(10-11):915-929.
* Han et al. (2020) Wenjuan Han, Liwen Zhang, Yong Jiang, and Kewei Tu. 2020. Adversarial attack and defense of structured prediction models. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_, Online. Association for Computational Linguistics.
* Huang et al. (2019) Haoyama Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Davin Jiang, and Ming Zhou. 2019. Unicoder: A universal language encoder by pretraining with multiple cross-lingual tasks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 2485-2494.
* Jin et al. (2020) Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2020. Is BERT really robust? A strong baseline for natural language attack on text classification and entailment. In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_, pages 8018-8025. AAAI Press.
* Joshi (1982) Aravind K. Joshi. 1982. Processing of sentences with intra-sentential code-switching. In _Coling 1982: Proceedings of the Ninth International Conference on Computational Linguistics_.
* Joty et al. (2017) Shafiq Joty, Preslav Nakov, Lluis Marquez, and Israa Jaradat. 2017. Cross-language learning with adversarial neural networks. In _Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017)_, pages 226-237, Vancouver, Canada. Association for Computational Linguistics.

 

Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. 2020. Cross-lingual ability of multilingual bert: An empirical study. In _International Conference on Learning Representations_.
* Kochru (1978) Braj B Kachru. 1978. Toward structuring code-mixing: An indian perspective. _International Journal of the Sociology of Language_, 1978(16):27-46.
* Khanuja et al. (2020) Simran Khanuja, Sandipan Dandapat, anirudh Srinivasan, Sunayana Sitaram, and Monojit Choudhury. 2020. GLUECoS: An evaluation benchmark for code-switched NLP. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 3575-3585, Online. Association for Computational Linguistics.
* Kharkhurin and Wei (2015) Anatoliy V Kharkhurin and Li Wei. 2015. The role of code-switching in bilingual creativity. _International Journal of Bilingual Education and Bilingual_, 18(2):153-169.
* Klementeiv et al. (2012) Alexandre Klementeiv, Ivan Titov, and Binod Bhattarai. 2012. Inducing crosslingual distributed representations of words. In _Proceedings of COLING 2012_, pages 1459-1474, Mumbai, India. The COLING 2012 Organizing Committee.
* Koehn (2010) Philipp Koehn. 2010. _Statistical Machine Translation_, 1st edition. Cambridge University Press, New York, NY, USA.
* Lample et al. (2018) Guillaume Lample, Alexis Conneau, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herve Jegou. 2018. Word translation without parallel data. In _International Conference on Learning Representations_.
* Lauscher et al. (2020) Anne Lauscher, Vinit Ravishankar, Ivan Vulic, and Goran Glavas. 2020. From zero to hero: On the limitations of zero-shot language transfer with multilingual transformers. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_, Online. Association for Computational Linguistics.
* Lee et al. (2019) Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 6086-6096, Florence, Italy. Association for Computational Linguistics.
* Leikin (2013) Mark Leikin. 2013. The effect of bilingualism on creativity: Developmental and educational perspectives. _International Journal of Biltingaulism_, 17(4):431-447.
* Li et al. (2019)Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, and Ting Wang. 2019. Textbugger: Generating adversarial text against real-world applications. In _26th Annual Network and Distributed System Security Symposium_.
* Liang et al. (2020) Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fengfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Dakin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Singhe Wei, Taroon Bharti, Ying Qiao, Jiunn-Hung Chen, Winie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, and Ming Zhou. 2020. Xglue: A new benchmark dataset for cross-lingual pre-training, understanding and generation. _arXiv preprint arXiv:2004.01401_.
* Linderman et al. (2019) George C Linderman, Manas Rachh, Jeremy G Hoskins, Stefan Steinerberger, and Yuval Kluger. 2019. Fast interpolation-based t-SNE for improved visualization of single-cell RNA-seq data. _Nature methods_, 16(3):243-245.
* Matras and Sakel (2007) Yaron Matras and Jeanette Sakel. 2007. _Grammatical borrowing in cross-linguistic perspective_, volume 38. Walter de Gruyter.
* Michel et al. (2019) Paul Michel, Xian Li, Graham Neubig, and Juan Pino. 2019. On evaluation of adversarial perturbations for sequence-to-sequence models. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 3103-3114, Minneapolis, Minnesota. Association for Computational Linguistics.
* Miller (1995) George A. Miller. 1995. Wordnet: A lexical database for english. _Communications of the ACM_, 38:39-41.
* Muller et al. (2020) Benjamin Muller, Benoit Sagot, and Djame Seddah. 2020. Can multilingual language models transfer to an unseen dialect? a case study on north african arabizi. _arXiv preprint arXiv:2005.00318_.
* Myers-Scotton (1997) Carol Myers-Scotton. 1997. _Dwelling languages: Grammatical structure in codeswitching_. Oxford University Press.
* Patwa et al. (2020) Parth Patwa, Gustavo Aguilar, Sudipta Kar, Suraj Pandey, Srinivas PYKL, Bjorn Gamback, Tanmoy Chakraborty, Thamar Solorio, and Amitava Das. 2020. Semeval-2020 task y: Overview of sentiment analysis of code-mixed tweets. In _Proceedings of the 14th International Workshop on Semantic Evaluation (SemEval-2020)_, Barcelona, Spain. Association for Computational Linguistics.
* Phathiyaphaibun et al. (2016) Wannaphong Phathiyaphaibun, Korakot Chaovananchikh Vanich, Archit Suriyawangkul Charin Polpanumas, Lalita Lowhansirikl, and Pattarawat Chormai. 2016. PyThailNLP: Thai Natural Language Processing in Python.
* Pires et al. (2019) Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual bert? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4996-5001.
* Policar et al. (2019) Pavlin G. Policar, Martin Strazar, and Blaz Zupan. 2019. openTSNE: a modular Python library for t-SNE dimensionality reduction and embedding. _bioRxiv_ 

M. Popel, M. Tomkova, J. Tomek, Lukasz Kaiser, Jakob Uszkoreit, Ondrej Bojar, and Z. Zabokrtsky (2020)Transforming machine translation: a deep learning system reaches news translation quality comparable to human professionals. Nature Communications11, pp. . External Links: Document Cited by: SS3.
* S. Poplack (1980)Sometimes i'll start a sentence in spanish y termino en espanol: toward a typology of code-switching. Linguistics18 (7-8), pp. 581-618. Cited by: SS4.
* S. Podleck, D. Sankoff, and C. Miller (1988)The social correlates and linguistic processes of lexical borrowing and assimilation. Linguistics26 (1), pp. 47-104. Cited by: SS1.
* A. Pratapa, G. Bhat, M. Choudhury, S. Sataram, S. Dandapat, and K. Bali (2018)Language modeling for code-mixing: the role of linguistic theory based synthetic data. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1543-1553. Cited by: SS1.
* P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang (2016)SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, Texas, pp. 2383-2392. Cited by: SS1.
* B. Reeves, N. Ram, T. N. Robinson, J. J. Cummings, C. Lee Giles, J. Pan, A. Chiatti, M. Cho, K. Roehrick, X. Yang, et al. (2019)Screenomics: a framework to capture and analyze personal life experiences and the ways that technology shapes them. Human-Computer Interaction, pp. 1-52. Cited by: SS1.
* S. Ren, Y. Deng, K. He, and W. Che (2019)Generating natural language adversarial examples through probability weighted word saliency. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia, pp. 1085-1097. Cited by: SS1.
* M. Tulio Ribeiro, S. Singh, and C. Guestrin (2018)Semantically equivalent adversarial rules for debugging NLP models. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia, pp. 856-865. Cited by: SS1.
* B. Roxf, L. Wolf-Sonkin, C. Kirov, S. J. Mielke, C. Johny, I. Demirsahin, and Keith Hall (2020)Processing south Asian languages written in the Latin script: the Dakshina dataset. In Proceedings of The 12th Language Resources and Evaluation Conference (LREC), Vol. , pp. 2413-2423. Cited by: SS1.
* U. Roy, N. Constant, R. Al-Rfou, A. Barua, A. Phillips, and Y. Yang (2020)LAReQA: language-agnostic answer retrieval from a multilingual pool. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 5919-5930. Cited by: SS1.
* M. Jalili Sabet, P. Dufter, and H. Schutze (2020)Simalign: high quality word alignments without parallel training data using static and contextualized embeddings. arXiv preprint arXiv:2004.08728. Cited by: SS1.
* S. Sutaram, K. Ehandu, S. Krishna Rollandadi, and A. W. Black (2019)A survey of code-switched speech and language processing. arXiv preprint arXiv:1904.00784. Cited by: SS1.
* T. Solorio, E. Blair, S. Maharjan, S. Bethard, M. Diab, M. Ghoneim, A. Hawwari, F. AlGhamdi, J. Hirschberg, A. Chang, and P. Fung (2014)Overview for the first shared task on language identification in code-switched data. In Proceedings of the First Workshop on Computational Approaches to Code Switching, Doha, Qatar, pp. 62-72. Cited by: SS1.
* K. Song, X. Tan, T. Qin, J. Lu, and T. Liu (2019)MASS: masked sequence to sequence pre-training for language generation. In International Conference on Machine Learning, pp. 5926-5936. Cited by: SS1.
* Shikaripur N. Sridhar and K. K. Sridhar (1980)The syntax and psycholinguistics of bilingual code mixing. Canadian Journal of Psychology/Revue canadienne de psychologie34 (4), pp. 407. Cited by: SS1.
* C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. J. Goodfellow, and R. Fergus (2014)Intriguing properties of neural networks. In 2nd International Conference on Learning Representations, Banff, AB, Canada, pp. . Cited by: SS1.
* P. Tang (1988)A description of patterns of code-mixing and code-switching in a multilingual household. In Newpion Peley, editor, New Englishes: The Case of Singapore, pp. . Note: US Press Cited by: SS1.
* S. Tan, S. Joty, M. Kan, and R. Socher (2020a)It's morphin' time! Combating linguistic discrimination with inflectional perturbations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 2920-2935. Cited by: SS1.
* S. Tan, S. Joty, L. Varshney, and M. Kan (2020b)MID your infections! Improving NLP for non-standard Englishes with Base-Inflection Encoding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, Online, Section for Computational Linguistics, pp. . Cited by: SS1.
* M. W. J. Tay (1989)Code switching and code mixing as a communicative strategy in multilingual discourse. World Englishes8 (3), pp. 407-417.

 

Jorg Tiedemann and Santhosh Thottingal. 2020. OPUS-MT -- Building open translation services for the World. In _Proceedings of the 22nd Annual Conference on European Association for Machine Translation (EAMT)_, Lisbon, Portugal.
* Tsipras et al. (2019) Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. 2019. Robustness may be at odds with accuracy. In _International Conference on Learning Representations_.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems 30_, pages 5998-6008. Curran Associates, Inc.
* Wallace et al. (2019) Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing NLP. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, Hong Kong, China. Association for Computational Linguistics.
* Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 1112-1122. Association for Computational Linguistics.
* Winata et al. (2019) Genta Indra Winata, Andrea Madduto, Chien-Sheng Wu, and Pascale Fung. 2019. Code-switched language models using neural based synthetic data from parallel sentences. In _Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)_, pages 271-280, Hong Kong, China. Association for Computational Linguistics.
* Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remii Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. _arXiv preprint arXiv:1910.03771_.
* Wu and Dredze (2019) Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of bert. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 833-844.
* Yakpo (2015) Kofi Yakpo. 2015. Code-switching and social change: Convergent language mixing in a multilingual society. _Code-switching Between Structural and Sociolinguistic Perspectives_, 43:259.
* Yang et al. (2020) Zhen Yang, Bojie Hu, Ambyera Han, Shen Huang, and Qi Ju. 2020. Code-switching pre-training for neural machine translation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_, Online. Association for Computational Linguistics.
* Zang et al. (2020) Yuan Zang, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Meng Zhang, Qun Liu, and Maosong Sun. 2020. Word-level textual adversarial attacking as combinatorial optimization. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 6066-6080.
* Zhang et al. (2019) Huangzhao Zhang, Hao Zhou, Ning Miao, and Lei Li. 2019. Generating fluent adversarial examples for natural languages. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 5564-5569, Florence, Italy. Association for Computational Linguistics.
* Zhang et al. (2020) Wei Emma Zhang, Quan Z. Sheng, Ahoud Alhazmi, and Chenliang Li. 2020. Adversarial attacks on deep-learning models in natural language processing: A survey. _ACM Trans. Intell. Syst. Technol._, 11(3).

 

Examples of Real Code-Mixed Text

**English + Spanish**Sridhar and Sridhar, 1980):
* The man que vino ayer (who came yesterday) wants into buyun carro nuevo (a new car).
* El (The) old man esta enojide (is mad).
* Me lleve chile va foasted y peeled ... para hacerlo ells. (I picked up the chile already roasted and peeled for making it there.)
**Hindi + English**Bali et al., 2014):
* Befitting reply to mere papa ne maaraa (My father gave a befitting reply):
* ... and the party workers [will] come with me without virodin (protest/objection)
**Sarnami + Sranan + Dutch**Yakpo, 2015):
* Rus gewoem calclar jaiye, tab ego kerki links ki rechts. (So just keep on walking, then [there's] achurch, left or right.)
* kaun wala damru, ego haiga jaum men na verfi bhail, ma ego wel hai. (Which [kind of] damru drum, there's one which is not coloured inside but one actually is.)

## Appendix B Attack Implementation Details

### PolyGloss

```
0: Clean example-label pair \((x,y)\), Target Model \({\cal M}\), Embedded languages \(\mathbb{L}\) Enspective: Adversarial example \(x^{\prime}\)\(T\leftarrow\textsc{Translate}(x,\text{target-languages}=\mathbb{L})\)\(\mathcal{L}_{x}\leftarrow\textsc{GetLoss}({\cal M},x,y)\)\(\triangleright\) Initialize beam whileNotEmpty(\(B\)do\(\mathcal{L}_{x},c_{x},c_{i}\leftarrow\textsc{Poll}(B)\)\(C\gets\textsc{GetCandidates}(x_{c},\text{token-id}=i)\)\(C\leftarrow\textsc{FilterCandidates}(C,T)\)\(\mathcal{L}\leftarrow\textsc{GetLoss}({\cal M},C,y)\)\(\triangleright\) Losses for \(C\)\(i\gets i+1\) UpdateBeam\((B,\mathcal{L},C,i)\) endwhile\(x^{\prime}\leftarrow\textsc{Poll}(B)\)return\(x^{\prime}\)
```

**Algorithm 2****PolyGloss**

To reduce the practical running time of our attack, we make use of cross-lingual dictionaries released by Lample et al. (2018) for generating candidate perturbations instead of translating the words in an online fashion. We also use the gold translations of the clean examples when they are available (such as in XNLI), and use the models released by Tiedemann and Thottingal (2020) in the transformers library Wolf et al. (2019) to translate the examples to other languages. We also cache them in hashtables for fast retrieval.

### Bumbblebee

We use the gold translations where available and Tiedemann and Thottingal (2020)'s translation models for the other languages, and align sentences with a neural word aligner Sabet et al. (2020) backed by XLM-Rbase in our implementation of Bumbblebee. Although Sabet et al. (2020) found the "Itermax" algorithm to yield the best performance for their experimental settings, we suggest using the high recall ("Match") algorithm for candidate generation. We inspected the output of both algorithms and found that while Itermax generates more candidates, it also tends to generate noisier alignments compared to Match, which we found to be more conservative.

## Appendix C PolygLoss and CAT Samples

Almost random samples for PolyGloss and CAT (we tried to include the sentences with Thai and Hindi characters but did not manage to get them to render with pdflatex).

 

Experiment Details

### Datasets

Standard XNLI3 comprises 7,500 _parallel_ examples (2,490 dev., 5,010 test) in fifteen languages: English (en), Spanish (es), German (de), Greek (el), Russian (ru), Turkish (tr), Arabic (ar), Vietnamese (vi), Thai (th), Chinese (zh), Hindi (hi), French (fr), Bulgarian (bg), Swahili (sw), and Urdu (ur). The labels are uniformly distributed between the three classes (contradiction, neutral, entailment).

Footnote 3: cims.nyu.edu/~sbowman/xnli

The machine-translated training set4 of standard XNLI comprises the MNLI training examples in addition to their translations in the same fourteen non-English languages as the dev. and test sets.

XNLI-13 comprises all standard XNLI languages except Swahili and Urdu due to the lack of suitable dictionaries for PolyGloss. XNLI-31 comprises all languages in XNLI-13, in addition to another eighteen: Afrikaans (af), Albanian (sq), Catalan (ca), Czech (cs), Danish (da), Dutch (nl), Estonian (et), Filipino (tl), Finnish (fi), Hebrew (he), Hungarian (hu), Indonesian (id), Italian (it), Macedonian (mk), Romanian (ro), Slovak (sk), Swedish (sv), and Ukrainian (uk).

Footnote 4: Same link as XNLI dev/test set.

MNLF comprises 392,702 examples in English with the following label distribution: 130,899tailment, 130,900 neutral, 130,903 contradiction.

XQuAD6 comprises 1,190 question-answer pairs with guaranteed answers in eleven languages: English (en), Spanish (es), German (de), Greek (el), Russian (ru), Turkish (tr), Arabic (ar), Vietnamese (vi), Thai (th), Chinese (zh), and Hindi (hi). XQuAD examples are drawn from the SQuAD 1.1 development set.

Footnote 6: github.com/deepmind/xquad

SQuAD 1.17 comprises 87,599 question-answer pairs with guaranteed answers in English.

Footnote 7: rajpurkar.github.io/../train-v1.l.json

#### Metrics

The metric used for XNLI is simple accuracy:

\[{\rm Accuracy}=\frac{\#\,{\text{true positive}}}{\#\,{\text{total}}} \tag{2}\]

The metrics used for SQuAD are F\({}_{1}\):

\[{\rm F}_{1}=\frac{2\cdot{\rm precision}\cdot{\rm recall}}{{\rm precision}+{ \rm recall}} \tag{3}\]

and exact match:

\[{\rm Exact\;Match}=\frac{\#\,\hat{y}=y}{\#\,{\rm total}} \tag{4}\]

#### XQuAD Preprocessing

We use Phathiyaphaibun et al. (2016) to tokenize Thai text, jieba8 for Chinese text, and split all other languages on whitespace.

Footnote 8: github.com/fxsy/jieba

### Training Details

Hyperparameters.Table 9 contains the hyperparameters we used to fine-tune our models on MNLI. We used the hyperparameters suggested by Devlin et al. (2019)9 for mBERT, the hyperparameters suggested by Bari et al. (2020) for the XLM-R models, and the hyperparameters from Liang et al. (2020) for Unicoder.

Footnote 9: [https://github.com/google-research/.../multilingual.md](https://github.com/google-research/.../multilingual.md)

Table 10 contains the hyperparameters we used to fine-tune our models on SQuAD 1.1. We used the default SQuAD hyperparameters from the transformers library10 for XLM-Rbase and adjusted the hyperparameters for XLM-Rlarge to fit it onto the GPU. The mBERT model from the HuggingFace model repository11 was used instead of fine-tuning our own. The clean scores reported in Table 4 are similar to those reported in the XQuAD GitHub repository.

Footnote 10: github.com/huggingface/.../question-answering

Footnote 11: huggingface.co/salti/bert-base-multimingual-cased-finetuned-squad

\begin{table}
\begin{tabular}{p{56.9pt} p{56.9pt} p{56.9pt} p{56.9pt}} \hline \hline
**Model** & **Learning rate** & **Batch Size** & **Epochs** \\ \hline XLM-Rlarge & 1e-05 & 32 & 3 \\ XLM-Rbase & 3e-05 & 64 & 2 \\ \hline \hline \end{tabular}
\end{table}
Table 10: Hyperparameters for model fine-tuning on XQuAD (SQuAD 1.1).

\begin{table}
\begin{tabular}{p{56.9pt} p{56.9pt} p{56.9pt} p{56.9pt}} \hline \hline
**Model** & **Params.** & **Lr** & **Bsz** & **Epochs** \\ \hline XLM-Rlarge & 550M & 1e-06 & 64 & 10 \\ XLM-Rbase & 270M & 5e-06 & 64 & 10 \\ mBERTbase & 172M & 5e-05 & 64 & 2 \\ Unicoderbase & 270M & 5e-06 & 64 & 10 \\ \hline \hline \end{tabular}
\end{table}
Table 9: Hyperparameters for model fine-tuning on NNLI (MNLI). Number of parameters reproduced from Conneau et al. (2020).

 



### Validation Performance for CAT Exps.

#### Infrastructure Details

Models were trained on single V100 GPUs. Attacks were run on 8 V100 GPUs, parallelized with ray12 to make full use of GPU memory. On the standard XNLI test set (15 languages, 5,010 examples), Bumblebee takes 60-90 minutes in total. With 1 embedded language, Bumblebee runs on the test set in under 10 minutes. PolyGloss is generally much faster (under 30 minutes on XNLI-31) due to not needing a neural aligner.

Footnote 12: github.com/ray-project/ray

## Appendix E Extra Bumblebee Experiments

Beam search.In our experiments, we found that increasing the beam width yielded a higher attack success rate. However, this increases running time with only minor improvements (Table 12). We found increasing the number of embedded languages (and hence candidates) to be a more efficient method of increasing the success rate with a minor increase in running time. Although the time complexity (in number of model queries) is \(O(|B||C||{\mathbbm{L}}||S|)\) where \(|S|\) is the sentence length, increasing \(|{\mathbb{L}}|\) had a greater impact on the success rate than increasing \(|B|\) by the same number.

Fully unsupervised adversaries.A potential drawback of Bumblebee is that it requires translations of the clean example, which may be challenging to obtain for low-resource languages. However, it is possible to use unsupervised MT models for this purpose. We use Song et al. (2019)'s unsupervised English-French model to generate translations as a proof of concept and find that they achieve similar results (Table 13).

## Appendix F Plausible Language Combinations

To explore the effect of different combinations of languages on a multilingual model's performance, we run Bumblebee with different sets of embedded languages that could be plausibly spoken by (adversarial) polyplots around the world. English is used as the matrix language for all experiments.

We observe a general trend of decreasing model accuracy as we increase the number of mixed languages (Table 14), and XLM-Rbase's robustness to an embedded language appears to be generally more dependent on the size of its pretraining dataset than language typology. For example, Russian _(en+ru)_ and Indonesian _(en+id)_, languages with notably high accuracies, were the two most resourced languages after English in the corpus used for pretraining Conneau et al. (2020a), while Swahili and Filipino were among the lower-resourced languages. A notable outlier is Afrikaans _(en+af)_, which was the lowest resourced language in our experiments yet XLM-Rbase was quite robust to adversaries constructed from it and English. A possible explanation is that Afrikaans' language family, Indo-European, is highly represented in the pretraining corpus. Another notable outlier is Vietnamese, which was the fourth most resourced language in the pretraining corpus, yet the model was more vulnerable to adversaries constructed from English and Vietnamese than adversaries constructed from English and Filipino, one of the lowest resourced languages. A _possible_ explanation for this is the use of Latin characters combined with little vocabulary overlap between English and Vietnamese, and differing adjective positions.

\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Model** & **Clean** & **Supervised** & **Unsupervised** \\ \hline XLM-Rbase & 81.32 & 49.00 & 46.78 \\ \hline \hline \end{tabular}
\end{table}
Table 13: Accuracy after running Bumblebee on XNLI in supervised and unsupervised settings (matrix: English, embedded: French).

\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Variable (\(v\))** & \(v=1\) & \(v=2\) & \(v=3\) \\ \hline Beam width & 48.52\% & 48.95\% & 49.67\% \\ Embedded lgs. & 48.52\% & 69.09\% & 75.73\% \\ \hline \hline \end{tabular}
\end{table}
Table 12: Effect of increasing the beam width vs. the number of embedded languages on the attack success rate (%) while holding the other variable constant at 1. We use Swahili, Swahili and French, and Swahili, French, and Spanish as the embedded languages when \(v=1,2,3\), respectively. Rates are computed relative to the average clean XNLI score on the languages involved.



We also find XLM-Rbase to be twice as robust in the _en+da+de+nl+sv_ condition as the _en+id+th+tl+vi_ condition. It is likely that the structural similarity of the mixed languages also plays an important role in determining the model robustness, reinforcing K et al. (2020)'s similar findings on cross-lingual transfer. Investigating the effects of typology, vocabulary and orthographic overlap, pretraining corpus size, and interaction effects between different sets of languages on a model's robustness to code-mixed adversaries could lead to new insights into how different languages interact in the multilingual embedding space and we leave this to future work.

## Appendix G More Tables, Figures, and Algorithms

```
Original examples \(X\), Embedded languages L, Num. perturbed examples \(k\), Adversarial distribution \({\cal P}_{adv}\), Max. langs. per example \(n\), Phrase perturbation prob. \(\rho\) Answures: Adversarial training set \(X^{\prime}\)\(X^{\prime}\leftarrow\left\{\varnothing\right\}\) for\(x\) in \(X\)do \(\mathbb{S}\leftarrow\textsc{SampleLanguages}(\mathrm{L},n,{\cal P}_{adv})\) \(T\leftarrow\textsc{Translate}(x,\text{target-languages}=\mathbb{ S})\) \(\mathbb{P}\leftarrow\textsc{AlionandExtractPhrases}(x,T)\) for\(i=1\) to \(k\)do \(x^{\prime}\leftarrow\textsc{Peturby}(x,\mathbb{P},\rho)\) \(X^{\prime}\gets X^{\prime}\cup\{x^{\prime}\}\) endfor return\(X\cup X^{\prime}\)
```

**Algorithm 3** CAT Example Generation

Figure 3: Domain Adversarial Neural Network





\begin{table}
\begin{tabular}{l|p{213.8pt}} \hline \hline English & **P:** Americans should also consider how to do it-organizing their government in a different way. \\  & **H:** The American government might be organized in a different way. \\ French (R) & **P:** Les Americans devraient aussi reflechir to de le faire, en organismant in la different way. \\  & **H:** The governement americain might be organized in a different way. \\ French (Bb) & **P:** Americans should also penser à comment organiser leur gouvernement differenmment. \\  & **H:** Le gouvernement americain est maybe organized differently. \\ Hindi (R) & **P:** Americans ko yeh bhi sochna hai ki kaise karna haï-organizing their government in a different way. \\  & **H:** America ki sarkar might be organized in a different way. \\ Hindi (Bb) & **P:** amerikiyon chahivek iasi oichvar lese to karna it-organizing their government in a different way. \\ Chinese (R) & **P:** Americanaßßßßsonider [7]ögovernment \\  & **H:** Gögoverment [7]öorganization/?_ä_’ \\ Vietnamese (R) & **P:** Nguü Mi cung nén consider how to do it-organizing their government theo mût cach khâc. \\  & **H:** Chinh ph Mü co thé duc organized theo mût cach khâc. \\ \hline English & **P:** When that occurs, the lending fund sacrifices interest from Treasury securities on its invested balances and instead receives interest from the borrowing fund on the amount of the loan. \\  & **H:** The lending fund doesn’t get all the interest in some cases. \\ French (R) & **P:** Lorsque cela se produit, le lending funds de sacrife interest des Tresor securities sur ses investiss balances et plutôt receives intérêtts du borrowing fund sur le montant du les loan. \\  & **H:** The lending fund ne recoir pass all the interest in some cases. \\ French (Bb) & **P:** Quand ca arrive, le lending fund sacrifice l’intéret des Treasury securities sur ses investissements et recoil des interest from the borrowing fund on the amount du prét. \\  & **H:** Le fending fund ne recoir pastous les interest in some cases. \\ Hindi (R) & **P:** lab aisa hota hai, the lending fund sacrifices interest from Treasury securities on its invested balances and instead receives interest from the borrowing fund on the amount of the loan. \\  & **H:** Aise marlo rein, the lending fund doesn’t get all the interest. \\ Chinese (R) & **P:** Ji xh t w  lendering fund [7]ötreasy securities!!j investment interests, [7]ötorwing fund [7]ötespi Interest \\  & **H:** lending fund [7]ötespi Interest \\ Chinese (Bb) & **P:** Ji that occurs, the lending fund sacrifices [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]ötespi [7]öötespi [7]ötespi [7]öötespi [7]ötespi [7]öötespi [7]öötespi [7]öötespi [7]öötespi [7]ööötespi [7]öötespi [7]ööööötespi [7]



\begin{table}
\begin{tabular}{c|c c c c c} \hline \hline
**Model** & **Clean** & **PGutil.** & **PGfit.** & **Bumblebee** & **Bumblebee**unconstr \\ \hline XLM-R\({}_{\text{large}}\) & 81.10 & 6.06 & 28.28 & 11.31 & 5.22 \\ XLM-R\({}_{\text{base}}\) & 75.42 & 2.17 & 12.27 & 5.08 & 1.47 \\ mBERT\({}_{\text{base}}\) & 67.54 & 2.15 & 9.24 & 6.10 & 1.19 \\ Unicoder\({}_{\text{base}}\) & 74.98 & 1.99 & 11.33 & 4.81 & 1.29 \\ \hline \hline \end{tabular}
\end{table}
Table 16: PolyGloss (PG) and Bumblebee results on the XNLI-13 test set with a beam width of 1. PG\({}_{\left\{\text{fil, unfit.}\right\}}\) indicates whether the candidate substitutions were filtered using reference translations. Bumblebee\({}_{\text{unconstr}}\) refers to the setting without the equivalence constraint (§3.2).

Figure 4: t-SNE visualizations of XLM-R\({}_{\text{base}}\) representations fine-tuned using different methods. We use Linderman et al. (2019)’s t-SNE algorithm implemented in openTSNE (Poličar et al., 2019) and tried to arrange related languages close to each other on the color spectrum (to the extent possible on one dimension) so it would be obvious if similar languages were getting clustered together, as in (c).



Figure 5: t-SNE visualizations of XLM-R\({}_{\text{base}}\) representations fine-tuned using different methods. Here, we average all the token embeddings in the sentence instead of just using the \(<\)s\(>\) embedding.