Alignment of Embeddings for Parallel Documents	If language-adversarial training encourages language-independent features, then the English documents and their translations should be close in the embedding space. To examine this hypothesis, we take the English documents from the MLDoc training corpus and translate them into German, Spanish, French, etc. using Amazon Translate. We construct the embeddings for each document using BERT models finetuned on MLDoc. We mean-pool each document embedding to create a single vector per document. We then calculate the cosine similarity between the embeddings for the English document and its translation. In Table TABREF21 , we observe that the median cosine similarity increases dramatically with adversarial training, which suggests that the embeddings became more language-independent.

Adversarial Learning with Contextual Embeddings for Zero-resource Cross-lingual Classification and NER	Contextual word embeddings (e.g. GPT, BERT, ELMo, etc.) have demonstrated state-of-the-art performance on various NLP tasks. Recent work with the multilingual version of BERT has shown that the model performs very well in zero-shot and zero-resource cross-lingual settings, where only labeled English data is used to finetune the model. We improve upon multilingual BERT's zero-resource cross-lingual performance via adversarial learning. We report the magnitude of the improvement on the multilingual MLDoc text classification and CoNLL 2002/2003 named entity recognition tasks. Furthermore, we show that language-adversarial training encourages BERT to align the embeddings of English documents and their translations, which may be the cause of the observed performance gains.

Translation as Pretraining	  and  use the encoders from machine translation models as a starting point for task-specific finetuning, which permits various degrees of multilingual transfer.  add an additional masked translation task to the BERT pretraining process, and the authors observed an improvement in the cross-lingual setting over using the monolingual masked text task alone.

Model Training	We present an overview of the adversarial training process in Figure FIGREF1 . We used the pretrained cased multilingual BERT model as the initialization for all of our experiments. Note that the BERT model has 768 units. We always use the labeled English data of each corpus. We use the non-English text portion (without the labels) for the adversarial training. We formulate the adversarial task as a binary classification problem (i.e. English versus non-English.) We add a language discriminator module which uses the BERT embeddings to classify whether the input sentence was written in English or the non-English language. We also add a generator loss which encourages BERT to produce embeddings that are difficult for the discriminator to classify correctly. In this way, the BERT model learns to generate embeddings that do not contain language-specific information. The pseudocode for our procedure can be found in Algorithm SECREF11 . In the description that follows, we use a batch size of 1 for clarity.

In the case of NER, the task-specific loss has an additional summation over the length of the sequence: INLINEFORM0 INLINEFORM1  where INLINEFORM0 (dim: INLINEFORM1 ) is the prediction for the INLINEFORM2 word, INLINEFORM3 is the number of words in the sentence, INLINEFORM4 (dim: INLINEFORM5 ) is the matrix of one-hot entity labels, and INLINEFORM6 (dim: INLINEFORM7 ) refers to the BERT embedding of the INLINEFORM8 word. The generator and discriminator losses remain the same for NER, and we continue to use the mean-pooled BERT embedding during adversarial training. We then take the gradients with respect to the 3 losses and the relevant parameter subsets. The parameter subsets are INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 . We apply the gradient updates sequentially at a 1:1:1 ratio. During BERT finetuning, the learning rates for the task loss, generator loss and discriminator loss were kept constant; we do not apply a learning rate decay. All hyperparameters were tuned on the English dev sets only, and we use the Adam optimizer in all experiments.

Discussion	For many of the languages examined, we were able to improve on BERT's zero-resource cross-lingual performance on the MLDoc classification and CoNLL NER tasks. Language-adversarial training was generally effective, though the size of the effect appears to depend on the task. We observed that adversarial training moves the embeddings of English text and their non-English translations closer together, which may explain why it improves cross-lingual performance. Future directions include adding the language-adversarial task during BERT pre-training on the multilingual Wikipedia corpus, which may further improve zero-resource performance, and finding better stopping criteria for zero-resource cross-lingual tasks besides using the English dev set.

There are many recent approaches to zero-resource cross-lingual classification and NER, including adversarial learning  ,  ,  ,  , using a model pretrained on parallel text  ,  ,  and self-training  . Due to the newness of the subject matter, the definition of `zero-resource' varies somewhat from author to author. For the experiments that follow, `zero-resource' means that, during model training, we do not use labels from non-English data, nor do we use human or machine-generated parallel text. Only labeled English text and unlabeled non-English text are used during training, and hyperparameters are selected using English evaluation sets. Our contributions are the following:

Introduction	Contextual word embeddings  ,  ,  have been successfully applied to various NLP tasks, including named entity recognition, document classification, and textual entailment. The multilingual version of BERT (which is trained on Wikipedia articles from 100 languages and equipped with a 110,000 shared wordpiece vocabulary) has also demonstrated the ability to perform `zero-resource' cross-lingual classification on the XNLI dataset  . Specifically, when multilingual BERT is finetuned for XNLI with English data alone, the model also gains the ability to handle the same task in other languages. We believe that this zero-resource transfer learning can be extended to other multilingual datasets. In this work, we explore BERT's zero-resource performance on the multilingual MLDoc classification and CoNLL 2002/2003 NER tasks. We find that the baseline zero-resource performance of BERT exceeds the results reported in other work, even though cross-lingual resources (e.g. parallel text, dictionaries, etc.) are not used during BERT pretraining or finetuning. We apply adversarial learning to further improve upon this baseline, achieving state-of-the-art zero-resource results.

CoNLL NER Results	We finetuned BERT on the English portion of the CoNLL 2002/2003 NER corpus  . We followed the text preprocessing in  . We used a learning rate of INLINEFORM0 for the task loss, INLINEFORM1 for the generator loss and INLINEFORM2 for the discriminator loss. In Table TABREF19 , we report the F1 scores for all of the CoNLL NER languages. When combined with adversarial learning, the BERT cross-lingual F1 scores increased for German over the non-adversarial baseline, and the scores remained largely the same for Spanish and Dutch. Regardless, the BERT zero-resource performance far exceeds the results published in previous work.   and  do use some cross-lingual resources (like bilingual dictionaries) in their experiments, but it appears that BERT with multilingual pretraining performs better, even though it does not have access to cross-lingual information.

MLDoc Classification Results	We finetuned BERT on the English portion of the MLDoc corpus  . The MLDoc task is a 4-class classification problem, where the data is a class-balanced subset of the Reuters News RCV1 and RCV2 datasets. We used the english.train.1000 dataset for the classification loss, which contains 1000 labeled documents. For language-adversarial training, we used the text portion of german.train.10000, french.train.10000, etc. without the labels. We used a learning rate of INLINEFORM0 for the task loss, INLINEFORM1 for the generator loss and INLINEFORM2 for the discriminator loss. In Table TABREF13 , we report the classification accuracy for all of the languages in MLDoc. Generally, adversarial training improves the accuracy across all languages, and the improvement is sometimes dramatic versus the BERT non-adversarial baseline. In Figure FIGREF15 , we plot the zero-resource German and Japanese test set accuracy as a function of the number of steps taken, with and without adversarial training.

For language-adversarial training for the classification task, we have 3 loss functions: the task-specific loss INLINEFORM0 , the generator loss INLINEFORM1 , and the discriminator loss INLINEFORM2 : INLINEFORM3 INLINEFORM4  where INLINEFORM0 is the number of classes for the task, INLINEFORM1 (dim: INLINEFORM2 ) is the task-specific prediction, INLINEFORM3 (dim: scalar) is the probability that the input is in English, INLINEFORM4 (dim: INLINEFORM5 ) is the mean-pooled BERT output embedding for the input word-pieces INLINEFORM6 , INLINEFORM7 is the BERT parameters, INLINEFORM8 (dim: INLINEFORM9 , INLINEFORM10 , INLINEFORM11 , scalar) are the output projections for the task-specific loss and discriminator respectively, INLINEFORM12 (dim: INLINEFORM13 ) is the one-hot vector representation for the task label and INLINEFORM14 (dim: scalar) is the binary label for the adversarial task (i.e. 1 or 0 for English or non-English).

Acknowledgments	We would like to thank Jiateng Xie, Julian Salazar and Faisal Ladhak for the helpful comments and discussions.

The plot shows that the variation in the test accuracy is reduced with adversarial training, which suggests that the cross-lingual performance is more consistent when adversarial training is applied. (We note that the batch size and learning rates are the same for all the languages in MLDoc, so the variation seen in Figure FIGREF15 are not affected by those factors.)

We report results based on the average of 4 training runs. [ht] Pseudocode for adversarial training on the multilingual text classification task. The batch size is set at 1 for clarity. The parameter subsets are INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 . Input: pre-trained BERT model INLINEFORM3 , data iterators for English and the non-English language INLINEFORM4 , learning rates INLINEFORM5 for each loss function, initializations for discriminator output projection INLINEFORM6 , task-specific output projection INLINEFORM7 , and BERT parameters INLINEFORM8 [1] not converged INLINEFORM9 get English text and task-specific labels INLINEFORM10 INLINEFORM11 compute the prediction for the task INLINEFORM12 compute task-specific loss INLINEFORM13 update model based on task-specific loss INLINEFORM14 get non-English and English text INLINEFORM15 INLINEFORM16 discriminator prediction on non-English text INLINEFORM17 discriminator prediction on English text INLINEFORM18 compute discriminator loss INLINEFORM19 update model based on discriminator loss INLINEFORM20 INLINEFORM21 INLINEFORM22 INLINEFORM23 INLINEFORM24 compute generator loss INLINEFORM25 update model based on generator loss

Self-training Techniques	Self-training, where an initial model is used to generate labels on an unlabeled corpus for the purpose of domain or cross-lingual adaptation, was studied in the context of text classification  and parsing  ,  . A similar idea based on expectation-maximization, where the unobserved label is treated as a latent variable, has also been applied to cross-lingual text classification in  .

Adversarial Learning	Language-adversarial training  was proposed for generating bilingual dictionaries without parallel data. This idea was extended to zero-resource cross-lingual tasks in NER  ,  and text classification  , where we would expect that language-adversarial techniques induce features that are language-independent.

Question: How do they quantify alignment between the embeddings of a document and its translation?
Answer: median cosine similarity