How Language-Neutral is Multilingual BERT?	Multilingual BERT (mBERT) provides sentence representations for 104 languages, which are useful for many multi-lingual tasks. Previous work probed the cross-linguality of mBERT using zero-shot transfer learning on morphological and syntactic tasks. We instead focus on the semantic properties of mBERT. We show that mBERT representations can be split into a language-specific component and a language-neutral component, and that the language-neutral component is sufficiently general in terms of modeling semantics to allow high-accuracy word-alignment and sentence retrieval but is not yet good enough for the more difficult task of MT quality estimation. Our work presents interesting challenges which must be solved to build better language-neutral representations, particularly for tasks requiring linguistic transfer of semantics.

Probing Tasks ::: Language Identification.	With a representation that captures all phenomena in a language-neutral way, it should be difficult to determine what language the sentence is written in. Unlike other tasks, language identification does require fitting a classifier. We train a linear classifier on top of a sentence representation to try to classify the language of the sentence.

Our results show that the mBERT representations, even after language-agnostic fine-tuning, are not very language-neutral. However, the identity of the language can be approximated as a constant shift in the representation space. An even higher language-neutrality can still be achieved by a linear projection fitted on a small amount of parallel data. Finally, we present attempts to strengthen the language-neutral component via fine-tuning: first, for multi-lingual syntactic and morphological analysis; second, towards language identity removal via a adversarial classifier.

Related Work	Since the publication of mBERT , many positive experimental results were published.  reached impressive results in zero-shot dependency parsing. However, the representation used for the parser was a bilingual projection of the contextual embeddings based on word-alignment trained on parallel data.  recently examined the cross-lingual properties of mBERT on zero-shot NER and part-of-speech (POS) tagging but the success of zero-shot transfer strongly depends on how typologically similar the languages are. Similarly,  trained good multilingual models for POS tagging, NER, and XNLI, but struggled to achieve good results in the zero-shot setup.  assessed mBERT on cross-lingual sentence retrieval between three language pairs. They observed that if they subtract the average difference between the embeddings from the target language representation, the retrieval accuracy significantly increases. We systematically study this idea in the later sections. Many experiments show , ,  that downstream task models can extract relevant features from the multilingual representations. But these results do not directly show language-neutrality, i.e., to what extent are similar phenomena are represented similarly across languages.

Introduction	Multilingual BERT (mBERT; ) is gaining popularity as a contextual representation for various multilingual tasks, such as dependency parsing , , cross-lingual natural language inference (XNLI) or named-entity recognition (NER) , , .  present an exploratory paper showing that mBERT can be used cross-lingually for zero-shot transfer in morphological and syntactic tasks, at least for typologically similar languages. They also study an interesting semantic task, sentence-retrieval, with promising initial results. Their work leaves many open questions in terms of how good the cross-lingual mBERT representation is for semantics, motivating our work. In this paper, we directly assess the semantic cross-lingual properties of mBERT. To avoid methodological issues with zero-shot transfer (possible language overfitting, hyper-parameter tuning), we selected tasks that only involve a direct comparison of the representations: cross-lingual sentence retrieval, word alignment, and machine translation quality estimation (MT QE). Additionally, we explore how the language is represented in the embeddings by training language identification classifiers and assessing how the representation similarity corresponds to phylogenetic language families.

The models can obtain the task-specific information based on the knowledge of the language, which (as we show later) can be easily identified. Our choice of evaluation tasks eliminates this risk by directly comparing the representations. Limited success in zero-shot setups and the need for explicit bilingual projection in order to work well , ,  also shows limited language neutrality of mBERT.

Fine-tuning mBERT ::: lng-free	In this experiment, we try to make the representations more language-neutral by removing the language identity from the model using an adversarial approach. We continue training mBERT in a multi-task learning setup with the masked LM objective with the same sampling procedure  jointly with adversarial language ID classifiers . For each layer, we train one classifier for the [cls] token and one for the mean-pooled hidden states with the gradient reversal layer  between mBERT and the classifier. The results reveal that the adversarial removal of language information succeeds in dramatically decreasing the accuracy of the language identification classifier; the effect is strongest in deeper layers for which the standard mBERT tend to perform better (see Figure FIGREF22). However, other tasksare not affected by the adversarial fine-tuning.

Probing Tasks ::: MT Quality Estimation.	MT QE assesses the quality of an MT system output without having access to a reference translation. The standard evaluation metric is the correlation with the Human-targeted Translation Error Rate which is the number of edit operations a human translator would need to do to correct the system output. This is a more challenging task than the two previous ones because it requires capturing more fine-grained differences in meaning. We evaluate how cosine distance of the representation of the source sentence and of the MT output reflects the translation quality. In addition to plain and centered representations, we also test trained bilingual projection, and a fully supervised regression trained on training data.

Results ::: MT Quality Estimation.	Qualitative results of MT QE are tabulated in Table TABREF18. Unlike sentence retrieval, QE is more sensitive to subtle differences between sentences. Measuring the distance of the non-centered sentence vectors does not correlate with translation quality at all. Centering or explicit projection only leads to a mild correlation, much lower than a supervisedly trained regression;and even better performance is possible . The results show that the linear projection between the representations only captures a rough semantic correspondence, which does not seem to be sufficient for QE, where the most indicative feature appears to be sentence complexity.

Probing Tasks	We employ five probing tasks to evaluate the language neutrality of the representations.

Fine-tuning mBERT	We also considered model fine-tuning towards stronger language neutrality. We evaluate two fine-tuned versions of mBERT: UDify, tuned for a multi-lingual dependency parser, and lng-free, tuned to jettison the language-specific information from the representations.

Results ::: Parallel Sentence Retrieval.	Results in Table TABREF12 reveal that the representation centering dramatically improves the retrieval accuracy, showing that it makes the representations more language-neutral. However, an explicitly learned projection of the representations leads to a much greater improvement, reaching a close-to-perfect accuracy, even though the projection was fitted on relatively small parallel data. The accuracy is higher for mean-pooled states than for the [cls] embedding and varies according to the layer of mBERT used (see Figure FIGREF13).

Centering mBERT Representations	Following , we hypothesize that a sentence representation in mBERT is composed of a language-specific component, which identifies the language of the sentence, and a language-neutral component, which captures the meaning of the sentence in a language-independent way. We assume that the language-specific component is similar across all sentences in the language. We thus try to remove the language-specific information from the representations by centering the representations of sentences in each language so that their average lies at the origin of the vector space. We do this by estimating the language centroid as the mean of the mBERT representations for a set of sentences in that language and subtracting the language centroid from the contextual embeddings. We then analyze the semantic properties of both the original and the centered representations using a range of probing tasks. For all tasks, we test all layers of the model. For tasks utilizing a single-vector sentence representation, we test both the vector corresponding to the [cls] token and mean-pooled states.

Probing Tasks ::: Language Similarity.	Experiments with POS tagging  suggest that similar languages tend to get similar representations on average. We quantify that observation by measuring how languages tend to cluster by the language families using V-measure over hierarchical clustering of the language centeroid .

Results ::: Language Identification.	Table TABREF7 shows that centering the sentence representations considerably decreases the accuracy of language identification, especially in the case of mean-pooled embeddings. This indicates that the proposed centering procedure does indeed remove the language-specific information to a great extent.

Conclusions	Using a set of semantically oriented tasks that require explicit semantic cross-lingual representations, we showed that mBERT contextual embeddings do not represent similar semantic phenomena similarly and therefore they are not directly usable for zero-shot cross-lingual tasks. Contextual embeddings of mBERT capture similarities between languages and cluster the languages by their families. Neither cross-lingual fine-tuning nor adversarial language identity removal breaks this property. A part of language information is encoded by the position in the embedding space, thus a certain degree of cross-linguality can be achieved by centering the representations for each language. Exploiting this property allows a good cross-lingual sentence retrieval performance and bilingual word alignment (which is invariant to the shift). A good cross-lingual representation can be achieved by fitting a supervised projection on a small parallel corpus.

Fine-tuning mBERT ::: UDify	The UDify model  uses mBERT to train a single model for dependency parsing and morphological analysis of 75 languages. During the parser training, mBERT is fine-tuned, which improves the parser accuracy. Results on zero-shot parsing suggest that the fine-tuning leads to more cross-lingual representations with respect to morphology and syntax. However, our analyses show that fine-tuning mBERT for multilingual dependency parsing does not remove the language identity information from the representations and actually makes the representations less semantically cross-lingual.

Probing Tasks ::: Parallel Sentence Retrieval.	For each sentence in a multi-parallel corpus, we compute the cosine distance of its representation with representations of all sentences on the parallel side of the corpus and select the sentence with the smallest distance. Besides the plain and centered [cls] and mean-pooled representations, we evaluate explicit projection into the “English space”. For each language, we fit a linear regression projecting the representations into English representation space using a small set of parallel sentences.

Results ::: Language Similarity.	Figure FIGREF9 is a tSNE plot  of the language centroids, showing that the similarity of the centroids tends to correspond to the similarity of the languages. Table TABREF10 confirms that the hierarchical clustering of the language centroids mostly corresponds to the language families.

Probing Tasks ::: Word Alignment.	While sentence retrieval could be done with keyword spotting, computing bilingual alignment requires resolving detailed correspondence on the word level. We find the word alignment as a minimum weighted edge cover of a bipartite graph. The graph connects the tokens of the sentences in the two languages and edges between them are weighted with the cosine distance of the token representation. Tokens that get split into multiple subwords are represented using the average of the embeddings of the subwords. Note that this algorithm is invariant to representation centering which would only change the edge weights by a constant offset. We evaluate the alignment using the F$_1$ score over both sure and possible alignment links in a manually aligned gold standard.

Experimental Setup	We use a pre-trained mBERT model that was made public with the BERT release. The model dimension is 768, hidden layer dimension 3072, self-attention uses 12 heads, the model has 12 layers. It uses a vocabulary of 120k wordpieces that is shared for all languages. To train the language identification classifier, for each of the BERT languages we randomly selected 110k sentences of at least 20 characters from Wikipedia, and keep 5k for validation and 5k for testing for each language. The training data are also used for estimating the language centroids. For parallel sentence retrieval, we use a multi-parallel corpus of test data from the WMT14 evaluation campaign  with 3,000 sentences in Czech, English, French, German, Hindi, and Russian. The linear projection experiment uses the WMT14 development data. We use manually annotated word alignment datasets to evaluate word alignment between English on one side and Czech (2.5k sent.; ), Swedish (192 sent.; ), German (508 sent.), French (447 sent.; ) and Romanian (248 sent.; ) on the other side.

Results ::: Word Alignment.	Table TABREF15 shows that word-alignment based on mBERT representations surpasses the outputs of the standard FastAlign tool even if it was provided large parallel corpus. This suggests that word-level semantics are well captured by mBERT contextual embeddings. For this task, learning an explicit projection had a negligible effect on the performance.

We compare the results with FastAlign  that was provided with 1M additional parallel sentences from ParaCrawl  in addition to the test data. For MT QE, we use English-German data provided for the WMT19 QE Shared Task  consisting training and test data with source senteces, their automatic translations, and manually corrections.

Question: What challenges this work presents that must be solved to build better language-neutral representations?
Answer: contextual embeddings do not represent similar semantic phenomena similarly and therefore they are not directly usable for zero-shot cross-lingual tasks