Conclusion	We have defined a new evaluation framework for cross-lingual document classification in eight languages. This corpus largely extends previous corpora which were also based on the Reuters Corpus Volume 2, but mainly considered the transfer between English and German. We also provide detailed baseline results using two competitive approaches (multilingual word and sentence embeddings, respectively), for cross-lingual document classification between all eight languages. This new evaluation framework is freely available at https://github.com/facebookresearch/MLDoc.

Baseline results	In this section, we provide comparative results on our new Multilingual Document Classification Corpus. Since the initial work by  many alternative approaches to cross-lingual document classification have been developed. We will encourage the respective authors to evaluate their systems on MLDoc. We believe that a large variety of transfer language pairs will give valuable insights on the performance of the various approaches. In this paper, we propose initial strong baselines which represent two complementary directions of research: one based on the aggregation of multilingual word embeddings, and another one, which directly learns multilingual sentence representations. Details on each approach are given in section "Multilingual word representations" and "Multilingual sentence representations" respectively. In contrast to previous works on cross-lingual document classification with RVC2, we explore training the classifier on all languages and transfer it to all others, ie. we do not limit our study to the transfer between English and a foreign language. One can envision several ways to define cross-lingual document classification, in function of the resources which are used in the source and transfer language (see Table 3 ). The first scheme assumes that we have no resources in the transfer language at all, neither labeled nor unlabeled.

A Corpus for Multilingual Document Classification in Eight Languages	Cross-lingual document classification aims at training a document classifier on resources in one language and transferring it to a different language without any additional resources. Several approaches have been proposed in the literature and the current best practice is to evaluate them on a subset of the Reuters Corpus Volume 2. However, this subset covers only few languages (English, German, French and Spanish) and almost all published works focus on the the transfer between English and German. In addition, we have observed that the class prior distributions differ significantly between the languages. We argue that this complicates the evaluation of the multilinguality. In this paper, we propose a new subset of the Reuters corpus with balanced class priors for eight languages. By adding Italian, Russian, Japanese and Chinese, we cover languages which are very different with respect to syntax, morphology, etc. We provide strong baselines for all language transfer directions using multilingual word and sentence embeddings respectively. Our goal is to offer a freely available framework to evaluate cross-lingual document classification, and we hope to foster by these means, research in this important area.

Corpus description	The Reuters Corpus Volume 2  , in short RCV2, is a multilingual corpus with a collection of 487,000 news stories. Each news story was manually classified into four hierarchical groups: CCAT (Corporate/Industrial), ECAT (Economics), GCAT (Government/Social) and MCAT (Markets). Topic codes were assigned to capture the major subject of the news story. The entire corpus covers thirteen languages, i.e. Dutch, French, German, Chinese, Japanese, Russian, Portuguese, Spanish, Latin American Spanish, Italian, Danish, Norwegian, and Swedish, written by local reporters in each language. The news stories are not parallel. Single-label stories, i.e. those labeled with only one topic out of the four top categories, are often used for evaluations. However, the class distributions vary significantly across all the thirteen languages (see Table 1 ). Therefore, using random samples to extract evaluation corpora may lead to very imbalanced test sets, i.e. undesired and misleading variability among the languages when the main focus is to evaluate cross-lingual transfer.

Multilingual document classification	In this work, we propose a new evaluation framework for highly multilingual document classification which significantly extends the current state. We continue to use Reuters Corpus Volume 2, but based on the above mentioned limitations of the current subset of RCV2, we propose new tasks for cross-lingual document classification. The design choices are as follow: Uniform class coverage: we sample from RCV2 the same number of examples for each class and language; Split the data into train, development and test corpus: for each languages, we provide training data of different sizes (1k, 2k, 5k and 10k stories), a development (1k) and a test corpus (4k); Support more languages: German (DE), English (EN), Spanish (ES), French (FR), Italian (IT), Japanese (JA), Russian (RU) and Chinese (ZH). Reference baseline results are available for all languages. Most works in the literature use only 1 000 examples to train the document classifier. To invest the impact of more training data, we also provide training corpora of 2 000, 5 000 and 10 000 documents.

Introduction	There are many tasks in natural language processing which require the classification of sentences or longer paragraphs into a set of predefined categories. Typical applications are for instance topic identification (e.g. sports, news, $\ldots $ ) or product reviews (positive or negative). There is a large body of research on approaches for document classification. An important aspect to compare these different approaches is the availability of high quality corpora to train and evaluate them. Unfortunately, most of these evaluation tasks focus on the English language only, while there is an ever increasing need to perform document classification in many other languages. One could of course collect and label training data for other languages, but this would be costly and time consuming. An interesting alternative is “cross-lingual document classification”. The underlying idea is to use a representation of the words or whole documents which is independent of the language. By these means, a classifier trained on one language can be transferred to a different one, without the need of resources in that transfer language. Ideally, the performance obtained by cross-lingual transfer should be as close as possible to training the entire system on language specific resources.

Multilingual sentence representations	A second direction of research is to directly learn multilingual sentence representations. In this paper, we evaluate a recently proposed technique to learn joint multilingual sentence representations  . The underlying idea is to use multiple sequence encoders and decoders and to train them with aligned corpora from the machine translation community. The goal is that all encoders share the same sentence representation, i.e. we map all languages into one common space. A detailed description of this approach can be found in  . We have developed two versions of the system: one trained on the Europarl corpus  to cover the languages English, German, French, Spanish and Italian, and another one trained on the United Nations corpus  which allows to learn a joint sentence embedding for English, French, Spanish, Russian and Chinese. We use a one hidden-layer MLP as classifier. For comparison, we have evaluated its performance on the original subset of RCV2 as used in previous publications on cross-lingual document classification: we are able to outperform the current state-of-the-art in three out of six transfer directions.

Multilingual word representations	Several works have been proposed to learn multilingual word embeddings, which are then combined to perform cross-lingual document classifications. These word embeddings are trained on either word alignments or sentence-aligned parallel corpora. To provide reproducible benchmark results, we use MultiCCA word embeddings published by  . There are multiple ways to combine these word embeddings for classification. We train a simple one-layer convolutional neural network (CNN) on top of the word embeddings, which has shown to perform well on text classification tasks regardless of training data size  . Specifically, convolutional filters are applied to windows of word embeddings, with a max-over-time pooling on top of them. We freeze the multilingual word embeddings while only training the classifier. Hyper-parameters such as convolutional output dimension, window sizes are done by grid search over the Dev set of the same language as the train set.

Cross-lingual document classification	A subset of the English and German sections of RCV2 was defined by  to evaluate cross-lingual document classification. This subset was used in several follow-up works and many comparative results are available for the transfer between German and English.  extended the use of RCV2 for cross-lingual document classification to the French and Spanish language (transfer from and to English). An analysis of these evaluation corpora has shown that the class prior distributions vary significantly between the classes (see Table 2 ). For German and English, more than 80% of the examples in the test set belong to the classes GCAT and MCAT and at most 2% to the class CCAT. These class prior distributions are very different for French and Spanish: the class CCAT is quite frequent with 21% and 15% of the French and Spanish test set respectively. One may of course argue that variability in the class prior distribution is typical for real-world problems, but this shifts the focus from a high quality cross-lingual transfer to “tricks” for how to best handle the class imbalance.

We will name this case “zero-shot cross-lingual document classification”. To simplify the presentation, we will assume that we transfer from English to German. The training and evaluation protocol is as follows. First, train a classifier using resources in the source language only, eg. the training and development corpus are in English. All meta parameters and model choices are performed using the English development corpus. Once the best performing model is selected, it is applied to the transfer language, eg. the German test set. Since no resources of the transfer language are used, the same system can be applied to many different transfer languages. This type of cross-lingual document classification needs a very strong multilingual representation since no knowledge on the target language was used during the development of the classifier. In a second class of cross-lingual document classification, we may aim in improving the transfer performance by using a limited amount of resources in the target language. In the framework of the proposed MLDoc we will use the development corpus of target language for model selection. We will name this method “targeted cross-lingual document classification” since the system is tailored to one particular transfer language.

Such a task was first proposed by  using the Reuters Corpus Volume 2. The aim was to first train a classifier on English and then to transfer it to German, and vice versa. An extension to the transfer between English and French and Spanish respectively was proposed by  . However, only few comparative results are available for these transfer directions. The contributions of this work are as follows. We extend previous works and use the data in the Reuters Corpus Volume 2 to define new cross-lingual document classification tasks for eight very different languages, namely English, French, Spanish, Italian, German, Russian, Chinese and Japanese. For each language, we define a train, development and test corpus. We also provide strong reference results for all transfer directions between the eight languages, e.g. not limited to the transfer between a foreign language and English. We compare two approaches, based either on multilingual word or sentence embeddings respectively. By these means, we hope to define a clear evaluation environment for highly multilingual document classification.

The development corpus for each language is composed of 1 000 and the test set of 4 000 documents respectively. All have uniform class distributions. An important aspect of this work is to provide a framework to study and evaluate cross-lingual document classification for many language pairs. In that spirit, we will name this corpus “Multilingual Document Classification Corpus”, abbreviated as MLDoc. The full Reuters Corpus Volume 2 has a special license and we can not distribute it ourselves. Instead, we provide tools to extract all the subsets of MLDoc at https://github.com/facebookresearch/MLDoc.

It is unlikely that this system will perform well on other languages than the ones used for training or model selection. If the goal is to build one document classification system for many languages, it may be interesting to use already several languages during training and model selection. To allow a fair comparison, we will assume that these multilingual resources have the same size than the ones used for zero-shot or targeted cross-language document classification, e.g. a training set composed of five languages with 200 examples each. This type of training is not a cross-lingual approach any more. Consequently, we will refer to this method as “joint multilingual document classification”.

Joint multilingual document classification	The classification accuracies for joint multilingual training are given in Table 6 . We use a multilingual train and Dev corpus composed of 200 examples of each of the five languages. One could argue that the data collection and annotation cost for such a corpus would be the same than producing a corpus of the same size in one language only. This leads to important improvement for all languages, in comparison to zero-shot or targeted transfer learning.

Zero-shot cross-lingual document classification	The classification accuracy for zero-shot transfer on the test set of our Multilingual Document Classification Corpus are summarized in Table 4 . The classifiers based on the MultiCCA embeddings perform very well on the development corpus (accuracies close or exceeding 90%). The system trained on English also achieves excellent results when transfered to a different languages, it scores best for three out of seven languages (DE, IT and ZH). However, the transfer accuracies are quite low when training the classifiers on other languages than English, in particular for Russian, Chinese and Japanese. The systems using multilingual sentence embeddings seem to be overall more robust and less language specific. They score best for four out of seven languages (EN, ES, FR and RU). Training on German or French actually leads to better transfer performance than training on English. Cross-lingual transfer between very different languages like Chinese and Russian also achieves remarkable results.

Targeted cross-lingual document classification	The classification accuracy for targeted transfer are summarized in Table 5 . Due to space constraints, we provide only the results for multilingual sentence embeddings and five target languages. Not surprisingly, targeting the classifier to the transfer language can lead to important improvements, in particular when training on Italian.

Indeed, in previous research the transfer between English and German achieves accuracies higher than 90%, while the performance is below 80% for EN/FR or even 70% EN/ES. We have seen experimental evidence that these important differences are likely to be caused by the discrepancy in the class priors of the test sets.

Question: Which baselines were they used for evaluation?
Answer: aggregation of multilingual word embeddings, multilingual sentence representations