A Corpus for Multilingual Document Classification in Eight Languages

Abstract: Cross-lingual document classification aims at training a document classifier on resources in one language and transferring it to a different language without any additional resources. Several approaches have been proposed in the literature and the current best practice is to evaluate them on a subset of the Reuters Corpus Volume 2. However, this subset covers only few languages (English, German, French and Spanish) and almost all published works focus on the the transfer between English and German. In addition, we have observed that the class prior distributions differ significantly between the languages. We argue that this complicates the evaluation of the multilinguality. In this paper, we propose a new subset of the Reuters corpus with balanced class priors for eight languages. By adding Italian, Russian, Japanese and Chinese, we cover languages which are very different with respect to syntax, morphology, etc. We provide strong baselines for all language transfer directions using multilingual word and sentence embeddings respectively. Our goal is to offer a freely available framework to evaluate cross-lingual document classification, and we hope to foster by these means, research in this important area.

There are many tasks in natural language processing which require the classification of sentences or longer paragraphs into a set of predefined categories. Typical applications are for instance topic identification (e.g. sports, news, $\ldots $ ) or product reviews (positive or negative). There is a large body of research on approaches for document classification. An important aspect to compare these different approaches is the availability of high quality corpora to train and evaluate them. Unfortunately, most of these evaluation tasks focus on the English language only, while there is an ever increasing need to perform document classification in many other languages. One could of course collect and label training data for other languages, but this would be costly and time consuming. An interesting alternative is “cross-lingual document classification”. The underlying idea is to use a representation of the words or whole documents which is independent of the language. By these means, a classifier trained on one language can be transferred to a different one, without the need of resources in that transfer language. Ideally, the performance obtained by cross-lingual transfer should be as close as possible to training the entire system on language specific resources. Such a task was first proposed by BIBREF0 using the Reuters Corpus Volume 2. The aim was to first train a classifier on English and then to transfer it to German, and vice versa. An extension to the transfer between English and French and Spanish respectively was proposed by BIBREF1 . However, only few comparative results are available for these transfer directions.
The contributions of this work are as follows. We extend previous works and use the data in the Reuters Corpus Volume 2 to define new cross-lingual document classification tasks for eight very different languages, namely English, French, Spanish, Italian, German, Russian, Chinese and Japanese. For each language, we define a train, development and test corpus. We also provide strong reference results for all transfer directions between the eight languages, e.g. not limited to the transfer between a foreign language and English. We compare two approaches, based either on multilingual word or sentence embeddings respectively. By these means, we hope to define a clear evaluation environment for highly multilingual document classification.
The Reuters Corpus Volume 2 BIBREF2 , in short RCV2, is a multilingual corpus with a collection of 487,000 news stories. Each news story was manually classified into four hierarchical groups: CCAT (Corporate/Industrial), ECAT (Economics), GCAT (Government/Social) and MCAT (Markets). Topic codes were assigned to capture the major subject of the news story. The entire corpus covers thirteen languages, i.e. Dutch, French, German, Chinese, Japanese, Russian, Portuguese, Spanish, Latin American Spanish, Italian, Danish, Norwegian, and Swedish, written by local reporters in each language. The news stories are not parallel. Single-label stories, i.e. those labeled with only one topic out of the four top categories, are often used for evaluations. However, the class distributions vary significantly across all the thirteen languages (see Table 1 ). Therefore, using random samples to extract evaluation corpora may lead to very imbalanced test sets, i.e. undesired and misleading variability among the languages when the main focus is to evaluate cross-lingual transfer.
A subset of the English and German sections of RCV2 was defined by BIBREF0 to evaluate cross-lingual document classification. This subset was used in several follow-up works and many comparative results are available for the transfer between German and English. BIBREF1 extended the use of RCV2 for cross-lingual document classification to the French and Spanish language (transfer from and to English). An analysis of these evaluation corpora has shown that the class prior distributions vary significantly between the classes (see Table 2 ). For German and English, more than 80% of the examples in the test set belong to the classes GCAT and MCAT and at most 2% to the class CCAT. These class prior distributions are very different for French and Spanish: the class CCAT is quite frequent with 21% and 15% of the French and Spanish test set respectively. One may of course argue that variability in the class prior distribution is typical for real-world problems, but this shifts the focus from a high quality cross-lingual transfer to “tricks” for how to best handle the class imbalance. Indeed, in previous research the transfer between English and German achieves accuracies higher than 90%, while the performance is below 80% for EN/FR or even 70% EN/ES. We have seen experimental evidence that these important differences are likely to be caused by the discrepancy in the class priors of the test sets.
In this work, we propose a new evaluation framework for highly multilingual document classification which significantly extends the current state. We continue to use Reuters Corpus Volume 2, but based on the above mentioned limitations of the current subset of RCV2, we propose new tasks for cross-lingual document classification. The design choices are as follow:
Uniform class coverage: we sample from RCV2 the same number of examples for each class and language;
Split the data into train, development and test corpus: for each languages, we provide training data of different sizes (1k, 2k, 5k and 10k stories), a development (1k) and a test corpus (4k);
Support more languages: German (DE), English (EN), Spanish (ES), French (FR), Italian (IT), Japanese (JA), Russian (RU) and Chinese (ZH). Reference baseline results are available for all languages.
Most works in the literature use only 1 000 examples to train the document classifier. To invest the impact of more training data, we also provide training corpora of 2 000, 5 000 and 10 000 documents. The development corpus for each language is composed of 1 000 and the test set of 4 000 documents respectively. All have uniform class distributions. An important aspect of this work is to provide a framework to study and evaluate cross-lingual document classification for many language pairs. In that spirit, we will name this corpus “Multilingual Document Classification Corpus”, abbreviated as MLDoc. The full Reuters Corpus Volume 2 has a special license and we can not distribute it ourselves. Instead, we provide tools to extract all the subsets of MLDoc at https://github.com/facebookresearch/MLDoc.
In this section, we provide comparative results on our new Multilingual Document Classification Corpus. Since the initial work by BIBREF0 many alternative approaches to cross-lingual document classification have been developed. We will encourage the respective authors to evaluate their systems on MLDoc. We believe that a large variety of transfer language pairs will give valuable insights on the performance of the various approaches.
In this paper, we propose initial strong baselines which represent two complementary directions of research: one based on the aggregation of multilingual word embeddings, and another one, which directly learns multilingual sentence representations. Details on each approach are given in section "Multilingual word representations" and "Multilingual sentence representations" respectively. In contrast to previous works on cross-lingual document classification with RVC2, we explore training the classifier on all languages and transfer it to all others, ie. we do not limit our study to the transfer between English and a foreign language.
One can envision several ways to define cross-lingual document classification, in function of the resources which are used in the source and transfer language (see Table 3 ). The first scheme assumes that we have no resources in the transfer language at all, neither labeled nor unlabeled. We will name this case “zero-shot cross-lingual document classification”. To simplify the presentation, we will assume that we transfer from English to German. The training and evaluation protocol is as follows. First, train a classifier using resources in the source language only, eg. the training and development corpus are in English. All meta parameters and model choices are performed using the English development corpus. Once the best performing model is selected, it is applied to the transfer language, eg. the German test set. Since no resources of the transfer language are used, the same system can be applied to many different transfer languages. This type of cross-lingual document classification needs a very strong multilingual representation since no knowledge on the target language was used during the development of the classifier.
In a second class of cross-lingual document classification, we may aim in improving the transfer performance by using a limited amount of resources in the target language. In the framework of the proposed MLDoc we will use the development corpus of target language for model selection. We will name this method “targeted cross-lingual document classification” since the system is tailored to one particular transfer language. It is unlikely that this system will perform well on other languages than the ones used for training or model selection.
If the goal is to build one document classification system for many languages, it may be interesting to use already several languages during training and model selection. To allow a fair comparison, we will assume that these multilingual resources have the same size than the ones used for zero-shot or targeted cross-language document classification, e.g. a training set composed of five languages with 200 examples each. This type of training is not a cross-lingual approach any more. Consequently, we will refer to this method as “joint multilingual document classification”.
Several works have been proposed to learn multilingual word embeddings, which are then combined to perform cross-lingual document classifications. These word embeddings are trained on either word alignments or sentence-aligned parallel corpora. To provide reproducible benchmark results, we use MultiCCA word embeddings published by BIBREF3 .
There are multiple ways to combine these word embeddings for classification. We train a simple one-layer convolutional neural network (CNN) on top of the word embeddings, which has shown to perform well on text classification tasks regardless of training data size BIBREF4 . Specifically, convolutional filters are applied to windows of word embeddings, with a max-over-time pooling on top of them. We freeze the multilingual word embeddings while only training the classifier. Hyper-parameters such as convolutional output dimension, window sizes are done by grid search over the Dev set of the same language as the train set.
A second direction of research is to directly learn multilingual sentence representations. In this paper, we evaluate a recently proposed technique to learn joint multilingual sentence representations BIBREF5 . The underlying idea is to use multiple sequence encoders and decoders and to train them with aligned corpora from the machine translation community. The goal is that all encoders share the same sentence representation, i.e. we map all languages into one common space. A detailed description of this approach can be found in BIBREF5 . We have developed two versions of the system: one trained on the Europarl corpus BIBREF6 to cover the languages English, German, French, Spanish and Italian, and another one trained on the United Nations corpus BIBREF7 which allows to learn a joint sentence embedding for English, French, Spanish, Russian and Chinese. We use a one hidden-layer MLP as classifier. For comparison, we have evaluated its performance on the original subset of RCV2 as used in previous publications on cross-lingual document classification: we are able to outperform the current state-of-the-art in three out of six transfer directions.
The classification accuracy for zero-shot transfer on the test set of our Multilingual Document Classification Corpus are summarized in Table 4 . The classifiers based on the MultiCCA embeddings perform very well on the development corpus (accuracies close or exceeding 90%). The system trained on English also achieves excellent results when transfered to a different languages, it scores best for three out of seven languages (DE, IT and ZH). However, the transfer accuracies are quite low when training the classifiers on other languages than English, in particular for Russian, Chinese and Japanese.
The systems using multilingual sentence embeddings seem to be overall more robust and less language specific. They score best for four out of seven languages (EN, ES, FR and RU). Training on German or French actually leads to better transfer performance than training on English. Cross-lingual transfer between very different languages like Chinese and Russian also achieves remarkable results.
The classification accuracy for targeted transfer are summarized in Table 5 . Due to space constraints, we provide only the results for multilingual sentence embeddings and five target languages. Not surprisingly, targeting the classifier to the transfer language can lead to important improvements, in particular when training on Italian.
The classification accuracies for joint multilingual training are given in Table 6 . We use a multilingual train and Dev corpus composed of 200 examples of each of the five languages. One could argue that the data collection and annotation cost for such a corpus would be the same than producing a corpus of the same size in one language only. This leads to important improvement for all languages, in comparison to zero-shot or targeted transfer learning.
We have defined a new evaluation framework for cross-lingual document classification in eight languages. This corpus largely extends previous corpora which were also based on the Reuters Corpus Volume 2, but mainly considered the transfer between English and German. We also provide detailed baseline results using two competitive approaches (multilingual word and sentence embeddings, respectively), for cross-lingual document classification between all eight languages. This new evaluation framework is freely available at https://github.com/facebookresearch/MLDoc.

Figure and table captions:
Table 1: Class distribution of all single-label stories per language of the entire Reuters Corpus Volume 2.
Table 2: Class distribution of the test set of the RCV2 subsets as used in previous publications on cross-lingual document classification.
Table 3: Different schemes of cross- and multilingual document classification.
Table 4: Baseline classification accuracies for zero-shot transfer on the test set of the proposed Multilingual Document Classification Corpus. All classifiers were trained on 1 000 news stories and model selection is performed on the Dev corpus of the training language. The same system is then applied to all test languages. Underlined scores indicate the best result on each transfer language for each group, bold scores the overall best accuracy, and italic ones the second best results.
Table 5: Baseline classification accuracies for targeted transfer on the test set of the proposed MLDoc. All classifiers were trained on 1 000 news stories and model selection is performed on the Dev corpus of the target language. Each entry corresponds to a specifically optimized system.
Table 6: Baseline classification accuracies on the test set of the proposed MLDoc for joint multilingual training. Train and test sets are composed of 200 examples form each of the five languages.

Questions about this paper:
What is the difference in size compare to the previous model?

Answers:
larger