Question: How do the authors assume to allow a fair comparison?
Evidence: In this section, we provide comparative results on our new Multilingual Document Classification Corpus. Since the initial work by BIBREF0 many alternative approaches to cross-lingual document classification have been developed. We will encourage the respective authors to evaluate their systems on MLDoc. We believe that a large variety of transfer language pairs will give valuable insights on the performance of the various approaches.
In this paper, we propose initial strong baselines which represent two complementary directions of research: one based on the aggregation of multilingual word embeddings, and another one, which directly learns multilingual sentence representations. Details on each approach are given in section "Multilingual word representations" and "Multilingual sentence representations" respectively. In contrast to previous works on cross-lingual document classification with RVC2, we explore training the classifier on all languages and transfer it to all others, ie. we do not limit our study to the transfer between English and a foreign language.
One can envision several ways to define cross-lingual document classification, in function of the resources which are used in the source and transfer language (see Table 3 ). The first scheme assumes that we have no resources in the transfer language at all, neither labeled nor unlabeled. We will name this case “zero-shot cross-lingual document classification”. To simplify the presentation, we will assume that we transfer from English to German. The training and evaluation protocol is as follows. First, train a classifier using resources in the source language only, eg. the training and development corpus are in English. All meta parameters and model choices are performed using the English development corpus. Once the best performing model is selected, it is applied to the transfer language, eg. the German test set. Since no resources of the transfer language are used, the same system can be applied to many different transfer languages. This type of cross-lingual document classification needs a very strong multilingual representation since no knowledge on the target language was used during the development of the classifier.
In a second class of cross-lingual document classification, we may aim in improving the transfer performance by using a limited amount of resources in the target language. In the framework of the proposed MLDoc we will use the development corpus of target language for model selection. We will name this method “targeted cross-lingual document classification” since the system is tailored to one particular transfer language. It is unlikely that this system will perform well on other languages than the ones used for training or model selection.
If the goal is to build one document classification system for many languages, it may be interesting to use already several languages during training and model selection. To allow a fair comparison, we will assume that these multilingual resources have the same size than the ones used for zero-shot or targeted cross-language document classification, e.g. a training set composed of five languages with 200 examples each. This type of training is not a cross-lingual approach any more. Consequently, we will refer to this method as “joint multilingual document classification”.
To allow a fair comparison, we will assume that these multilingual resources have the same size than the ones used for zero-shot or targeted cross-language document classification, e.g. a training set composed of five languages with 200 examples each. This type of training is not a cross-lingual approach any more. Consequently, we will refer to this method as “joint multilingual document classification”.
Answer: The authors assume that these multilingual resources have the same size than the ones used for zero-shot or targeted cross-language document classification.