Title: EXPLOITING CROSS-LINGUAL SUBWORD SIMILARI-TIES IN LOW-RESOURCE DOCUMENT CLASSIFICATION

Abstract: Text classification must sometimes be applied in situations with no training data in a target language. However, training data may be available in a related language. We introduce a cross-lingual document classification framework (CACO) between related language pairs. To best use limited training data, our transfer learning scheme exploits cross-lingual subword similarity by jointly training a character-based embedder and a word-based classifier. The embedder derives vector representations for input words from their written forms, and the classifier makes predictions based on the word vectors. We use a joint character representation for both the source language and the target language, which allows the embedder to generalize knowledge about source language words to target language words with similar forms. We propose a multi-task objective that can further improve the model if additional cross-lingual or monolingual resources are available. CACO models trained under low-resource settings rival cross-lingual word embedding models trained under high-resource settings on related language pairs.

1. INTRODUCTION: CLASSIFIERS ACROSS LANGUAGES:

Building an accurate document classifier with traditional supervised learning techniques requires a large labeled corpus. Unfortunately, annotated data is often unavailable (or unreliable) in low-resource languages. Sometimes the need for a low-resource classifier is acute: an earthquake in Haiti,1 unrest in Ukraine, or food shortages in East Africa. Cross-lingual document classification (Klementiev et al., 2012, CLDC) attacks this problem by using annotated dataset from a source language to build classifiers for a target language.

The key challenge of CLDC is to find a shared representation for documents from both languages. Once we have a cross-lingual feature space, we can train a classifier on source language documents and apply it on target language documents. Previous work uses a bilingual lexicon (Wu et al., 2008; Shi et al., 2010; Andrade et al., 2015) , machine translation (Banea et al., 2008; Wan, 2009; Zhou et al., 2016, MT) , topic models (Mimno et al., 2009) , or pre-trained cross-lingual word embeddings (Klementiev et al., 2012; Chen et al., 2016, CLWE) to extract cross-lingual features. However, this may be impossible in low-resource languages, as these methods require some combination of large parallel or comparable text, high-coverage dictionaries, and monolingual corpora.

We propose a new CLDC method for a truly low-resource setting, where unlabeled or parallel data in target language is also limited or unavailable. Our system, Classification Aided by Convergent Orthography (CACO) capitalizes on subword similarities between related language pairs. When the source language and the target language come from the same family, they share cognate words, many of which have similar forms and semantics. For example, "religious" in English, "religieux" in French, and "religioso" in Italian and Spanish share the same meaning.

Previous CLDC methods treat words as atomic symbols and ignore subword patterns. We instead use a bi-level model with two components: a character-based embedder and a word-based classifier. The embedder creates vectors for input words from their character sequences. Using the word embeddings, the classifier then labels the document. We hope the embedder can learn morpho-semantic regularities, while the classifier connects lexical semantics to labels. To allow cross-lingual transfer, we use a single model for both languages, and we share character embeddings between source and target languages. We jointly train the embedder and the classifier on annotated source language documents. The embedder transfers knowledge about source language words to target language words with similar orthographic features.

If we have additional unlabeled data such as a dictionary, pre-trained word embeddings, or parallel text, we can fine-tune the model with multi-task learning. Specifically, we encourage the embedder to produce similar word embeddings for dictionary word pairs, which captures patterns between cognate pairs. We also teach the embedder to MIMICK pre-trained word embeddings in the source language (Pinter et al., 2017) , which exposes the model to more word types. When we have a good reference model in a high resource language, we can train the model to match the output of the reference model on parallel text.

We experiment on CLDC between nine related language pairs on two datasets. CACO can match the accuracy of CLWE-based models without using any target language data, and fine-tuning the embedder with a small amount of additional resources further improves the accuracy of CACO.

2. CLASSIFICATION AIDED BY CONVERGENT ORTHOGRAPHY:

This section introduces our model, CACO, which trains a multilingual document classifier using labeled datasets in a source language S and applies the classifier to a low-resource target language T .

2.1. MODEL ARCHITECTURE:

Our model has a two-level architecture (Figure 1 ) that includes a character-level embedder e and a word-level classifier f . The embedder e takes characters as inputs and produces a word embedding vector. The classifier then computes a label distribution from the embeddings of all input words. Formally, let w be an input document with a sequence of tokens w = w 1 , w 2 , • • • , w n . Our model maps the document w to a distribution over possible labels y in two steps. First, we generate a word embedding v i for each input word w i using the embedder e:

v i = e(w i ).

(1)

We then feed the word embeddings to the classifier f to compute the distribution over labels y:

EQUATION

We can use any sequence model for the embedder e and the classifier f . For our experiments, we use a bidirectional LSTM (Graves & Schmidhuber, 2005 , BI-LSTM) embedder and a deep averaging network (Iyyer et al., 2015, DAN) classifier.

BI-LSTM Embedder: BI-LSTM is a powerful sequence model that captures complex non-local dependencies. Character-level BI-LSTM embedders are successfully used in many natural language processing tasks (Ling et al., 2015a; b; Ballesteros et al., 2015; Lample et al., 2016) . To embed a word w, we pass its character sequence c to a left-to-right LSTM and the reversed character sequence c to a right-to-left LSTM. We concatenate the final hidden states of the two LSTM and apply a linear transformation:

EQUATION

where the functions ---→ LSTM and ← ---LSTM compute the final hidden states of the two LSTMs.

DAN Classifier: A DAN is an unordered model that passes the arithmetic mean of the input word embeddings through a multilayer perceptron and feed the final layer's representation to a softmax layer. We choose DAN because it ignores cross-lingual variations in word orders (i.e., syntax) and thus generalizes well in CLDC. Despite its simplicity, DAN achieves state-of-the-art accuracies on both monolingual and cross-lingual document classification (Iyyer et al., 2015; Chen et al., 2016) .

Let v 1 , v 2 , • • • , v n be the input word embeddings. A DAN uses the average of the word embeddings as the document representation z 0 :

EQUATION

and z 0 is passed through k layers of non-linearity:

EQUATION

c a t s e a t f i s h w 1 w 2 w 3 BI-LSTM Embedder DAN Classifier v 1 v 2 v 3 z 0 = 1 3 (v 1 + v 2 + v 3 ) k layers of non-linearity z k p(y | w) = softmax(W k+1 z k + b k+1 )

Figure 1 : In CACO, each input word w i is mapped to a word vector v i by passing its characters through a BI-LSTM embedder. The average of the word vectors z 0 is then passed through k layers of non-linearity and a final softmax layer to predict the label y.

where i ranges from 1 to k, and g is a non-linear activation function. The final representation z k is passed to a softmax layer to obtain a distribution over the label y,

EQUATION

2.2. CHARACTER-LEVEL CROSS-LINGUAL TRANSFER:

To transfer character-level information across languages, the embedder uses the same character embeddings for both languages. The character-level BI-LSTM vocabulary is the union of the alphabets for the two languages, and the embedder does not differentiate identical characters from different languages. For example, a Spanish "a" has the same character embedding as a French "a". Consequently, the embedder maps words with similar forms from both languages to similar vectors.

If the source language and the target language are lexically similar, the embedder can generalize knowledge learned about source language words to target language words through shared orthographic features. As an example, if the model learns that the Spanish word "religioso" is predictive of label y, the model automatically infers that "religioso" in Italian is also predictive of y, even though the model never sees any Italian text.

2.3. OBJECTIVE:

Our main training objective is supervised document classification. We jointly train the classifier and the embedder to minimize average negative log-likelihood on labeled source language documents S:

EQUATION

where θ is model parameter, and S contains source language examples with words w and label y.

Sometimes we have additional resources for the source or target language. We use them to improve our model with multi-task learning (Collobert et al., 2011) via three auxiliary tasks.

Word Translation (DICT)::

There are many patterns when translating cognate words between related languages. For example, Italian "e" often becomes "ie" when translating into Spanish. "Tempo" (time) in Italian becomes "tiempo" in Spanish, and "concerto" (concert) in Italian becomes "concierto" in Spanish. The embedder can learn these word translation patterns from a bilingual dictionary.

Let D be a bilingual dictionary with a set of word pairs w, w , where w and w are translations of each other. We add a term to our objective to minimize average squared Euclidean distances between the embeddings of translation pairs (Mikolov et al., 2013) :

EQUATION

A bilingual dictionary can also reveal false friends-words from different languages that have similar forms but different meanings. For example, the Italian word "burro" (butter) is "mantequilla" in Spanish, while the Spanish word "burro" (donkey) is "asino" in Italian. However, our current model's joint character embedding space treats "burro" (Spanish) and "burro" (Italian) identically. Therefore, we consider a variant of CACO with language identifiers (in § 4 these variants are denoted by + ). Specifically, the vocabulary of the embedder now includes two special language identifier characters for the two languages. We prepend and append the identifier to each input word. This allows the embedder to differentiate words from different languages, but the added expressiveness might induce overfitting to the dictionary and prevent generalization through orthographic features.

Mimicking Word Embeddings (MIM): Monolingual text classifiers often benefit from initializing embeddings with word vectors pre-trained on large unlabeled corpus (Collobert et al., 2011) . This semi-supervised learning strategy helps the model generalize to word types outside labeled training data. Similarly, our embedder can MIMICK (Pinter et al., 2017; Stratos, 2017) an existing source language word embeddings to learn and transfer useful representations.

Suppose we have a pre-trained source language word embedding matrix X with V rows. The i-th row x i is a vector for the i-th word type w i . We add an objective to minimize the average squared Euclidean distances between the output of the embedder and X:

EQUATION

Parallel Projection: Sometimes we have a reliable reference classifier in another high-resource language S (for example, English). If we have parallel text between S and S , we can use parallel projection (Yarowsky et al., 2001) to add additional training signal. Let P be a set of parallel documents w, w , where w is from source language S, and w is the translation of w in S . We add another objective term to minimize the average Kullback-Leibler divergence between the predictions of our model predictions and the reference model:

EQUATION

where p is the output of the reference classifier (in language S ), and p is the output of CACO. In § 4, we mark models that use parallel projection with a superscript "P".

We train our model on the four tasks jointly. Our final objective is:

L(θ) = L s (θ) classifier +λ d L d (θ)

dictionary +λ e L e (θ)

EQUATION

where the hyperparameters λ d , λ e , and λ p trade off between the four tasks.

3. RELATED WORK:

Previous CLDC methods are typically word-based and rely on one of the following cross-lingual signals to transfer knowledge: large bilingual lexicons (Wu et al., 2008; Shi et al., 2010; Andrade et al., 2015) , MT systems (Banea et al., 2008; Wan, 2009; Zhou et al., 2016) , or cross-lingual word representations (Klementiev et al., 2012; Chen et al., 2016) . Unfortunately, these resources are not available in every language. Recent work proposes unsupervised methods for building CLWE (Artetxe et al., 2018a; Conneau et al., 2018; Zhang et al., 2017a; b) and MT systems (Artetxe et al., 2018b; Lample et al., 2018a; b) without any cross-lingual signal. However, these methods still require large monolingual corpora in the target language, and they might fail when the monolingual corpora for the two languages come from different domains (Søgaard et al., 2018) . In contrast, CACO is much more Our work builds on the success of character-level BI-LSTM embedders in monolingual NLP tasks, including language modeling and part-of-speech tagging (Ling et al., 2015a) , named entity recognition (Lample et al., 2016) , and dependency parsing (Ballesteros et al., 2015) . Character-level embedders can generate useful representations for rare and unseen words, especially for morphologically rich languages. We extend this to CLDC. Recent work also uses character BI-LSTMs to directly reconstruct pre-trained monolingual word embeddings (Pinter et al., 2017; Stratos, 2017) , which motivates our MIMICK objective.

Cross-lingual transfer at the character-level is successfully used in low-resource paradigm completion (Kann et al., 2017) , morphological tagging (Cotterell & Heigold, 2017) , POS tagging (Kim et al., 2017) , and named entity recognition (Cotterell & Duh, 2017; Lin et al., 2018) , where the authors train a character-level model jointly on a small labeled corpus in target language and a large labeled corpus in source language. Our method is similar in spirit, but we focus on CLDC, where it is less obvious if orthographic features are helpful. Moreover, we introduce a novel multi-task objective to use different types of monolingual and cross-lingual resources; we can match CLWE-based models using little or no target language data.

4. EXPERIMENTS:

We evaluate our method on two CLDC datasets: Reuters multilingual corpora (Lewis et al., 2004, RCV2) and LORELEI language packs (Strassel & Tracey, 2016) . To show the effectiveness of our method, we train CACO models in a low-resource setting and compare two CLWE-based model trained in a high-resource setting and a supervised monolingual model. Our goal is not to beat the 

4.1. CLASSIFICATION DATASET:

Our first dataset is RCV2, a multilingual collection of news stories labeled with topics (Lewis et al., 2004 ).2 Following Klementiev et al. (2012) , we remove documents with multiple topic labels. For each language, we sample 1,500 training documents and 200 test documents with balanced labels. We conduct CLDC experiments between two North Germanic languages, Danish (DA) and Swedish (SV), and three Romance languages, French (FR), Italian (IT), and Spanish (ES).

To test CACO on truly low-resource languages, we build a second CLDC dataset with famine-related documents sampled from Tigrinya (TI) and Amharic (AM) LORELEI language packs (Strassel & Tracey, 2016) . 3 We train models to classify whether the document describes widespread crime or not. 4 The Amharic language pack does not have annotations, so we manually label Amharic sentences based on English reference translations. 5 Our final dataset contains 394 Tigrinya and 370 Amharic documents with balanced labels.

4.2. MODELS:

We experiment with several variants of CACO that use different resources. The SRC model is only trained on labeled source language documents and do not use any unlabeled data. The DICT model is trained on both labeled documents and parallel dictionary with the supervised objective and the word translation objective. The MIM model is jointly trained to label source language documents and mimick the source language part of multiCCA embeddings. The ALL model is trained using both the word translation and the mimick tasks.

All of the above models do not use language identifier characters and therefore cannot distinguish words from different languages with identical forms. For DICT and ALL, we also experiment with their variants that prepend and append a language identifier character to each word. We name these models DICT + and ALL + . We also use parallel projection to provide more classification signals for some models. We mark these models with a superscript "P".

For comparison, we train two word-based DANs with pre-trained CLWE features. We use the multiCCA (mCCA) and multiCluster (mClu) embeddings (Ammar et al., 2016) , which are trained on large corpora with millions of tokens and high-coverage dictionaries with hundreds of thousands of word types. The CLWE-based DANs are strong high-resource models that require large corpora and dictionary for the target language. In contrast, CACO models are trained with few or no target language data in a simulated low-resource setting. Therefore, it is unfair to directly compare the results of CACO and CLWE-based models. However, CACO models are often competitive with CLWE-based DANs in our experiment, demonstrating the effectiveness of our proposed method. We also compare CACO with a lightly-supervised monolingual model SUP,6 a DAN trained on fifty labeled target language documents. Without using any target language supervision, CACO models achieves similar (and sometimes higher) test accuracies as SUP.

Finally, we experiment with a model that combines CACO and CLWE. Specifically, we use pre-trained multiCluster CLWE as additional features for the classifier of a SRC model. The combined model on average significantly outperforms both CACO models and clwe-based DAN, which shows that our method is useful even when we have a high-quality cross-lingual word embedding.

4.3. DICTIONARY AND PARALLEL TEXT:

Some of the CACO models use a dictionary to learn word translation patterns. We train them on the same training dictionary used for learning multiCCA and multiCluster. To simulate the low-resource setting, we sample only 100 translation pairs from the original dictionary for CACO.

The original dictionary contains many word types with identical forms. We remove these pairs when sampling the dictionary for DICT and ALL, because without the language identifiers, the embedder always maps words with the same form to the same vector. However, our pilot experiment shows that removing these pairs could hurt the test accuracy for DICT + and ALL + . The embedder likely overfits the translation pairs by aggressively tuning the character embedder for the language identifiers. As a result, the embedder behaves very differently for the two languages, which prevents cross-lingual knowledge transfer. Therefore, we keep translation pairs with same surface forms when sampling dictionaries for DICT + and ALL + .

The Amharic labeled dataset is very small compared to other languages,7 so we experiment with the parallel projection technique for the Amharic to Tigrinya CLDC experiment using English-Amharic parallel text. We first train a reference English DAN on a large collection of labeled English documents compiled from other LORELEI language packs. We then use the parallel projection objective to train the CACO models to match the output of the English model on 1,200 English-Amharic parallel documents sampled from the Amharic language pack. 8 We do not use parallel projections on other language pairs, because we have enough labeled examples for the RCV2 languages, and we do not have enough unlabeled parallel text in the Tigrinya language pack.

4.4. RESULTS AND ANALYSIS:

We train each model using ten different random seeds and report their average test accuracy. 9 We describe training details and hyperparameters in the appendix. Table 1 and Table 2 show average test accuracies of different models on RCV2 and LORELEI on nine related language pairs. CACO vs. DAN: The low-resource CACO models have similar average test accurcy as the highresource DAN models. The SRC variant does not use any target language data, and yet it outperforms mCCA on three language pairs. When we already have a good CLWE, we can get the best of both world by combining them (COM), which achieves the highest average test accuracy.

Multi-Task Learning: Training CACO with multi-task learning further improves the accuracy.

For almost all language pairs, the multi-task CACO variants have higher test accuracies than SRC. On RCV2, word translation (DICT) is particularly effective, even with only 100 translation pairs. Interestingly, word translation and mimick tasks together (ALL) do not consistently increase the accuracy over only using the dictionary (DICT). On the LORELEI dataset where labeled document is limited, parallel projection task (SRC P and MIM P ) also significantly increases the accuracies.

Language Identifier: The variants that use language identifiers (DICT + and ALL + ) have lower average test accuracies than their counterparts (DICT and ALL). As discussed in §2.3, language identifiers could cause the embedders to overfit the training dictionary. The embedder might learn to behave very differently for the two languages, which prevents generalization across languages.

Language Relatedness: We expect CACO to be less effective when the source language and the target language are less close to each other. For comparison, we experiment on RCV2 with transferring between more distantly related language pairs: a North Germanic language and a Romance language. We include the results in the appendix (Table 5 ). Indeed, CACO models score consistently lower than the DAN models when transferring from a North Germanic source language to a Romance target language. However, CACO models are surprisingly competitive with DAN models when transferring from a Romance language to a North Germanic language, which shows that our method is useful even for loosely related languages. This asymmetry is likely due to morphology differences between the two families.

Multi-Source Transfer: We experiment with multi-source transfer learning on RCV2 by training models on two Romance languages and testing on another Romance language. Languages can be similar along different dimensions, and therefore adding more source languages may be beneficial. Moreover, using multiple source languages has a regularization effect and prevents the model from overfitting to a single language. For fair comparison, we sample 750 training documents from each source language, so that the multi-source models are still trained on 1,500 training documents (same as the single-source models). We use a similar strategy to sample the training dictionaries and pre-trained word embeddings. Multi-source models (Table 3 ) consistently outperform single-source models (Table 1 ) for both CACO and DAN.

Learned Word Representation: We evaluate the word representations learned by CACO with a word translation task between French, Italian, and Spanish. Specifically, we use the SRC embedder to generate embeddings for all French, Italian, and Spanish words that appear in both multiCCA and multiCluster vocabulary.10 Table 4 shows the word translation accuracy (P@1) on the test dictionaries from the MUSE library (Conneau et al., 2018) . Although the SRC embedder is not exposed to any cross-lingual signal, it still rivals CLWE on the word translation task by exploiting subword similarities between languages.

Qualitative Analysis: To understand how cross-lingual subword similarity can help document classification, we manually compare the output of a DAN model and a CACO model. Specifically, we use the mCCA model and the DICT model from the Spanish to Italian CLDC experiment, and we inspect their predictions on an Italian dev set. Sometimes CACO can avoid the mistakes of DAN by correctly aligning word pairs that are misaligned in the multiCCA embedding space. For example, in multiCCA, "relevancia" (relevance) is the closest Spanish word for the Italian word "interesse" (interest), while the CACO embedder maps both the Italian word "interesse" (interest) and the Spanish word "interesse" (interest) to the same point. Consequently, CACO correctly classifies an Italian document about the interest rate with GCAT (government), while the DAN model predicts MCAT (market). As another example, in multiCCA, the closest Spanish word for the Italian word "anticipazioni" (advance) is "anuncios" (advertisement), and DAN assigns a wrong label to an Italian document about wage advance. The CACO embedder aligns the Italian word "anticipazioni" (advance) with the Spanish word "aprovisionada" (provision) and therefore correctly labels the document.

5. CONCLUSION AND FUTURE WORK:

We introduce a CLDC framework, CACO, that trains a document classifier with labeled training documents from a related language and optionally a small dictionary, pre-trained embeddings, or parallel text. Our method exploits subword similarities between related languages to generalize from source language data. This technique is particularly useful in low-resource settings where unlabeled monolingual or parallel texts are limited. We provide empirical evaluation of CACO on multiple related language pairs, and CACO matches high resource CLWE-based methods without using monolingual or parallel text from the target language. In the future, we plan to extend our method to related languages with different scripts by combining with a transliteration tool or a grapheme-to-phoneme transducer (Mortensen et al., 2018) . We also want to investigate alternative architectures for the embedder, including morpheme-based models (Luong et al., 2013) .

Table 5 : CLDC experiments between languages from different families on RCV2. When transferring from a North Germanic language to a Romance language, CACO models score much lower than DAN models (top). Surprisingly, CACO models are on par with DAN when transferring from a Romance language to a North Germanic language (bottom). We boldface the best result for each row, underline CACO results that outperform one DAN model, and double underline CACO results that outperform both DAN models. 

:

https://www.ushahidi.com/categories/haiti

Corporate/Industrial (CCAT), Economics (ECAT), Government/Social (GCAT), and Markets (MCAT).

Amharic: LDC2016E87. Tigrinya: LDC2017E57.

For Tigrinya documents, the labels are extracted from the situation frame annotation in the language pack. We mark all documents with a "crimeviolence" situation frame as positive.

Annotations available after blind review.

We only apply this baseline to RCV2, because the test sets in LORELEI are too small to split further.

Each Amharic document only contains one sentence.

To avoid introducing extra bias, we sample the parallel documents such that the English model output approximately follows a uniform distribution.

For models that use dictionaries, we also re-sample the training dictionary for each run.

ES: 220,063 words, FR: 219,635 words, IT: 234,366 words.

Figures and Tables:

CLDC experiments between eight related European language pairs on RCV2 topic identification. The CACO models are competitive with DAN models that use far more resources. The combined model (COM) achieves the highest average test accuracy. We boldface the best result for each row, underline CACO results that outperform at least one DAN model.

Table content: <table><tr><td/><td/><td/><td>DAN</td><td/><td/><td/><td>CACO</td><td/><td/><td/><td/></tr><tr><td colspan="4">source target mCCA mClu</td><td>SUP</td><td>SRC</td><td colspan="2">DICT DICT +</td><td>MIM</td><td colspan="2">ALL ALL +</td><td>COM</td></tr><tr><td>DA</td><td>SV</td><td>69.3</td><td>63.0</td><td>59.7</td><td>56.0</td><td>62.8</td><td>62.8</td><td>60.4</td><td>62.9</td><td>62.9</td><td>69.7</td></tr><tr><td>SV</td><td>DA</td><td>51.4</td><td>40.8</td><td>54.7</td><td>56.7</td><td>60.2</td><td>63.4</td><td>58.4</td><td>62.2</td><td>56.0</td><td>67.5</td></tr><tr><td>FR</td><td>ES</td><td>63.9</td><td>71.8</td><td>56.6</td><td>49.6</td><td>59.3</td><td>59.8</td><td>48.3</td><td>57.4</td><td>60.2</td><td>70.8</td></tr><tr><td>IT</td><td>ES</td><td>43.4</td><td>55.6</td><td>56.6</td><td>50.2</td><td>54.6</td><td>48.5</td><td>51.4</td><td>54.7</td><td>44.9</td><td>63.5</td></tr><tr><td>ES</td><td>FR</td><td>63.1</td><td>60.0</td><td>48.9</td><td>48.5</td><td>49.7</td><td>43.2</td><td>49.2</td><td>48.8</td><td>47.4</td><td>61.3</td></tr><tr><td>IT</td><td>FR</td><td>26.7</td><td>66.5</td><td>48.9</td><td>45.9</td><td>52.1</td><td>52.2</td><td>46.6</td><td>48.2</td><td>47.2</td><td>62.8</td></tr><tr><td>FR</td><td>IT</td><td>43.6</td><td>59.6</td><td>44.9</td><td>43.3</td><td>53.2</td><td>54.4</td><td>44.3</td><td>51.2</td><td>48.5</td><td>60.2</td></tr><tr><td>ES</td><td>IT</td><td>51.3</td><td>51.4</td><td>44.9</td><td>49.7</td><td>53.5</td><td>47.6</td><td>53.4</td><td>52.5</td><td>48.2</td><td>59.7</td></tr><tr><td/><td>average</td><td>51.6</td><td>58.6</td><td>51.9</td><td>50.0</td><td>55.7</td><td>54.0</td><td>51.5</td><td>54.7</td><td>51.9</td><td>64.5</td></tr></table>

CLDC experiments between Amharic and Tigrinya on LORELEI disaster response dataset.

Table content: <table><tr><td/><td/><td/><td/><td>SRC</td><td>MIM</td><td colspan="2">SRC P MIM P</td></tr><tr><td>AM</td><td>TI</td><td>59.1</td><td>55.8</td><td>55.5</td><td>56.3</td><td>57.0</td><td>57.6</td></tr><tr><td>TI</td><td>AM</td><td>58.1</td><td>50.0</td><td>56.8</td><td>55.1</td><td>-</td><td>-</td></tr></table>

Results of CLDC experiments using two source languages. Models trained on two source languages are generally better than models trained on only one source language (Table1). We boldface the best result for each row, underline CACO results that outperform at least one DAN model.

Table content: <table><tr><td/><td/><td>DAN</td><td/><td/><td>CACO</td><td/><td/><td/></tr><tr><td>source</td><td colspan="2">target mCCA mClu</td><td>SRC</td><td colspan="2">DICT DICT +</td><td>MIM</td><td colspan="2">ALL ALL +</td></tr><tr><td>FR+IT</td><td>ES</td><td>74.2 77.0</td><td>58.8</td><td>67.0</td><td>61.1</td><td>55.8</td><td>65.3</td><td>60.4</td></tr><tr><td>ES+IT</td><td>FR</td><td>55.2 65.3</td><td>51.8</td><td>55.8</td><td>50.3</td><td>50.3</td><td>56.0</td><td>52.5</td></tr><tr><td>ES+FR</td><td>IT</td><td>45.4 61.0</td><td>53.2</td><td>56.1</td><td>55.6</td><td>55.9</td><td>56.5</td><td>53.2</td></tr><tr><td colspan="2">average</td><td>58.3 67.8</td><td>54.6</td><td>59.6</td><td>55.7</td><td>54.0</td><td>59.3</td><td>55.4</td></tr></table>

Word translation accuracies (P@1) for different embeddings. The CACO embeddings are generated by the embedder of a SRC model trained on the source language. Without any cross-lingual signal, the CACO embedder has similar word translation accuracy as mCCA and mCluster, which are supervisedly trained on large dictionaries.

Table content: <table><tr><td colspan="5">source target mCCA mClu CACO</td></tr><tr><td>ES</td><td>FR</td><td>36.8</td><td>40.8</td><td>31.1</td></tr><tr><td>ES</td><td>IT</td><td>44.0</td><td>35.7</td><td>33.1</td></tr><tr><td>FR</td><td>ES</td><td>34.0</td><td>40.2</td><td>30.9</td></tr><tr><td>FR</td><td>IT</td><td>33.5</td><td>40.3</td><td>29.6</td></tr><tr><td>IT</td><td>ES</td><td>42.1</td><td>37.8</td><td>37.5</td></tr><tr><td>IT</td><td>FR</td><td>35.6</td><td>39.8</td><td>36.4</td></tr><tr><td/><td>average</td><td>37.7</td><td>39.1</td><td>33.1</td></tr><tr><td colspan="5">baselines, but to show that CACO is competitive with the baselines despite training on much</td></tr><tr><td>less target language data.</td><td/><td/><td/><td/></tr></table>

Peer review (iclr2019):
 Overview: This paper proposes an approach to document classification in a low-resource language using transfer learning from a related higher-resource language. For the case where limited resources are available in the target low-resource language (e.g. a dictionary, pretrained embeddings, parallel text), multi-task learning is incorporated into the model. The approach is evaluated in terms of document classification performance using several combinations of source and target language. Main strengths: 1. The paper is well written. The model description in Section 2 is very clear and precise. 2. The proposed approach is simple but still shows good performance compared to models trained on corpora and dictionaries in the target language. 3. A large number of empirical experiments are performed to analyse different aspects and the benefits of different target-language resources for multi-task learning. Main weaknesses: 1. The application of this model to document classification seems to be new (I am not a direct expert in document classification), but the model itself and the components are not (sequence models, transfer learning and multitask learning are well-established). So this raises a concern about novelty (although the experimental results are new). 2. With regards to the experiments, it is stated repeatedly that the DAN model which are compared to uses "far more resources." The best ALL-CACO model also relies on several annotated but "smaller" resources (dictionaries, parallel text, embeddings). Would it be possible to have a baseline where a target-language model is trained on only a small amount of annotated in-domain document classification data in the target language? I am proposing this baseline in order to answer two questions. (i) Given a small amount of in-domain data for the task at hand, how much benefit do we get from additionally using data from a related language? (ii) How much benefit do we get from using target-language resources that do not address the task directly (dictionaries, embeddings) compared with using a "similar" amount of data from the specific task? Overall feedback: This is a well-written paper, but I think since the core of the paper lies in its empirical evaluation, the above experiments (or something similar) would greatly strengthen the work. Edit: I am changing my rating from 5 to 6 based on the authors* response.