SlowGuess/ACL_2020_OCR huggingface dataset:
Document Translation vs. Query Translation for Cross-Linguual Information Retrieval in the Medical Domain
Shadi Saleh and Pavel Pecina
Charles University, Faculty of Mathematics and Physics
Institute of Formal and Applied Linguistics
Malostranské nám. 25, 118 00 Prague, Czech Republic
{saleh,pecina}@ufal.mff.cuni.cz
We present a thorough comparison of two principal approaches to Cross-Linguual Information Retrieval: document translation (DT) and query translation (QT). Our experiments are conducted using the cross-lingual test collection produced within the CLEF eHealth information retrieval tasks in 2013–2015 containing English documents and queries in several European languages. We exploit the Statistical Machine Translation (SMT) and Neural Machine Translation (NMT) paradigms and train several domain-specific and task-specific machine translation systems to translate the non-English queries into English (for the QT approach) and the English documents to all the query languages (for the DT approach). The results show that the quality of QT by SMT is sufficient enough to outperform the retrieval results of the DT approach for all the languages. NMT then further boosts translation quality and retrieval quality for both QT and DT for most languages, but still, QT provides generally better retrieval results than DT.
Multilingual content has been growing significantly in the last few years simultaneously with rapid internet access growth all over the world. Monolingual information retrieval task allows users to find information in documents that are written in the language that they use to write their queries. This ignores a vast amount of information that is represented in other languages. Cross-Lingual Information Retrieval (CLIR) breaks this language barrier by allowing users to look up information that is represented in documents written in languages different from the language of the query.
We reinvestigate the effectiveness of two principal approaches to CLIR: document translation (DT) and query translation (QT). The existing comparison studies of the two approaches are outdated (e.g.
Oard, 1998) and do not reflect the current advances in Machine Translation (MT). Even in very recent works, the authors have blindly assumed that DT is superior to QT (Khiroun et al., 2018), giving the argument that in DT, the text is translated in a larger context compared to the translation of short isolated queries in QT. The larger context should help in translation disambiguation and better lexical selection during translation, which should subsequently lead to better retrieval results.
This hypothesis needs to be revised, taking into consideration the significant improvement of machine translation quality in recent years, despite the strong practical disadvantages of DT over QT: DT is computationally expensive and hard to scale (every document needs to be translated into each supported language and then indexed) while QT is performed in query time and only a short text (the query) is translated into the document language.
In this work, state-of-the-art Statistical Machine Translation (SMT) and Neural Machine Translation (NMT) systems are deployed for document translation and query translation to investigate their effect on retrieval quality in the cross-lingual setting. The experiments are conducted using the cross-lingual test collection produced within the CLEF eHealth tasks on patient-centered information retrieval in 2013–2015 extended with additional relevance assessments and manual query translations (Saleh and Pecina, 2019). Though this is a very specific domain and the results cannot be thoughtlessly generalized to other domains, the choice of this test collection was motivated by two facts: First, it provides resources for large-scale experimentation (1 million in-domain documents, 166 queries in 8 languages, thorough relevance assessment). Second, the medical domain in MT has been well studied (Jimeno Yepes et al., 2017; Dušek et al., 2014), and there are enough resources to develop well-performing MT systems for multiple languages.
6849
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6849-6860 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
In CLIR, documents and queries are written in different languages. The traditional term-matching retrieval methods require both documents and queries to be represented in the same language. In practice, either the queries need to be translated into the document language (QT), or the documents need to be translated into the query language (DT). Not many studies and experiments have been conducted in order to compare these two approaches.
Oard (1998) investigated the performance of DT, QT, and a hybrid system combining both. They found that the system translating English queries into German (the document language) outperformed the system translating the documents from German into English (the query language). They hypothesized that documents, which are typically longer than queries, provide more contextual and linguistic information that helps reduce translation ambiguity and thus improves translation quality. McCarley (1999) presented a hybrid DT/QT system, which averaged the retrieved document scores from DT and QT systems and thus outperformed both of them. Fujii and Ishikawa (2000) employed a two-step method where QT was first used to retrieve a limited number of documents that were translated into the query language and reranked by their DT retrieval scores.
Pirkola (1998) presented a new method for CLIR, which was referred to as structured queries. The idea was that a document containing one possible translation candidate of a query term is more relevant than a document that contains multiple translations of that term. This probabilistic structured queries approach was also applied to Cross-Language Speech Retrieval (Nair et al., 2020). Darwish and Oard (2003) also exploited alternative translations of query terms. Their experiments showed that combining multiple translations outperformed the selection of one best translation.
Nikoulina et al. (2012) investigated reranking SMT translation hypotheses towards better CLIR performance and showed that SMT systems are usually trained to give the best results in terms of translation accuracy, adequacy, and fluency. However, an improvement will be achieved when they are optimized towards retrieval quality. We followed this approach in our previous work and introduced a richer set of features and adopted the hypothesis reranker for multiple languages in the medical domain (Saleh and Pecina, 2016b,a).
Several recent papers employed methods based on Deep Learning. Litschko et al. (2018) presented an unsupervised CLIR approach employing shared cross-lingual word embedding model, which was trained using monolingual data only. They used those embeddings to translate query terms word by word into the document language. Rücklé et al. (2019) trained NMT model for CLIR using out-domain data and synthetic data (created by translating in-domain monolingual English into German) to retrieve answers to German questions from English collection in the technical domain (AskUbuntu and StackOverflow).
CLIR in the medical domain has been investigated within the series of CLEF ShARE/eHealth labs since 2013 which focused on improving access of laypeople (non-medical experts) to reliable medical information (Goeuriot et al., 2013, 2014; Palotti et al., 2015; Kelly et al., 2016; Palotti et al., 2017; Jimmy et al., 2018; Kelly et al., 2019).
In this paper, we compare the performance of both QT and DT using the traditional SMT and state-of-the-art NMT methods trained on the same data to make the comparison as fair as possible. We present a novel approach for NMT model selection that is optimized towards CLIR performance and investigate the effect of morphological pre- and post-processing on the performance on CLIR.
Two types of data were used in our experiments: The data for training, tuning, and testing MT (Section 3.1) and the CLIR test collection (Section 3.2).
3.1 Machine Translation Resources
Parallel data is essential for training both SMT and NMT systems. We exploited the UFAL Medical Corpus<sup>1</sup> which was assembled during the course of several EU projects aiming at more reliable machine translation of medical texts and used for the purposes of WMT Biomedical Translation Task (Bojar et al., 2014). It mainly includes the EMEA corpus by Tiedemann (2009), UMLS metathesaurus (Humphreys et al., 1998), titles from Wikipedia articles in the medical categories mapped to other languages using Wikipedia Interlingual links, medical domain patent applications (Waschle and Riezler, 2012; Pouliquen and Mazenc, 2011), and various web-crawled data.
<sup>1</sup>http://ufal.mff.cuni.cz/ufal_medical Corpus
Monolingual data is used to build a language model during the development of SMT systems. The language model helps select a candidate translation that is as coherent and fluent as possible in the target language (which is certainly important for document translation, but less important for query translation). Our procedure of data selection (both parallel and monolingual data) follows the work of Pecina et al. (2014), where two language models are trained on in-domain and general-domain data respectively, then each sentence from the corpus is scored by its cross-perplexity between the two models. Finally, the top 10 million scored sentences are chosen. In NMT training, the monolingual data is used to enlarge the parallel data training data by back-translation, where target language monolingual data is machine translated to the source language and added to parallel data for training. The monolingual data used in our experiments includes multiple resources such as the CLEF eHealth 2014 English document collection (Goeuriot et al., 2014), Genia corpus (Ohta et al., 2002), and medical Wikipedia articles in English.
MT development and test data: used for tuning and evaluating our MT systems consists of the Khresmoi Summary Translation Test Data $^2$  used by the DT models and Khresmoi Query Translation Test Data  $2.0^{3}$  used by the QT models. Both were developed within the Khresmoi project $^4$  and later extended within the KConnect $^5$  and HimL $^6$  projects. The summary test data includes sentences (1,000 for testing and 500 for development) from summaries of English medical articles manually translated from English to all relevant languages. The query test data includes English queries (1,000 for testing and 500 for tuning) sampled from a query log of a medical search engine and manually translated to the same set of languages.
3.2 CLIR Test Collection
For CLIR experiments, we use the CLIR test collection<sup>7</sup>. that we developed in our previous work (Saleh and Pecina, 2019). It is based on the data used within the CLEF eHealth lab IR tasks in 2013-2015 (Suominen et al., 2013; Goeuriot et al., 2014;
Palotti et al., 2015). It contains about 1.1 million web pages that were crawled automatically from various trusted medical websites (Goeuriot et al., 2015). There are 166 queries in total (100 for training and 66 for testing) originally formulated in English (to mimic real patient queries) and then manually translated by medical experts into seven European languages (Czech, French, German, Spanish, Swedish, Polish, and Hungarian). The relevance judgments consist of the official relevance assessments provided by the task organizers and additional assessments, as described in (Saleh and Pecina, 2019).
We clean the document collection by removing HTML tags and other scripts in the documents. All the lemmatization experiments in our work are done using UDPipe (Straka and Straková, 2017), while for stemming, we use the Snowball algorithm (Moral et al., 2014).
4 Retrieval System
The document collection is indexed using Terrier (Ounis et al., 2005), an open-source tool for information retrieval experiments. For retrieval, we use Terrier's implementation of the language model with Bayesian smoothing and Dirichlet prior (Smucker and Allan, 2005) with the default value of the smoothing parameter.
5 Machine Translation Systems
In this section, we provide details on training the SMT and NMT systems used in the CLIR experiments. The SMT systems fully replicate the work by Dusek et al. (2014); we only provide the most important information. The NMT systems are described in full detail.
5.1 Statistical Machine Translation
The SMT systems are based on the phrase-based SMT paradigm implemented in Moses (Koehn et al., 2007). The system for the QT experiments was developed within the Khresmoi project (Dušek et al., 2014). The system was tuned to translate medical search queries (using the Khresmoi Query development set) and optimized towards PER (Position-independent word Error Rate, Tillmann et al., 1997) instead of the traditionally preferred BLEU (Papineni et al., 2002) as this was shown to be more effective for tuning SMT parameters for translating search queries (Pecina et al., 2014). The system is denoted as QT-SMT-form.
$^{2}$ http://hdl.handle.net/11234/1-2122
3http://hdl.handle.net/11234/1-2121
<sup>4</sup>http://khresmoi.eu/
5http://www.kconnect.eu/
http://www.himl.eu/
$^{7}$ http://hdl.handle.net/11234/1-2925
6851
For the DT experiments, we train two SMT systems:  $DT-SMT$ -form, which is a replication of the SMT system that translates standard sentences by Dusek et al. (2014), and our own system  $DT-SMT$ -pre-lem that translates English sentences into lemmatized sentences in the target language. This is done by lemmatizing the monolingual data and the target side of the parallel data prior to training. In both the systems, we use fast align (Dyer et al., 2013) to train word alignment model on the lowercased word forms between English and the target language, then we replace the word forms in the target language with word lemmas. Moses (with its default settings) is used to train a phraseable model using the tokenized and lowercased English word forms, and the tokenized and lemmatized data in the target language plus a 5-gram language model. Minimum Error Rate Training (MERT, Och, 2003) is used to tune the model parameter weights using the development data sets. We also experiment with another system ( $DT-SMT$ -post-lem), which produces lemmatized output but obtained as post-lemmatization of the output of the  $DT-SMT$ -form system, and a system ( $DT-SMT$ -post-stem) which produces stemmed output obtained by the Snowball stemmer applied again to the output of  $DT-SMT$ -form. This is to allow better comparison of the DT and QT approaches. Translating documents into a morphologically richer language enlarge the vocabulary (term diversity) and thus make retrieval more difficult. The three systems produce morphologically reduced translations of documents and thus make them comparable to the English ones (in terms of vocabulary size).
5.2 Neural Machine Translation
Neural Machine Translation (NMT) has become the state-of-the-art approach in MT and recently achieved superior results and lead to a significant improvement over the SMT systems (Jean et al., 2015). We implement two types of NMT systems: one for query translation (denoted as QT-NMT-form) and one for document translation (denoted as DT-NMT-form). Both produce standard (non-lemmatized) output.
The systems are based on the Marian (Junczys-Dowmunt et al., 2018) implementation of the Transformer (Vaswani et al., 2017) model with backtranslation (Edunov et al., 2018). SMT has an advantage over NMT in employing monolingual data in its language model. This gap can be bridged
Figure 1: A schema of the iterative back-translation mechanism for NMT training.
by back-translation, a technique that exploits another MT model to translate monolingual data from the target language into the source language and adds this "synthetic" data to the original parallel data(Sennrich et al., 2016a). This approach also helps for domain adaption of NMT when the monolingual data is taken from a specific domain. We follow the back-translation approach in this work iteratively.
5.2.1 Task-Oriented NMT Training
The NMT systems are trained using the same training data as the SMT systems. However, in NMT, all data sets (monolingual and parallel) are encoded into Byte-Pair Encoding (BPE), which helps reduce the out-of-vocabulary problem in NMT by encoding rare words as sequences of subword units (Sennrich et al., 2016b). We train the Transformer model using the same parameters as reported by Vaswani et al. (2017). Figure 1 shows the architecture of the proposed iterative back-translation NMT model, inspired by the work of Hoang et al. (2018): for each language pair, we first train initial models for both directions, English to target, and source to English. We use the authentic (non-synthetic) parallel data that is presented in Section 3.1 for training the initial models.
During training the Transformer models, multiple epochs (iterations through the entire training data) are needed. It is known that too many training epochs can cause over-fitting of the model, and a few iterations might cause under-fitting (Popel and Bojar, 2018). To avoid this, the early-stopping of the training is employed to terminate the process when the intermediate model satisfies some
6852
stopping criteria (training objective). We stop training when there are three consecutive checkpoints without any improvement in the translation performance of the validation data. Then, we use the initial model to translate monolingual text in the target language coming from two resources:
MT parallel training corpus: the target side of the parallel training data (Section 3.1) is translated into English using the  $SRC \rightarrow EN$  NMT model to create the synthetic data for the models that are used in DT experiments. The English side of the parallel corpus is translated using the  $EN \rightarrow TGT$  model for the QT experiment. This is done to investigate the effect of the source of the monolingual data on the CLIR performance. We randomly select 2 million sentences in each iteration.
CLIR test collection: we select randomly 2 million sentences from the test collection (Section 3.2) (after filtering sentences that are longer than 80 words), then we use  $EN \rightarrow TGT$  model to translate them into the target language. This is done for models that are used for the query translation approach. The motivation of choosing the collection is to make the model adapted to translate the medical queries into English (the document language).
After translating this monolingual data, we create the synthetic data by adding the monolingual data and their translations to the authentic parallel data. Then we continue training of the models in both directions. We conduct back-translation three times, and in each iteration, we use the updated models from the previous one.
5.2.2 NMT Model Selection
We setup Marian to save the intermediate models (checkpoints) after every 5,000 iterations where each iteration is a batch sized of instances from the training data. This is done instead of saving each epoch to avoid loosing effective intermediate models in between. The model selection is based on evaluating each checkpoint by BLEU (Papineni et al., 2002) and PER (Tillmann et al., 1997) using the Khresmoi Summary development set (DT) and Khresmoi Query development set (QT).
Figure 2 shows the evaluation results of the intermediate models using the two MT metrics and how they correlate with P@10 (IR metric). P@10 is calculated by query translation of the Czech training queries into English using the corresponding NMT model, and then conducting retrieval as we describe in Section 4. Choosing the model that gives the best BLEU scores (iteration 400,000) does not cor
Figure 2: Performance comparison of the intermediate QT-NMT-form models at each checkpoint (after each 5,000 iterations) in terms of BLEU, 1-PER, and  $\mathbb{P}@\mathbb{10}$  when employed in the Czech QT CLIR system.
relate with the best value for  $\mathrm{P}@\mathbf{10}$  , nor the best score for PER (500,000). This is understandable because these metrics evaluate translation quality.
In order to select the best checkpoint that guarantees the advantages of both metrics (BLEU, which penalizes word order and PER which does not), we ensemble the two models together (best BLEU and best PER) during decoding by setting up the weights for both models equally. Marian decoder supports model ensembling since they share the same vocabularies. For the document translation experiments, we select the NMT models with the highest BLEU scores.
6.1 MT Evaluation
In this section, we present intrinsic evaluation of the MT systems. We evaluate how well the systems translate sentence/query queries given their reference translations in the test data. We present both BLEU and PER scores (all as percentages). The higher the BLEU score, the better the translation quality is. BLEU is based on measuring the similarity of n-grams counts between a translation hypothesis and its reference translation(s), and as such is sensitive to word order. PER, on the other hand, does not penalize word order between a translation hypothesis and its reference translation as BLEU does. Instead, it considers both as a "bag of words". PER captures all words that appear in a translation hypothesis but do not exist in the reference. These words are known as PER errors; thus, the higher the PER value, the lower the translation quality.
6853
MT System	EN-CS	EN-FR	EN-DE	EN-HU	EN-ES	EN-SV	EN-PL
BLEU	PER	BLEU	PER	BLEU	PER	BLEU	PER	BLEU	PER	BLEU	PER	BLEU	PER
DT-SMT-form	19.0	51.1	37.8	68.3	18.7	53.4	10.5	41.6	25.7	63.2	33.6	64.6	11.5	41.3
DT-NMT-form	25.9	56.5	38.8	66.5	19.8	51.4	8.2	39.5	23.2	55.2	35.1	64.4	10.2	35.9
DT-SMT-post-lem	30.9	65.6	43.5	74.7	23.6	60.4	13.2	48.6	35.4	72.3	40.9	69.9	16.1	50.5
DT-SMT-pre-lem	28.7	64.2	41.2	72.6	13.0	48.0	14.3	51.9	28.4	65.7	39.1	70.0	12.5	46.9
Table 1: Intrinsic evaluation of MT systems for document translation using the Khresmoi Summary Test set.
MT System	CS-EN	FR-EN	DE-EN	HU-EN	ES-EN	SV-EN	PL-EN
BLEU	PER	BLEU	PER	BLEU	PER	BLEU	PER	BLEU	PER	BLEU	PER	BLEU	PER
QT-SMT-form	36.4	70.2	38.7	75.9	37.0	65.2	39.7	67.3	31.2	73.7	39.2	62.7	26.0	58.6
QT-NMT-form	22.5	48.9	30.6	65.4	28.7	58.1	36.7	63.2	17.8	45.5	40.9	63.0	18.7	47.9
Table 2: Intrinsic evaluation of MT systems for query translation using the Khresmoi Query Test set.
The MT evaluation scores cannot be directly compared across language pairs, and for the  $*$ -form and  $*$ -lem systems (since the test sets differ), but they indicate to what extent the translated queries differ from the reference translations, which in term-matching IR is important. Also, the results of the two systems producing lemmas instead of the word forms are indicative only. They cannot be directly compared to those producing word forms.
Table 1 displays the (intrinsic) evaluation of the MT systems for document translation using the Khresmoi Summary test set (in terms of BLEU and PER). The results are not very consistent: For six out of the seven translation directions,  $DT$ -NMT-form outperforms  $DT$ -SMT-form in terms of PER. In terms of BLEU,  $DT$ -NMT-form wins for four language pairs.
The effect of lemmatization on the scores is not surprising. Naturally, lemmatization reduces the vocabulary size in the target language; thus, the BLEU scores are higher for the systems which employ lemmatization in either way. However, post-lemmatization is constantly better (with the exception of Hungarian, which is a very specific language, and its scores are generally much lower than for other languages). In terms of PER, the situation is different, and despite the fact that lemmatization reduces the target language, the systems without lemmatization often achieve better scores (except in German and Spanish).
Table 2 presents the (intrinsic) evaluation of the MT systems for QT using the Khresmoi Query test set. QT-SMT-form outperforms QT-NMT-form in terms of BLEU in all the languages except Swedish. However, in terms of PER (which is preferred), QT-NMT-form is always better. This can be partially