SlowGuess/ACL_2020_OCR huggingface dataset:
explained because of the way we ensembled NMT models towards better CLIR performance. The bold font indicates which of the two  $*$ -form systems is better (for each language pair and each measure).
6.2 CLIR experiments
Table 3 presents the results of the CLIR experiments altogether. Motivated by the organization of the CLEF eHealth CLIR tasks, we adopt P@10 (the percentage of relevant documents among the top ten retrieved ones) as the main evaluation measure. In all the experiments, all the top 10 ranked documents for each query are assessed for relevance. We also report MAP (Mean Average Precision) as a secondary evaluation measure. The *-SMT-form systems are treated as baselines. The figures in bold denote results better than the baseline. Those, which are statistically significantly better are in bold and also in italics. The significance tests were performed using the paired Wilcoxon signed-rank test (Hull, 1993) with  $\alpha = 0.05$ , and no correction was applied.
First, we conduct monolingual experiments using the English queries and the English document collection to set a reference (oracle) system for our CLIR task, that is why all the results of monolingual systems are the same for all the languages. We report the following: Mono-form system uses the original English queries and the English collection (no morphological processing applied). Mono-lem and Mono-stem report the results after performing lemmatization and stemming of the document collection and the English queries, respectively. The purpose of these systems is to study the effect of the morphological processing of the English documents on retrieval performance.
6854
MT System	Czech	French	German	Hungarian	Spanish	Swedish	Polish
P@10 MAP	P@10	MAP	P@10	MAP	P@10	MAP	P@10	MAP	P@10	MAP	P@10	MAP	P@10
Monolingual (Oracle)
Mono-form	53.0	28.3	53.0	28.3	53.0	28.3	53.0	28.3	53.0	28.3	53.0	28.3	53.0	28.3
Mono-lem	52.1	27.5	52.1	27.5	52.1	27.5	52.1	27.5	52.1	27.5	52.1	27.5	52.1	27.5
Mono-stem	52.1	26.4	52.1	26.4	52.1	26.4	52.1	26.4	52.1	26.4	52.1	26.4	52.1	26.4
Query translation
QT-SMT-form	47.2	22.6	48.0	23.6	44.2	21.7	45.9	22.9	46.9	23.2	40.0	20.2	42.1	20.1
QT-NMT-form	57.2	26.0	51.5	24.1	50.3	22.5	50.7	24.0	49.0	22.6	50.1	23.8	47.2	22.3
Document translation
DT-SMT-form	39.0	17.4	42.1	21.5	40.4	22.1	40.0	17.2	45.6	26.9	38.3	17.0	40.7	20.4
DT-SMT-post-stem	36.9	16.7	44.5	22.7	39.2	22.9	35.4	17.0	46.3	27.3	33.9	16.7	35.3	18.7
DT-SMT-post-lem	39.3	18.3	41.9	21.7	37.7	22.4	37.1	17.0	42.7	25.0	33.0	16.0	37.1	22.2
DT-SMT-pre-lem	42.8	21.3	43.6	20.6	42.1	19.8	36.5	16.8	47.7	22.4	30.7	12.6	34.8	19.7
DT-NMT-form	42.1	15.6	46.0	19.8	36.6	14.0	26.0	10.5	43.9	17.5	33.9	11.6	38.9	12.3
Table 3: Extrinsic evaluation of the MT systems in the CLIR task. The CLIR experiments are evaluated using the Extended CLEF eHealth 2013-2015 test collection and compared with the results of monolingual retrieval (queries in English).
The QT experiments are done using the SMT and NMT systems, both translating into word forms (QT-SMT-form and QT-NMT-form). We want to stress here that the used MT systems for QT are different from the MT systems for DT, not only in the translation direction but also in the way that they were trained and tuned. Details are presented in Section 5.1 and Section 5.2.
In DT experiments, we exploit several configurations of the MT systems. DT-SMT-form translates the collection from English into the target language by the SMT system for document translation (no morphological processing applied). DT-SMT-post-stem refers to the results obtained by stemming the output of the the DT-SMT-form system. DT-SMT-post-lem lemmatizes the output of the DT-SMT-form, while DT-SMT-pre-lem lemmatizes the training data prior SMT training. (i.e., the translated documents in this system are already lemmatized). To compare the performance of DT when employing the NMT model, we report DT-NMT-form, which uses the presented NMT models to translate the collection into all the languages.
6.3 Result Analysis
In this work, we are mainly interested in comparing NMT vs. SMT employed in both the CLIR approaches (DT and QT), comparing the two approaches as such and analyzing the effect of morphological normalization in DT.
NMT versus SMT: For the QT approach, we can conclude that in terms of  $\mathrm{P@10}$ , the NMT-based CLIR systems (using the QT-NMT-form MT systems) significantly outperform the SMT-based ones. Moreover, QT-NMT-form in Czech outperforms not only all other QT systems but also outperforms the monolingual system, which means that the NMT translations are on average better than the reference ones. This situation is illustrated in Table 4 which provides several examples of queries in which NMT not only provides translations which are better (in terms of  $\mathrm{P@10}$ ) than the ones provided by SMT but also better than the reference translations (for each translation, the  $\mathrm{P@10}$  score is in parentheses). This can be explained by the fact that the NMT models in our work are adapted to translate medical content by employing the collection itself in the back-translation process. This gives the model access to the collection vocabularies that are frequent in the retrieval collection, and in the relevant documents eventually.
To investigate this hypothesis. We train another QT-NMT-form system (for  $\mathrm{CS} \rightarrow \mathrm{EN}$  only) using a different source of the back-translation data, namely the English side of the MT parallel text, which is also from the medical domain but different from the CLIR collection (the other settings of the system remain the same). The performance of this system decreased (as expected) from  $57.2\%$  to  $54.2\%$  (statistically significant). This shows that employing the document collection in back
Query: 2013.38 (Czech)
SRC: IM a dédičnéy
REF: mi and hereditary (0.0)
SMT: mi and hereditary (0.0)
NMT: hereditary myocardial infarction (10.0)
Query: 2015.61 (French)
SRC: hématomes sous les ongles
REF: fingernail bruises (40.0)
SMT: bruising under the nail (10.0)
NMT: nail hematoma (60.0)
Query: 2014.19 (Swedish)
SRC: Laneurysm i halspulsader
REF:  $l$  common carotid aneurysm (60.0)
SMT: l aneurysm in halspulsader (0.0)
NMT: carotid artery aneurysm (100.0)
Query: 2015.61 (Spanish)
SRC: hematomas en la una del dedo
SMT: bruising in toe nail (20.0)
Table 4: Comparison of query translations by two systems (QT-SMT-form and QT-NMT-form) and reference translations and their effect on retrieval quality. The figures in parentheses represent P@10 (in percentages) of retrieval when using the translation as a single query.
translation indeed helps produce translations that are more adapted to the collection domain.
NMT also helps deal with out-of-vocabulary (OOV) words (i.e., words do not appear in the training data), which is a common problem in SMT. For instance, the translations of Swedish queries produced by QT-SMT-form contain 40 untranslated terms. However, in QT-NMT-form translations, due to BPE, there are no OOVs at all (all words get translated, though the correct translation is not guaranteed). Very likely, this has a positive effect on the CLIR performance too.
QT versus DT: The most surprising observation in this work is the predominance of QT over DT in our experiments. In terms of  $\mathrm{P@10}$ , for all the languages, QT-SMT-form provides significantly better translations than DT-SMT-form. For German and Spanish, the systems based on the translation of documents into morphologically normalized forms (lemmas, stems) perform on par with the systems based on QT-SMT-form, but for the other languages, the baseline QT-SMT-form is the best performing SMT option. The NMT models unsurprisingly boost translation quality for both QT and DT, but QT unexpectedly stays superior to DT, and the results get very close to the monolingual performance (and even higher for the Czech system, see above).
This can be explained by a simple hypothesis that a well-trained MT system based on the state-of-the-art techniques and sufficient amounts of training data is good enough to provide query translations of sufficient quality and does not require to see any larger context. The translation quality may not be perfect, but still sufficient for retrieval. For example, the Czech query clef2015.test.33, which
is "bílá infekce hltanu", is translated into English as "white infection of pharynx". The reference translation for that query is "white infection in pharynx". We can see that the CS→EN SMT system fails in translating prepositions ("of" instead of "in"), but this does not affect the CLIR performance. However, we should keep in mind that our experiments are carried out in a very specific domain. This means that the queries are short, and often include symptoms and health conditions in which linguistics and contextual information may not play a significant role in solving the translation ambiguity
Morphological normalization: Producing document translations (lemmatized or stemmed) reduces collection vocabularies and improves term matching. However, in our experiments, none of the DT-SMT systems employing morphologically normalized translations of documents outperforms (in terms of  $\mathrm{P@10}$ ) the QT-SMT-form systems.
An example of a query where morphological normalization improved retrieval is the Czech query clef2013.test.18: "aspiracni pneumie a dysfagie hltanu" ("aspiration pneumonia and pharyngeal dysphagia" in English). The word "hltanu", which means "pharyngeal" is lemmatized in the training data of the SMT system and the Czech query into "hltan", which means "pharynx". When translating the English documents into Czech, "pharynx" and "pharyngeal" are translated back into "hltan". This helps retrieve more relevant documents, increasing P@10 to 0.9 in DT-SMT-pre-lem from 0.7 in the monolingual systems (Mono, Mono-lem and Mono-stem), 0.6 in QT-SMT-form and 0.0 in DT-SMT-form. In comparison of pre-lemmatization
6856
and-post lemmatization, there is no clear winner. In the intrinsic MT evaluation,  $DT$ -SMT-post-lem outperforms  $DT$ -SMT-pre-lem for most languages. But in the extrinsic CLIR evaluation,  $DT$ -SMT-pre-lem is better for four languages and worse for Hungarian, Swedish, and Polish.  $DT$ -SMT-pre-lem in Spanish is the only DT system that outperforms the QT system. No clear conclusion can be done regarding the  $DT$ -SMT-post-stem models.
Finally, it is important to give insights about the cost-oriented comparison of the two approaches in terms of time complexity. The training time of our MT systems (both NMT and SMT) for both the approaches (QT and DT) is almost the same. The major difference was in the translation process. In the DT approach, translating the document collection using SMT took on average around three days using 200 CPU cores (each has 20 GB of RAM) for each language, which means it took us 21 days to translate 1.1 mil English documents into seven languages. While NMT translation was around ten times faster, using 20 GPUs only (GeForce RTX 2080Ti and Quadro P5000) with 10 GB of GPU RAM took around 20 days to translate the documents into the target languages. While for the QT approach, the translation process was pretty fast, where it took around 15 minutes to translate 66 queries from seven languages into English using SMT systems and around 3 minutes to do the same using NMT.
We presented a comparative study between querytranslation (QT), and document translation (DT) approaches in the Cross-Linguial Information Retrieval (CLIR) task. To conduct this study, we investigated various MT systems and their configurations and performed a thorough large-scale evaluation based on the test collection produced within the CLEF eHealth tasks on patient-centered information retrieval during 2013-2015, and extended with additional relevance assessments.
We experimented with both statistical and neural MT paradigms. The SMT systems for QT were specifically trained and tuned to translate medical search queries. For DT, we trained two SMT systems: the first one was built to produce word forms, and the second one to produce word lemmas. We then used these two systems to translate the test collection into seven European languages. Furthermore, we performed lemmatization and stem
ming on the collection that was translated using the SMT system that produces word forms. The results showed that a well-tuned QT system outperforms DT, which is a positive result with an important impact on practical applications. So far, the QT approach has been preferred mainly for efficiency reasons (less space and computation needed). Our experiments suggest that this approach is even more effective (better retrieval results).
We also investigated the effect of using neural machine translation, which is now considered the state-of-the-art in many domains. This completely new paradigm in machine translation tends to improve the fluency of generated output (which is appreciated by humans), but often mismatches content and adequacy (which might hurt the performance in IR). In our experiments, NMT improved retrieval results in both QT and DT, but the QT approach is still superior, so the results are consistent with the findings from the SMT experiments.
However, we emphasize that the way we trained our MT systems is very domain-specific (medical domain), and we made use of a vast amount of medical data (monolingual and parallel). This makes our comparative study very task-oriented. When dealing with general domain test collection, some search terms might have a different meaning in different domains. For example, the word "development" probably in most cases means in medicine the growth or spread of a disease (or a tumor), while in the general domain we can not say without a context, and in that case, the need for linguistics information in the queries will be more important to solve the translation ambiguity. This should be considered when comparing QT and DT approaches; thus, the reader should be careful when drawing the same conclusion of this work while working on a different domain.
This work was supported by the Czech Science Foundation (grant n. 19-26934X) and data and tools provided by the LINDAT/CLARIAH-CZ Research Infrastructure (https://lindat.cz), supported by the Ministry of Education, Youth and Sports of the Czech Republic (Project No. LM2018101).
We would like to thank everyone who helped us with this work, including Ondrej Bojar for his feedback on the design of the MT systems and Zain Baity for this effort in the annotation of our system results.
Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Aleš Tamchyna. 2014. Findings of the 2014 Workshop on Statistical Machine Translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 12-58, Baltimore, Maryland, USA. ACL.
Kareem Darwish and Douglas W. Oard. 2003. Probabilistic Structured Query Methods. In Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 338-344, New York, USA. ACM.
Ondrej Dušek, Jan Hajic, Jaroslava Hlaváčová, Michal Novák, Pavel Pecina, Rudolf Rosa, and et al. 2014. Machine Translation of Medical Texts in the Khresmoi Project. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 221-228, Baltimore, USA. ACL.
Chris Dyer, Victor Chahuneau, and Noah A Smith. 2013. A Simple, Fast, and Effective Reparameterization of IBM Model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics, pages 644-648, Atlanta, GA, USA. ACL.
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding Back-Translation at Scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 489-500, Brussels, Belgium. ACL.
Atsushi Fujii and Tetsuya Ishikawa. 2000. Applying Machine Translation to Two-Stage Cross-Language Information Retrieval. In *Envisioning Machine Translation in the Information Future*, volume 1934, pages 13–24. Springer, Berlin, Germany.
Lorraine Goeuriot, Gareth JF Jones, Liadh Kelly, Johannes Leveling, Allan Hanbury, Henning Müller, Sanna Salantera, Hanna Suominen, and Guido Zuccon. 2013. ShARE/CLEF eHealth Evaluation Lab 2013, Task 3: Information Retrieval to Address Patients' Questions when Reading Clinical Reports. CLEF 2013 Online Working Notes, 8138:1-16.
Lorraine Goeuriot, Liadh Kelly, Wei Li, Joao Palotti, Pavel Pecina, Guido Zuccon, Allan Hanbury, Gareth Jones, and Henning Mueller. 2014. ShARE/CLEF eHealth Evaluation Lab 2014, Task 3: User-centred Health Information Retrieval. In Proceedings of CLEF 2014, pages 43-61, Sheffield, UK. CEURWS.org.
Lorraine Goeuriot, Liadh Kelly, Hanna Suominen, Leif Hanlen, Aurélie Néváol, Cyril Grouin, Joao Palotti, and Guido Zuccon. 2015. Overview of the CLEF eHealth Evaluation Lab 2015. In The 6th Conference and Labs of the Evaluation Forum, pages 429-443, Berlin, Germany. Springer.
Vu Cong Duy Hoang, Philipp Koehn, Gholamreza Haffari, and Trevor Cohn. 2018. Iterative Back-Translation for Neural Machine Translation. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 18-24, Melbourne, Australia. ACL.
David Hull. 1993. Using Statistical Testing in the Evaluation of Retrieval Experiments. In Proceedings of the 16th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 329-338, Pittsburgh, USA.
Betsy Humphreys, Donald Lindberg, Harold Schoolman, and Octo Barnett. 1998. The Unified Medical Language System. Journal of the American Medical Informatics Association, 5(1):1-11.
Sebastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. On Using Very Large Target Vocabulary for Neural Machine Translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1-10, Beijing, China. ACL.
Antonio Jimeno Yepes, Aurélie Néveol, Mariana Neves, Karin Verspoor, Ondrej Bojar, Arthur Boyer, Cristian Grozea, Barry Haddow, Madeleine Kittner, Yvonne Lichtblau, Pavel Pecina, Roland Roller, Rudolf Rosa, Amy Siu, Philippe Thomas, and Saskia Trescher. 2017. Findings of the WMT 2017 Biomedical Translation Shared Task. In Proceedings of the Second Conference on Machine Translation, pages 234-247, Copenhagen, Denmark. ACL.
Jimmy, Guido Zuccon, Joao Palotti, Lorraine Goeuriot, and Liadh Kelly. 2018. Overview of the CLEF 2018 Consumer Health Search Task. In CLEF 2018 Evaluation Labs and Workshop: Online Working Notes, Avignon, France. CEUR-WS.
Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, Andre F. T. Martins, and Alexandra Birch. 2018. Marian: Fast Neural Machine Translation in C++. In Proceedings of ACL 2018, System Demonstrations, pages 116-121, Melbourne, Australia. ACL.
Liadh Kelly, Lorraine Goeuriot, Hanna Suominen, Aurlie Nvol, Joao Palotti, and Guido Zuccon. 2016. Overview of the CLEF eHealth Evaluation Lab 2016. In Experimental IR Meets Multilinguality, Multimodality, and Interaction, volume 9822, pages 255-266, Cham. Springer.
Liadh Kelly, Hanna Suominen, Lorraine Goeuriot, Mariana Neves, Evangelos Kanoulas, Dan Li, Leif Azzopardi, Rene Spijker, Guido Zuccon, Harrison Scells, and João Palotti. 2019. Overview of the CLEF eHealth Evaluation Lab 2019. In Experimental IR Meets Multilinguality, Multimodality, and Interaction, pages 322-339, Cham. Springer.
6858
Oussama Ben Khiroun, Bilel Elayeb, and Narjes Bellamine Ben Saoud. 2018. Towards a Query Translation Disambiguation Approach using Possibility Theory. In ICAART (2), pages 606-613, Portugal. SciTePress.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, and et al. 2007. Moses: Open Source Toolkit for Statistical Machine Translation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics, Demo and Poster Sessions, pages 177-180, Stroudsburg, PA, USA.
Robert Litschko, Goran Glavas, Simone Paolo Ponzetto, and Ivan Vulic. 2018. Unsupervised Cross-lingual Information Retrieval Using Monolingual Data Only. In The 41st International ACM SIGIR Conference on Research &#38; Development in Information Retrieval, SIGIR '18, pages 1253–1256, New York, NY, USA. ACM.
J. Scott McCarley. 1999. Should We Translate the Documents or the Queries in Cross-language Information Retrieval? In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics on Computational Linguistics, pages 208-214, College Park, Maryland. ACL.
Cristian Moral, Angelica de Antonio, Ricardo Imbert, and Jaime Ramírez. 2014. A Survey of Stemming Algorithms in Information Retrieval. Information Research: An International Electronic Journal, 19(1):n1.
Suraj Nair, Anton Ragni, Ondrej Klejch, Petra Galuscakova, and Douglas Oard. 2020. Experiments with Cross-Language Speech Retrieval for Lower-Resource Languages. In Information Retrieval Technology, pages 145-157, Cham. Springer.
Vassilina Nikoulina, Bogomil Kovachev, Nikolaos Lagos, and Christof Monz. 2012. Adaptation of Statistical Machine Translation Model for Cross-lingual Information Retrieval in a Service Context. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 109-119, Stroudsburg, PA, USA. ACL.
Douglas Oard. 1998. A Comparative Study of Query and Document Translation for Cross-Language Information Retrieval. In Machine Translation and the Information Soup, volume 1529, pages 472-483. Springer, Berlin, Germany.
Franz Josef Och. 2003. Minimum Error Rate Training in Statistical Machine Translation. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 160-167, Sapporo, Japan. ACL.
Tomoko Ohta, Yuka Tateisi, and Jin-Dong Kim. 2002. The GENIA Corpus: An Annotated Research Abstract Corpus in Molecular Biology Domain. In Proceedings of the Second International Conference on