Assessing the Use of Terminology in Phrase-Based Statistical Machine Translation for Academic Course Catalogues Translation
Authors: Randy Scansani, Marcello Federico, Luisa Bentivogli (Università di Bologna, Fondazione Bruno Kessler, 2017)
Topic: Machine Translation

Abstract
In this contribution we describe an approach to evaluate the use of terminology in a phrase-based machine translation system to translate course unit descriptions from Italian into English.~The genre is very prominent among those requiring translation by universities in European countries where English is not a native language.~Two MT engines are trained on an in-domain bilingual corpus and a subset of the Europarl corpus, and one of them is enhanced adding a bilingual termbase to its training data.~Overall systems' performance is assessed through the BLEU score, whereas the f-score is used to focus the evaluation on term translation. Furthermore, a manual analysis of the terms is carried out. Results suggest that in some cases - despite the simplistic approach implemented to inject terms into the MT system - the termbase was able to bias the word choice of the engine.

Introduction
Availability of {course unit descriptions or course catalogues} in multiple languages has started to play a key role for universities especially after the Bologna process AUTHOR and the resulting growth in student mobility.~These texts aim at providing students with all the relevant information regarding contents, pre-requisites, learning outcomes, etc. Since course unit descriptions have to be drafted in large quantities on a yearly basis, universities would benefit from the use of machine translation (MT).~Indeed, the importance of developing MT tools in this domain is further testified by two previous projects funded by the EU Commission, i.e.~TraMOOC} and Bologna Translation Service}. The former differs from the present work since it does not focus on academic courses, while the latter does not seem to have undergone substantial development after 2013 and in addition to that, it does not include the Italian-English language combination. Automatically producing multilingual versions of course unit descriptions poses a number of challenges.~A first major issue for MT systems is the scarcity of high quality human-translated parallel texts of course unit descriptions. Also, descriptions feature not only terms that are typical of institutional academic communication, but also expressions that belong to specific disciplines AUTHOR. This makes it cumbersome to choose the right resources and the most effective method to add them to the MT engine. For this study, we chose to concentrate on course units belonging to the disciplinary domain of exact sciences, since Italian degree programmes whose course units belong to this domain translate their contents into English more often than other programmes. A phrase-based statistical machine translation system (PBSMT) was used to translate course unit descriptions from Italian into English. We trained one engine on a subset of the Europarl corpus and on a small in-domain corpus including course unit descriptions and degree programs (see sect.~Subsection3.2) belonging to the domain of the exact sciences. Then, we enriched the training data set with a bilingual terminology database belonging to the educational domain (see sect.~Subsection3.3) and built a new engine. To assess the overall performance of the two systems we automatically evaluated them with the BLEU score. We then focused on the evaluation of terminology translation, by computing the f-score on the list of termbase entries occurring both in the system outputs and in the reference translation (see sect.~results). Finally, to gather more information on term translation, a manual analysis was carried out (see sect. sect:ManualEvaluation).

Conclusion
This paper has described a preliminary analysis aimed at assessing the use of in-domain terminology in PBSMT in the institutional academic domain, and more precisely for the translation of course unit descriptions from Italian into English. Following the results of the present experiment and given its preliminary nature, we are planning to carry out further work in this field. In section results we have seen that the institutional academic terms contained in our testing data also appeared in the training data, thus limiting the impact of terminology on the output. However, course catalogues and course unit descriptions include terms belonging to the specific disciplines (see sect.~intro) as well. In our future works we are therefore planning to focus not only on academic terminology, but also on the disciplinary one testing its impact on the output of an MT engine translating course unit descriptions. After this first experiment on the widely-used PBSMT architecture,~in future work we are planning to exploit neural machine translation (NMT).~In particular, our goal is to develop an NMT engine able to handle terminology correctly in this text domain, in order to investigate its effect on the post-editor's work.~For this reason, a termbase focused on the institutional academic domain, e.g.~the UCL-K.U.Leuven University Terminology Database} or the Innsbrucker Termbank 2.0} could be used to select an adequate benchmark for the development and evaluation of an MT engine with a high degree of accuracy in the translation of terms.