# Using corpora for post-editing neural MT in highly specialised domains

Natalie Kübler, Hanna Martikainen, Alexandra Mestivier, Mojca Pecman

► To cite this version:

Natalie Kübler, Hanna Martikainen, Alexandra Mestivier, Mojca Pecman. Using corpora for post-editing neural MT in highly specialised domains. Using Corpora for Contrastive and Translation Studies UCCTS2021, Department of Interpreting and Translation of the University of Bologna, Sep 2021, Bertinoro, Forlì, Italy, Italy. [https://arxiv.org/abs/2103.09676](https://arxiv.org/abs/2103.09676)

HAL Id: hal-04990259

https://u-paris.hal.science/hal-04990259v1

Submitted on 13 Mar 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Using corpora for post-editing neural MT in highly specialised domains

## The case of complex noun phrases

Natalie Kübler¹, Hanna Martikainen², Alexandra Mestivier¹, Mojca Pecman¹

Université de Paris¹, CLILLAC-ARP, Paris, France

ESIT / Université Sorbonne Nouvelle², CLESTHIA, Paris, France
Corpus-Based Translation Studies and Learner Corpora

* Corpora and translation studies : specific features of translated texts (Baker 1993)

* Corpora and translation teaching : translation training, designing of pedagogical material, analysing features in learner translations, aid in translation (Bowker 1998, Castagnoli et al. 2011, Espunya 2014, Frankenberg-Garcia 2015, Kunilovskaya & Morgoun 2016, Loock 2016, Liu 2020, Granger & Lefer 2020, Verplaetse etc.) cf. Marie-Aude's talk!

Research in the classroom: students' translation error annotation ⇒ impact of corpus use on translation

⇒ impact of corpus use on specialised translation in the classroom since 2013.

=> impact of corpus use on specialised post-editing in 2018
# Post-editing MT: research questions

* Productivity & quality, spec. in comparison with human translation (e.g. Martikainen & Mestivier 2020, Schumacher 2020)

* Impact of translation experience (e.g. Guerberof Arenas 2014)

* Cognitive effort in post-editing, spec. with regards to MT errors (e.g. Koponen 2016, Daems et al. 2015)

* User perception and its impact on the use of technology (e.g. Rossi & Chevrot 2019)

Teaching MTPE (e.g. Guerberof Arenas & Moorkens 2019, Konttinen et al. 2020)

Post-editing in LSP (e.g. Martikainen 2019, Maniez 2020)

⇒ impact of corpus use on post-edited MT output in the specialised classroom

- none, to the best of our knowledge

- in the NLP domain, humans tend to disappear from the loop altogether, see for instance approaches using machine-generated corpora for automatic post-editing (e.g. Negri et al., 2018)
Complex NPs in specialised texts

Characteristic feature of specialised texts;
NPs involving pre- or post-modification of nouns by adjectives, nouns, PPs or reduced relative clauses or noun coordination:

volatile abundances and fluxes, subduction-zone serpentinites,

efficient carbon-concentrating mechanism

C-rich layer of clay-sized Fe oxides and alumina silicates, material obtained via drilling

* Lots of research on complex NPs, few of them in LSP (Maniez 2008, 2010, Ormrod 2000, 2004, Kageura 1997, Portelance 1987, Mestivier-Volanschi 2015, etc.)

* Maniez (2008, 2010, 2013) works on the strategies French translators adopt when dealing with complex NPs, especially in the medical field

* Maniez (2017) investigated the efficiency of neural MT (NMT) systems for translating compound adjectives from English into French

* Maniez (2020) identification of ambiguity in CNPS for human translators and MT post-editors
Project

Collaborative project (with EPS dpt) since 2004 at University of Paris
(Diderot), evaluated since 2013:

M1 students in specialised translation & L3 students in Earth and
Planetary Sciences S1: Terminology – S2: Translation of specialised
articles

Approach to teaching LSP translation (Kübler 2011, Kübler et al. 2016,
2018, forthcoming):

Objective: Raise students' linguistic awareness on potential problems

Method: Use of corpora, practice in classroom

Focus on students' problems in LSP, e.g. complex noun phrases,
terminology & phraseology
Classroom context

ca. 50 students per year

1st term:

comparable corpus compilation (merged : EN: 44, 332 million tokens; FR: 164,644 million tokens) and exploration;

terminology processing in ARTES online database (terminology + phraseology);

dialog with expert students & teachers in EPS.

2nd term:

human translation of a different extract for each student (250 words);

MT post-editing of a second extract (250 words);

⇒ ca. 100 different extracts to be annotated
Some Articles 2020-2021

The Atmosphere of Mars as observed by Insight (2020)

Drainage of a deep magma reservoir near Mayotte inferred from seismicity and
deformation (2020)

Variable water input controls evolution of the Lesser Antilles volcanic arc (2020)

Dynamic interaction between basin redox and the biogeochemical nitrogen cycle in an unconventional proterozoic petroleum system (2019)

Geology of the InSight landing site on Mars (2020)

Buoyant hydrous mantle plume from the mantle transition zone (2019)

Decline of giant impacts on Mars by 4.48 billion years ago and an early opportunity for
habitability (2019)

3D fault architecture controls the dynamism of earthquake swarms (2020)

Do slow slip events trigger large and great megathrust earthquakes? (2018)

The Great Oxidation Event preceded a Paleoproterozoic"snowball Earth" (2020)

Shallow seismic activity and young thrust faults on the Moon (2019)

Ammonium availability in the Late Archaean nitrogen cycle (2019)
Corpora 2018-2021 experiments

<table><thead><tr><th rowspan="2"></th><th colspan="2">Human translation<br>(MT not allowed)</th><th colspan="2">Post-editing</th></tr><tr><th>Bi-text size in words</th><th>Annotations (and feedback notes)</th><th>Bi-text size in words</th><th>Annotations (and feedback notes)</th></tr></thead><tbody><tr><td>2018-2021</td><td>84,613<br>(140 unique extracts<br>translated by 140 students)</td><td>2,999 (1,640)</td><td>81,332 (139 unique<br>extracts post-edited<br>by 139 students)</td><td>2,357 (1,599)</td></tr><tr><td>2020-2021</td><td>29,044 (48 unique extracts<br>translated by 48 students)</td><td>830 (504)</td><td>28,974<br>(48 unique extracts<br>translated by 48<br>students)</td><td>929 (538)</td></tr></tbody></table>
Spring 2021

1. **Students:** translation of extract 1 using term bases, corpora and Internet searches (MT not allowed)

2. **Teachers:** annotation of translations using the Mellange error typology on Brat

3. **Students:** revision using compiled comparable specialised corpus + all tools available

4. **Students:** post-editing of MT output of extract 2 using comparable corpus

5. **Teachers:** PE annotation using the Mellange error typology + MT/PE labels

6. **Students:** revision using compiled comparable specialised corpus

=> Final 500 wds extract with comment
# Annotation of MT/PE

2018-2020 experiments (Kübler *et al.* forthcoming) :
⇒ Mellange typology not sufficient for PE annotation
⇒ 6 new labels for annotating correct/incorrect MT output, correct/incorrect/lack of PE

* **Incorrect MT output:**

a. TAEBienCor: incorrect MT, correct PE

b. TAEMalCor: incorrect MT, incorrect PE (=> failure to correct)

c. TAENonCor: incorrect MT, no PE (=> under-edited)

* **Correct MT output**

d. TACorBienCor: correct MT, correct PE with an acceptable variation

e. TACorMalCor: correct MT, incorrect PE (=> overedited)

f. (TACorNonCor: correct MT, no PE)

* Case f not annotated (time-consuming) ⇒ indicative results

* All other cases annotated with Mellange + MT/PE labels
MeLLANGE error typology

<figure><img src="image_2.png" /></figure>
Brat double annotation scheme

EN Denitrification (the reduction of NO3- to N2) and anaerobic ammonium oxi-

TA La dénitrification de NO3- en N2) et l'oxydation de l'oxyde de N minimum et dans les zones de minimum d'oxygène ID:T14 aérobie de

PE La dénitrification de NO3- en N2) et l'oxydation anaérobie de zones de minimum d'oxygène

Type-annotateur_LA-TL-UD

TA_Erronee: TAEBienCor

"zones de minimum d'oxygène"

Note: terme ok

Type-annotateur_LA-TL-UD 1

zones de minimum d'oxygène ainsi que dans les sédiments marins.

Utilisation du corpus (oui/non) Oui, artes

OBS Oxygen minimum zones : OMZ (terme)
Correct MT correctly post-edited: NPs

TACorBienCor: 104 all annotations; 41 for NPs

Both *gravity-wave oscillations*, with vertical wavelengths of a couple
kilometres

⇒ **MT:** Les *oscillations des ondes gravitationnelles*, avec des longueurs d'ondes verticales de quelques kilomètres

⇒ **PE:** Les *oscillations des ondes de gravité*, d'une longueur d'onde verticale de quelques kilomètres

Both *onde de gravité* and *ondes gravitationnelles* are correct (corpus)
Correct MT, incorrect post-editing: NPs

TACorMalCorr: 235 all annotations; 126 for NP

(...) *by the weathering and breakdown of resistant basaltic rim blocks that continues at a greatly diminished *average rate*.*

⇒ **MT:** (...) *l'altération et la décomposition des blocs basaltiques résistants du bord qui se poursuivent à un rythme moyen considérablement réduit*

⇒ **PE:** (...) *l'altération et la désagrégation des blocs de basaltes résistantes du bord qui se poursuivent à une *cadence moyenne fortement diminuée*.*

A brief lookup in the corpus will confirm the use of « rythme » instead of « cadence » which is used in a different context (workplace).
Incorrect MT, not post-edited: NPs

TAENonCorr: 75 all annotations; 46 for NPs

*This study focuses on two main Omani metamorphic sole localities*

⇒ **MT:** Cette étude se concentre sur *deux principales localités de soles métamorphiques omanaises*

⇒ **PE:** cette études se concentre sur *deux principales localités de soles métamorphiques omanaises*

**Suggested PE:** cette étude se concentre sur **deux localisations principales de semelles métamorphiques omanaises.**
Incorrect MT, correct post-editing: NPs

TAEBienCor: 370 all annotations; 280 for NPs

*The resulting structure is bounded by low-angle thrusting along the external faults*

⇒ **MT:** La structure résultante est délimitée par des *poussées à faible angle le long des failles externes*

⇒ **PE:** la structure résultante est délimitée par **le chevauchement à faible pendage le long des failles externes**

Terminology :complex NP with embedded complex term
Incorrect MT, incorrect post-editing: NPs

TAEMalCorr: 145 all annotations, 94 for NPs

We also develop an algorithm to invert for **the centroïd MT** of the VLPs
⇒ **MT:** Nous **développons également un algorithme pour inverser** *le centroïde MT des VLP*

N.B. VLP stands for very-long-period; MT for moment tensor

⇒ **incorrect PE:** Nous **développons également un algorithme pour inverser**
*les moments sismiques centroïdes des signaux longue durée*

Correct PE: (...) Nous **développons également un algorithme pour inverser les moments sismiques centroïdes des signaux à longue période**
Some figures

<table><thead><tr><th>Label</th><th>All annotations</th><th>NP</th><th>NonNP</th></tr></thead><tbody><tr><td>TACorNonCor</td><td>N/A</td><td>N/A</td><td>N/A</td></tr><tr><td>TACorBienCor</td><td>104 (11.19%)</td><td>41 (4.41%)</td><td>63 (6.78%)</td></tr><tr><td>TACorMalCor</td><td>235 (25.29%)</td><td>126 (13.56%)</td><td>109 (11.73%)</td></tr><tr><td>TAENonCor</td><td>75 (8.07%)</td><td>46 (4.95%)</td><td>29 (3.12%)</td></tr><tr><td>TAEBienCor</td><td>370 (39.82%)</td><td>280 (30.13%)</td><td>90 (9.68%)</td></tr><tr><td>TAEMalCor</td><td>145 (15.60%)</td><td>94 (10.11%)</td><td>51 (5.48%)</td></tr><tr><td>TOTAL</td><td>929 (100%)</td><td>587 (63.18%)</td><td>342 (36.81%)</td></tr></tbody></table>
Results : MT in EPS

MT seems to produce less complex NP errors than human translation :
CNP syntactic issues are better dealt with by MT than by our students
(MT calque works better in most cases)?

⇒ our CNP error typology (Kübler *et al.* forthcoming) does not really apply for PE

MT difficulties for CNPs tend to be more terminology-related

⇒ linked to the specialised domain

MT difficulties for non CNP cases are phraseological

⇒ related to the genre : scientific article in EPS

MT learns from our input
# Corpus use and PE

## Correct MT, incorrect post-editing

* slight tendency to mistrust and over-edit MT output: NonNPs and NPs

* introduction of all types of errors

* no corpus use

* focus on terminology

## Incorrect MT, correct post-editing

* increased linguistic awareness, especially on CNPs

* corpus use shown in comments (students stated that they used the corpus for 65% of all sentences)

* aware of terminology => term base use
# Conclusion and Perspectives

Corpus use seems to improve PE

=> Students should be shown that MT errors differ from human errors

Longitudinal study ?

* Analyse students’ comments on corpus use

* Collect more corpus using the double annotation to provide better figures

* Train general MT system with specialised corpora?
References (1/2)

Bowker, L. (1998), ‘Using specialized monolingual native-language corpora as a translation resource: a pilot study’, *Meta: Journal des traducteurs* / *Meta: Translators’ Journal* 43 (4): 631-651.

Castagnoli, S., D. Ciobanu, N. Kübler, K. Kunz and A. Volanschi (2011), ‘Designing a Learner Translator Corpus for Training Purposes’, in N. Kübler (ed.) *Corpora, Language, Teaching, and Resources: From Theory to Practice*, 221-248, Bern: Peter Lang.

Daems, J., Vandepitte, S., Hartsuiker, R., Lieve, M. (2015). *The Impact of machine Translation Error Types on Post-Editing Effort Indicators*. In O’Brien Sh. & Simard M., (Eds) *Proceedings of 4th Workshop on Post-Editing technology and practice*. p.31-45.

Espunya, A. (2014), ‘The UPF Learner Translation Corpus as a Resource for Translator Training’, *Language Resources and Evaluation*, 48: 33–43.

Frankenberg-Garcia, Ana 2015. ‘Training translators to use corpora hands-on: challenges and reactions by a group of 13 students at a UK university’, *Corpora, 10* (2): 351–380.

Granger, S. and M.-A. Lefer (2020), ‘The Multilingual Student Translation Corpus: a Resource for Translation Teaching and Research’, *Language Resources & Evaluation*[A1] [AX2], 54 (4); 1183-1199.

Guerberof Arenas, A. (2014). The role of professional experience in post-editing from a quality and productivity perspective.

Guerberof Arenas, A., & Moorkens, J. (2019). Machine translation and post-editing training as part of a master's programme. *Jostrans: The Journal of Specialised Translation*, (31), 217-238.

Koponen, M. (2016). Machine translation post-editing and effort: Empirical studies on the post-editing process.

Kunilovskaya, M. and N. Morgoun (2016), ‘Available Corpora and Error-Annotated Student Translations in Translator Education’, in *Proceedings of the 6th Conference. The Future of Education*, 121–5, Padova: *Libreria Universitaria*.

Kübler, N. (2003), ‘Corpora and LSP translation.’ in F. Zanettin, S. Bernardini and D. Stewart (eds), *Corpora in Translator Education*, 25-42, Manchester: St Jerome Publishing.

Kübler, Natalie, Mestivier, Alexandra et Pecman, Mojca, in press. Using Comparable Corpora for Translating and Post-Editing Complex Noun Phrases in Specialised Texts: Insights from English-to-French Specialised Translation. In : GRANGER, Sylviane et LEFER, Marie-Aude (éd.), Extending the Scope of Corpus-Based Translation Studies. S.l. : Bloomsbury Academic. Bloomsbury Advances in Translation. ISBN 978-1-350-14325-8.
References (2/2)

Maniez, F. (2008), 'Using the Web and corpora as language resources for the translation of complex noun phrases in medical research articles', Panacea IX (26): 162-167.

Maniez, F. (2010), 'La traduction des adjectifs composés en langue médicale : étude d'un corpus bilingue anglais-français', Séminaire du CRTT, Université de Lyon 2, Lyon.

Maniez, F. (2013), 'The translation into French of adjectives formed with a noun and a past participle in English-language medical articles', Panacea XIV (38): 240-247.

Maniez, F. (2017), 'Evaluation des récentes avancées de la traduction automatique : le cas des adjectifs composés formés à partir d'un participe passé en anglais de spécialité', ASp 72: 29-48.

Maniez, F. (2020). The Identification of Potentially Ambiguous Noun Phrases in Scientific English: A Crucial Aspect of Translator and Post-editor Training. In Levey D. (Ed.), Strategies and Analyses of Language and Communication in Multilingual and International Contexts, Cambridge Scholars Publishing, 187.

Martikainen, H. (2019). Post-Editing neural MT in Medical LSP: Lexico-Grammatical patterns and distortion in the communication of specialized knowledge. In Informatics (Vol. 6, No. 3, p. 26).

Martikainen H., Mestivier A. Les outils de traduction nouvelle génération: quel effet sur la qualité des textes traduits ? Traduction & Qualité: biotraduction et traduction automatique, Université de Lille, 31 janvier 2020.

Negri, M., Turchi, M., Chatterjee, R., & Bertoldi, N. (2018). ESCAPE: a large-scale synthetic corpus for automatic post-editing. arXiv preprint arXiv:1803.07274.

Rossi, C., & Chevrot, J. P. (2019). Uses and perceptions of machine translation at the European Commission. The Journal of specialised translation (JoSTrans).

Schumacher, P. (2020). Post-édition et traduction humaine en contexte académique: une étude empirique.

Verplaetse, Heidi (2021) Translation quality in student specialized translation: The impact of corpus use. In Granger, S. and M.-A. Lefer (2020), 'The Multilingual Student Translation Corpus: a Resource for Translation Teaching and Research', Language Resources & Evaluation[A1] [AX2], 54 (4); 1183-1199.
Thanks for your attention!

* The Atmosphere of Mars as observed by Insight (2020)

* Drainage of a deep magma reservoir near Mayotte inferred from seismicity and deformation (2020)

* Variable water input controls evolution of the Lesser Antilles volcanic arc (2020)

* Dynamic interaction between basin redox and the biogeochemical nitrogen cycle in an unconventional proterozoic petroleum system (2019)

* Geology of the InSight landing site on Mars (2020)

* Buoyant hydrous mantle plume from the mantle transition zone (2019)

* Decline of giant impacts on Mars by 4.48 billion years ago and an early opportunity for habitability (2019)

* 3D fault architecture controls the dynamism of earthquake swarms (2020)

* Do slow slip events trigger large and great megathrust earthquakes? (2018)

* The Great Oxidation Event preceded a Paleoproterozoic "snowball Earth" (2020)

* Shallow seismic activity and young thrust faults on the Moon (2019)

* Ammonium availability in the Late Archaean nitrogen cycle (2019)