# Translating scientific abstracts in the bio-medical domain with structure-aware models

Sadaf Abdul Rauf, François Yvon

► To cite this version:

Sadaf Abdul Rauf, François Yvon. Translating scientific abstracts in the bio-medical domain with structure-aware models. Computer Speech and Language, 2024, 87, pp.101623. 10.1016/j.csl.2024.101623. hal-04476788

HAL Id: hal-04476788

https://hal.science/hal-04476788v1

Submitted on 25 Feb 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
Translating Scientific Abstracts in the Bio-Medical
Domain with Structure-Aware Models*

Sadaf Abdul Rauf<sup>a,*</sup>, François Yvon<sup>b</sup>

<sup>a</sup>Fatima Jinnah Women University, The Mall, 46000, Rawalpindi, Pakistan

<sup>b</sup>Sorbonne Université, CNRS, ISIR, 4 Place Jussieu, 75 0105, Paris, France

Abstract

Machine Translation (MT) technologies have improved in many ways and generate
usable outputs for a growing number of domains and language pairs. Yet, most
sentence based MT systems struggle with contextual dependencies, processing
small chunks of texts, typically sentences, in isolation from their textual context.
This is likely to cause systematic errors or inconsistencies when processing long
documents. While various attempts are made to handle extended contexts in
translation, the relevance of these contextual cues, especially those related to
the structural organization, and the extent to which they affect translation
quality remains an under explored area. In this work, we explore ways to take
these structural aspects into account, by integrating document structure as an
extra conditioning context. Our experiments on biomedical abstracts, which are
usually structured in a rigid way, suggest that this type of structural information
can be useful for MT and document structure prediction. We also present in
detail the impact of structural information on MT output and assess the degree
to which structural information can be learned from the data.

Keywords: Neural Machine Translation, Document-level Machine Translation, Bio-medical Natural Language Processing

*This paper is published as : https://doi.org/10.1016/j.csl.2024.101623. Most of this work was performed while the authors were affiliated with LIMSI-CNRS (Orsay, France).

*Corresponding author.

Email addresses: sadaf.abdulrauf@gmail.com (Sadaf Abdul Rauf), yvon@isir.upmc.fr
(François Yvon)
# 1. Introduction

Neural Machine Translation (NMT) systems have progressed remarkably well in the last decade with some claims of reaching human parity [1, 2] but much remains to be studied about the underlying learning of NMT systems. While substantial developments have been made towards integrating a document-level context [3, 4, 5, 6], the exploration of these contextual cues and the extent to which they affect translation quality remains an under explored area [7]. Translation depends upon multiple sources of dependencies between sentences in a document, that may be due to semantic, stylistic, discursive or pragmatic issues. Moreover, complex documents are also structured in parts, subparts and sections, and depending on their position within this global structure, sentences may convey varying pragmatic functions. Titles, introductory or conclusion statements, for instance, are typically thought to be playing a distinguished role and to contain the most important elements of content [8].

Sections in a document often follow distinct writing styles and lexical preferences. For instance, an analysis of 50 most frequent words from the medical abstracts used in our experiments (see Section 4.1) showed that *Titles* had the most domain specific content words; the word *"however"* was in the most frequent words for *Introduction* and *Conclusion* only; *Abstract*, *Results* and *Conclusion* had the most overlap of lexical space. Grammatically, sentences in the *Results* and *Materials and Methods* sections were majorly in past tense, whereas *Conclusion* sentences were in present continuous as well as past tense. *Objectives* are mostly written in present tense with a few sentences in past tense and so on. As sections in a document have their own grammatical and lexical blueprints, which must be reflected in their textual content, we hypothesize that informing MT systems about sentences' placement in a document may help improve the overall performance.

For a translation system, the capacity to model the context may notably improve certain translation decisions, e.g. a better or most consistent lexical choice [9], or a better translation of anaphoric pronouns [3, 5]. The research
efforts to augment NMT models with contextual or document-level information
may be broadly categorized as *data-* vs *model*-centric approaches, where *data- centric* approaches focus on increasing the scope of the input to include more
source and/or target texts [10, 11, 12, 13], while *model-centric* approaches aim
to adjust the training architecture and objective [14, 15, 16]. Recent surveys of
this subdomain of NMT are in [6, 17].

In this work, we study whether also incorporating *structural information*
can help improve the MT choices and thus positively impact translation quality.
Note that the structural information considered here is related to the pragmatic
organization of the documents, where sectioning helps to organize the discourse
into coherent subparts, each aiming to fulfill a specific goal. This is in contrast
with other attempts to translate structured documents [18, 19] which focus on
integrating syntactic markers of the structure or [20], where structural organi-
zation identifies thematic changes. In our *data-centric* approach, contextual
and structural information are injected for each sentence based on a hierarchy
of special tokens and does not require any change in the model. While simple,
this technique allows us to study whether learning coherent, section-specific,
translation styles can contribute to improving the overall performance.

As tags can be used in multiple ways, we further explore the differences
in tagging the source or the target side, which also allows us to evaluate how
reliably the structure information can be learned automatically from the data. We
systematically study the impact of each tag on metric score and the translation
output in conjunction with human analysis of section wise lexical preferences
versus the MT models section prediction capabilities.

Our experiments focus on abstracts of scientific texts in the bio-medical
domain [21, 22], where document structure information is often available. For
this target genre, texts are expected to follow a standardized structure com-
prising typical subsections of one to five lines, known as the IMRaD structure
(for **Introduction**, **Method**, **Results**, and, **Discussion**) [23]. As we discuss in
Section 3.1, preparing the data for these experiments has been challenging and a
significant outcome of our work will be a new dataset annotated with document
structure, that we will release to the community<fnref n="1" />.

In summary, the main research questions that we address in this work are:

1. [RQ1] Does incorporating document structure information in text help improve machine translation performance?

65

2. [RQ2] What is the most effective way to incorporate this auxiliary information?

3. [RQ3] When the structure is not observed, how well can it be automatically predicted? What will be the effect of prediction errors?

70

This paper is organized as follows: we start with a brief overview of Neural Machine Translation models (section 2), followed by an elaboration of our approach and architecture in sections 3 and 4. Results based on automatic metric scores are detailed in sections 5 and 6, whereas the analysis of a human evaluation is presented in 6.3. The paper concludes with an overview of related works (Section 7) and a general discussion in Section 8.

75

## 2. Neural Machine Translation

### 2.1. Probabilistic encoder-decoder models

In this section, we briefly introduce the main concepts of neural machine translation (NMT), with an emphasis on concepts whose understanding is important for accurately describing methods that handle context. A much more complete presentation of NMT is in [24, 25].

80

The general principle of all MT architectures is to generate the best possible target translation $e = e_1...e_T$ of the input source sentence $f = f_1...f_S$, according to a probabilistic decision rule:

$$e^* = \arg\max_{e} p_{\theta}(e|f), \qquad (1)$$

where the model parameters ($\theta$) are estimated from sentence-aligned parallel corpora. Learning such a distribution is unrealistic if one considers complete

<footnote n="1">¹https://github.com/fyvo/WMT-Biomed-Test</footnote>
sentences as the random variable in the model. However, neural implementations of this probabilistic model enable to directly factorize this distribution as:

$$p_{\theta}(\mathbf{e}|\mathbf{f}) = \prod_{t=1}^{T} p_{\theta}(\mathbf{e}_t|\mathbf{e}_{<t}, \mathbf{f}), \qquad (2)$$

without having to resort to conditional independence hypotheses (eg. Markovian dependencies).

Equation (2) simply states that the probability of each target word is generated conditioned on the current prefix of the target sentence ($\mathbf{e}_{<t} = e_1 \dots e_{t-1}$) and on the entire source sentence ($\mathbf{f}$). The manipulation of such distributions is made possible by transforming discrete contexts ($\mathbf{e}_{<t}, \mathbf{f}$) and words $\mathbf{e}_t$ in the vocabulary into continuous representation spaces. This means that each word $\mathbf{f}_t$ (and accordingly each context or each substring) is associated with a large numerical vector representing the useful information about that word or context.

Factorization (2) suggests that the program (1) can be solved approximately by generating words from left to right according to the following greedy procedure:

$$p_{\theta}(\mathbf{e}_t|\mathbf{f}) = \operatorname*{argmax}_{\mathbf{e}_t} \prod_{t=1}^{T} p_{\theta}(\mathbf{e}_t|\mathbf{e}_{<t}, \mathbf{f}). \qquad (3)$$

More effective search procedures can also be entertained (see [26] for a review).

The estimation of $\theta$ is performed by maximizing the conditional log-likelihood (or cross-entropy) $\sum_{t=1}^{T} \log p_{\theta}(\mathbf{e}_t|\mathbf{e}_{<t}, \mathbf{f})$ accumulated over a large set of sentences, resulting in a complex optimization program that is typically solved in an approximate manner by generic numerical optimization algorithms.

The main neural architectures are mainly distinguished according to the way the conditioning context is encoded. The Transformer architecture [27] replaces the recurrent component (in the source and target) introduced in [28] by generalized attentional modules, while retaining the source-target cross-attention component of early RNN-based architectures. This change makes all previous positions $t-1, t-2, \dots, 1$ equally important in the selection of the current word (and likewise for source representations). It also enables to process all the target tokens in parallel during training, causing vast decrease in training time.
Decoding must still be carried out from left to right, since each word already generated conditions the generation of future words.

## 2.2. Monolingual and Bilingual Attention Modules

In a nutshell, the Transformer architecture transforms a *structured context* (a sequence of tokens, but this can also be a tree or a graph) into a single numerical vector. The core operation in this transformation is the iterative computation of each individual token's representation based on their similarity to other tokens in the context. Denoting $H^{l-1} = [h_1^l, ..., h_T^l]$ the $(T \times d)$ matrix representing a context of length $T$ at the input of layer $l$, the representation $\tilde{h}_i^{kl}$ for token $i$ is computed by *attention head* $kl$ as:

$$ \tilde{h}_i^{kl} = \mathrm{softmax}\left(\frac{h_i^{l-1} Q^{kl} [H^{l-1} K^{kl}]^T}{\sqrt{d_k}}\right) H^{l-1} V^{kl}, \quad (4) $$

, with $Q^{kl}$, $K^{kl}$, $V^{kl}$ the parameter matrices associated to this head and layer, $d$ the model dimension, and $d_k$ the size of each of the $K$ heads ($d_k = d/K$). In this model, $H_0$ contains the lexical and positional embeddings. The output of these $k$ computations are then concatenated and passed through a feed-forward (FFN) layer with ReLU activation; each of these steps also includes a summation with $H^{l-1}$ and a layer normalization. In equations:

$$ \begin{aligned} \tilde{H}^{kl} &= \mathrm{Attn}(H^{l-1}Q, H^{l-1}K^{kl}, H^{l-1}V^{kl}) \\ H^l &= \mathrm{LayerNorm}(H^{l-1} + [\tilde{H}^{1l}, \dots, \tilde{H}^{Kl}]) \\ H^l &= \mathrm{LayerNorm}(H^l + \mathrm{FFN}(H^l)), \end{aligned} $$

where Attn denotes the computation expressed by equation (4).

When used for MT, the self-attention mechanism is computed on the source side on the full observed sentence. On the target side, the self-attention is restricted to the previous words; an additional *cross-attention* module combines at each layer the decoder-side representations with the encoder representations output by the top layer $H^K$: this way, information may flow from source to target, but not in the reverse direction.
### 3. Methods: Translating with Document Structure

We propose to introduce document structure information at the level of each sentence through a system of hierarchical tags. Tags have been widely used in NMT to incorporate additional discrete conditioning factors in (1), often for issues quite distinct from those caused by long-range dependency, e.g. to control the output language, the domain, or the level of formality and politeness [29, 30, 31]. Tags also provide us with a simple and effective way to take structure into account.

#### 3.1. Tagging sections within documents

##### 3.1.1. The IMRaD scheme

Our starting point is that scientific documents in our target domain often follow a rigid structure, known as the IMRaD structure [23]. This means that most abstracts will comprise one or several lines for the following sections: *Introduction*, *Method*, *Results*, and, *Discussion* (see example in Figure 1), where we also consider the title line to be part of the structure.

Note that this structure can be overt and introduced by appropriate headings, or remain covert. In the latter case, the document structure cannot be immediately recovered from the text. Also note that variations in the organization and in the heading labels can be observed depending on the subdomains and publication venues.

Our main hypothesis is that each of these sections can be characterized with specific terms and phraseological patterns: in our experiments, we thus explore how to use this information in NMT and measure the correlated impact. We notably expect that by informing the system with sub-document information, it will better translate the typical style and phraseology of sentences occurring in each part.

For this purpose, we identified in our data all the abstracts that were conforming to this basic structure and worked to make this structure as explicit and standardized as possible. This notably implied to normalize the main headings,
<table><tr><td>Title:</td><td>Comparison of two azithromycin distribution strategies for controlling trachoma in Nepal.</td></tr><tr><td>OBJECTIVE:</td><td>The study compares the effectiveness of two strategies for distributing azithromycin in an area with mild-to-moderate active trachoma in Nepal.</td></tr><tr><td>METHODS:</td><td>The two strategies investigated were the use of azithromycin for 1) mass treatment of all children, or 2) targeted treatment of only those children who were found to be clinically active, as well as all members of their household.</td></tr><tr><td>FINDINGS:</td><td>Mass treatment of children was slightly more effective in terms of decreasing the prevalence of clinically active trachoma (estimated by clinical examination) and of chlamydial infection (estimated by DNA amplification tests), although neither result was statistically significant.</td></tr><tr><td>CONCLUSION:</td><td>Both strategies appeared to be effective in reducing the prevalence of clinically active trachoma and infection six months after the treatment. Antibiotic treatment reduced the prevalence of chlamydial infection more than it did the level of clinically active trachoma.</td></tr></table>

**Figure 1:** A structured abstract with an overt organization, extracted from Scielo (fr-en) test set, a bio-medical corpus. In this example, the *Introduction* is missing, and an *Objective* part is present; the *Result* part is labeled 'FINDINGS'.

as there was a large degree of variance in subheadings and structures across corpora (see § 4.1.3). To incorporate the standard IMRaD format, we thus mapped each subheading to the corresponding IMRaD label using a system of tags displayed in Table 1. For instance, *Subjects and Methods* was mapped onto *Material and Methods* (\<MaM\>), *Pedagogical objectives* may be replaced with *Objectives* (\<OBJ\>), *Novel finding with Results* (\<RES\>), and so on. We also found it appropriate to merge some sections, such as *Discussion*, which was merged with *Conclusion*.

Document level corpora (see § 4.1.3) were first retrieved in XML format and structured into sub-sections based on subheading information. Each section was split<fnref n="2" /> into sentences and sentence aligned using Microsoft bilingual aligner [32].

<footnote n="2">²https://github.com/berkmancenter/mediacloud-sentence-splitter</footnote>
The identification and standardization of the subheading information was a tedious process, involving a lot of rule-based processing and human intervention to take the variability of sub-headings into account. In order to reconstruct the fully parallel versions with subheadings, we also had to reinsert explicit headings in the source or target files when they were missing. Also, note that this information was not available for all biomedical abstracts.

In this case, we tagged every line but the title with a generic tag (<ABS>).

## 3.2. A hierarchy of tags

Building state-of-the-art NMT systems for abstracts in the biomedical domain requires to consider multiple sources of parallel data, opportunistically collecting texts from a variety of genres and domains [33]. Our own training data, presented in Section 4.1, is accordingly very diverse, comprising in-domain parallel and out-of-domain parallel corpora, texts automatically retrieved parallel collections, as well in-domain monolingual data that is automatically back-translated. Some are made of lists of isolated sentences, while others retain the document information. Even within the in-domain data, some texts precisely match the genre of the test set (scientific abstracts), whereas others can be quite remote (eg. regulatory documents from the EMA<fnref n="3" />). This means that for most of our training data, the document structure will often be either unknown, or will not correspond to the scientific genre and will not obey the IMRaD pattern.

In order to reflect this diversity and provide tags for all parallel sentences in our corpus, we extended our tagging scheme with two additional levels of tagging. We then use hierarchical tags of the form (<1> <2> <3>), to incorporate a

<table><caption>**Table 1:** Standardized section heading and the corresponding tags</caption><tr><td>Title</td><td>&lt;H1&gt;</td></tr><tr><td>Introduction</td><td>&lt;INT&gt;</td></tr><tr><td>Objectives</td><td>&lt;OBJ&gt;</td></tr><tr><td>Material and Methods</td><td>&lt;MaM&gt;</td></tr><tr><td>Results</td><td>&lt;RES&gt;</td></tr><tr><td>Conclusion</td><td>&lt;CON&gt;</td></tr></table>

<footnote n="3">³European Medicines Agency, formerly known as EMEA.</footnote>
<table><caption>**Figure 2:** Examples of fully tagged sentences from Scielo, a bio-medical corpus. Tags appear as prefixes in the source sentence.</caption><tr><td>&lt;M&gt;</td><td>&lt;SCI&gt;</td><td>&lt;H1&gt;</td><td>Comparison of two azithromycin distribution strategies for controlling trachoma in Nepal.</td></tr><tr><td>&lt;M&gt;</td><td>&lt;SCI&gt;</td><td>&lt;OBJ&gt;</td><td>OBJECTIVE</td></tr><tr><td>&lt;M&gt;</td><td>&lt;SCI&gt;</td><td>&lt;OBJ&gt;</td><td>The study compares the effectiveness of two strategies for distributing azithromycin in an area with (...)</td></tr><tr><td>&lt;M&gt;</td><td>&lt;SCI&gt;</td><td>&lt;MaM&gt;</td><td>The two strategies investigated were the use of azithromycin for 1) mass treatment of all children, or 2) (...)</td></tr><tr><td>&lt;M&gt;</td><td>&lt;SCI&gt;</td><td>&lt;RES&gt;</td><td>FINDINGS</td></tr><tr><td>&lt;M&gt;</td><td>&lt;SCI&gt;</td><td>&lt;RES&gt;</td><td>Mass treatment of children was slightly more effective in terms of decreasing the prevalence of clinically (...)</td></tr><tr><td>&lt;M&gt;</td><td>&lt;SCI&gt;</td><td>&lt;CON&gt;</td><td>CONCLUSION</td></tr><tr><td>&lt;M&gt;</td><td>&lt;SCI&gt;</td><td>&lt;CON&gt;</td><td>Both strategies appeared to be effective in reducing the prevalence of clinically active trachoma and (...)</td></tr><tr><td>&lt;M&gt;</td><td>&lt;SCI&gt;</td><td>&lt;CON&gt;</td><td>Antibiotic treatment reduced the prevalence of chlamydial infection more than it did the level of clinically (...)</td></tr></table>

190 multi-level information about the domain, corpus and section for each sentence. The first level of tags distinguishes between out-of-domain data (\<G\>), and in-domain data (\<M\>). The second level of tag aims to distinguish between data sources, hence the use of one dedicated tag for each corpus, except for data collected with Information Retrieval (IR) techniques and monolingual data, which are respectively tagged with \<IR\> and \<BT\> [34, 35]. Finally, the third tag either corresponds to the structural information as described above, when it is available; otherwise all remaining sentences from other corpora were simply tagged with an "unspecified subheading" label (\<US\>).

195

Fully tagged sentences sampled from the Scielo test document displayed on Figure 1 are reproduced on Figure 2. In this example, tags are prefixed to the source sentence: the first tag (\<M\>) specifies the bio-medical domain, the second tag identifies the corpus (\<SCI\>), and the third tag corresponds to the subsection of each sentence.
### 3.3. Using tags in MT

Having defined a tagging scheme, the next question concerns its use in machine translation. A first alternative is to use tags as words and insert them as prefix on the source side. This assumes that tags are always known prior to translation, and will have the effect to differentiate the contextual representations of source words, enabling the system to disambiguate the translation of polysemous words, whenever this is beneficial for the training loss. The same effect will be observed on the target side, thanks to the cross-lingual attention mechanism alluded to above. It is further possible to explicitly amplify this effect by injecting the tag information into the input representation of every token (eg. [30, 36] for domain information, or [37] for language information).

The alternative is to inject the tags as a prefix of the *target* sentence. If we assume that tags are always observed, the expected effect is roughly similar to having tags on the source side. The main difference is that source token representations no longer depend on the tag value. We have not studied this approach, but have instead resorted to another way to use target side tags: to process them as if they were regular target words. The system will then be trained to predict their correct values using only the source side information (and the previous tag(s)). In this setting, source side representations will be adjusted to correctly generate tags and to extract features that are useful for this prediction [38]. With forced decoding [39], this approach can still be used when tags are observed. In addition, it can also be used to generate tag-dependent translations, for unstructured abstracts, with unobserved tags. These two schemes are considered in our experiments and help to better analyse the usefulness of the structural tags. Similar analyses, for a variety of tags, are in [40, 41, 42]. They suggest that when tags are observed, having them on the source side yields better results. Note that these studies only consider one level of tagging, where we consider three.

To sum up, three scenarios will be contrasted below: (i) observed source side tags, (ii) observed target side tags (forced decoding), (iii) predicted target side tags: (i) and (ii)-(iii) differ in the training scheme, while (ii) and (iii) only differ
<table><caption>**Table 2:** Data sources used in our study. Test sets of both directions with only document information were combined.</caption><thead><tr><th rowspan="2">Corpus</th><th colspan="3">Wrds (M)</th><th rowspan="2">Corpus</th><th colspan="3">Wrds (M)</th></tr><tr><th>English</th><th>French</th><th>Sents.</th><th>English</th><th>French</th><th>Sent.</th></tr></thead><tbody><tr><td colspan="8"><strong>In Domain</strong></td></tr><tr><td>Edp</td><td>0.04</td><td>0.04</td><td>3.3 K</td><td>Monolingual</td><td></td><td></td><td></td></tr><tr><td>Medline titles</td><td>5.97</td><td>6.43</td><td>0.62 M</td><td>Med_Fr</td><td>1.34</td><td>1.63</td><td>0.06 M</td></tr><tr><td>Medline abstracts</td><td>1.23</td><td>1.44</td><td>57.6 K</td><td>IsTex_Fr</td><td>6.92</td><td>7.84</td><td>0.42M</td></tr><tr><td>Scielo</td><td>0.17</td><td>0.21</td><td>12 K</td><td>Lissa_Fr</td><td>8.79</td><td>7.70</td><td>0.33 M</td></tr><tr><td>Cochrane-Reference</td><td>2.23</td><td>2.74</td><td>0.12 M</td><td>Med_En</td><td>3.40</td><td>4.02</td><td>0.22M</td></tr><tr><td>Cochrane-PE</td><td>0.43</td><td>0.53</td><td>25.6 K</td><td>Development</td><td></td><td></td><td></td></tr><tr><td>Cochrane-GooglePE</td><td>0.63</td><td>0.77</td><td>37.8 K</td><td>Medline 18</td><td>5.7K</td><td>6.9K</td><td>265</td></tr><tr><td></td><td></td><td></td><td></td><td>Medline 19</td><td>9.8K</td><td>12.4K</td><td>537</td></tr><tr><td>Total &lt;Section&gt;</td><td></td><td></td><td>0.87 M</td><td>Test</td><td></td><td></td><td></td></tr><tr><td>Ufal</td><td>89.5</td><td>100.3</td><td>2.62 M</td><td>Scielo (wmt16)</td><td>0.14</td><td>0.17</td><td>6475</td></tr><tr><td>Taus</td><td>20.1</td><td>23.2</td><td>0.88 M</td><td>Medline 20</td><td>25K</td><td>28K</td><td>997</td></tr><tr><td>Mlia</td><td>19.0</td><td>23.0</td><td>1.0M</td><td>Newstest</td><td>93K</td><td>0.1</td><td>3003</td></tr><tr><td>IR Retrieved</td><td>13.2</td><td>14.7</td><td>3.6M</td><td colspan="3">Out Domain</td></tr><tr><td>Total-in-dom&lt;US&gt;</td><td></td><td></td><td>8.2 M</td><td>Out-domain&lt;US&gt;</td><td>1139</td><td>1292</td><td>35M</td></tr></tbody></table>

235 in the way inference is performed.

# 4. Experimental Setup

## 4.1. Corpora and Pre-processing

We train our baseline systems on a collection of in-domain biomedical texts as well as out-of-domain parallel corpus, and use standard benchmarks from past studies for development and tests. Details are in Table 2.

### 4.1.1. Parallel corpora

We gathered parallel and monolingual corpora available for English-French in the biomedical domain. The former included the biomedical texts provided by the WMT’20 organizers: Edp, Medline abstracts and titles [43], Scielo [44] and the Ufal Medical corpus<fnref n="4" /> consisting of Cesta, Ecdc, Emea (OpenSubtitles), PatTR Medical and Subtitles. Other sources of in-domain data are the

<footnote n="4">⁴https://ufal.mff.cuni.cz/ufal_medical_corpus</footnote>
Cochrane bilingual parallel corpus [45]<fnref n="5" />, the Taus Corona Crisis corpus⁶ and the Mlia Covid corpus⁷. We also included additional in-domain data selected using Information Retrieval (IR) techniques from general domain corpora including News-Commentary, Books and Wikipedia corpus obtained from the Open Parallel Corpus (OPUS) [46]. These were selected using the data selection scheme described in [47]. Medline titles were used as queries to find relevant sentences. We included the 2-best sentences returned from the IR pipeline into our additional corpus [48].

250

Our out-of-domain corpora include the parallel data provided by the WMT’14 campaign for French-English: Gigafr-en, Common Crawl, Europarl, News Commentary and the UN corpora.

255

For development purposes, we used Medline testsets for WMT’18 [49] and 19 [50] bio-medical MT task, while Medline 20, Scielo (WMT’16) and Edp (WMT’17) were used as internal test data.⁸ Test sets for both directions were combined to get larger test sets and are used to evaluate systems in both directions. We only used the parts with document structure, i.e. sentences with <ABS> were not included. The combined test set for Scielo and Medline had 8,805 and 1315 sentences each, after removing the abstract sentences we get the test sets with 6475 and 997 sentences for Scielo and Medline20 respectively.

260

265

### 4.1.2. Monolingual sources

The backtranslation of monolingual sources has often been effectively used to cater for parallel corpus shortage in the bio-medical domain [51, 33]. We also augment our training data with backtranslations using both French and English in-domain monolingual corpora.

270

Supplementary French data from three monolingual sources were collected

<footnote n="5">⁵https://github.com/fyvo/CochraneTranslations/
⁶https://md.taus.net/corona
⁷http://eval.covid19-mlia.eu/task3/
⁸These test sets were sentence-aligned with in-house tools and can be retrieved from https://github.com/fyvo/WMT-Biomed-Test.</footnote>
from public archives: 1) abstracts of medical papers published by Elsevier from
the Lissa portal;<fnref n="9" /> 2) abstracts from the national ISTEX archive;<fnref n="10" /> 3) a collection
of research articles collected from various sources<fnref n="11" /> [52] henceforth referred to as
Med_Fr. These documents were automatically translated into English with an
NMT system trained only on bio-medical corpora, with a reported BLEU score
of 33.6 on Medline20 testset.

The English side of the Medline German and Spanish corpora distributed for
WMT/Biomed 2021¹² is used as supplementary English data for backtranslation.
Duplicate documents were removed based on the document id information. For
these abstracts, the internal structure of documents is often available and has
been tagged as for the parallel data. These texts were then split into sentences<fnref n="13" />
and translated into French using an NMT system trained on all bio-medical
corpora with a BLEU score of 36.4 on Medline20 testset.

4.1.3. *Structured sources*

The document structure is only available for a subset of the training data
corresponding to the following sources:

1. *Medline and Scielo:* These included both abstracts with and without subheadings, sometimes also missing a title line. There were a total of 189 subheadings in these corpora, including variations in case, word order and morphology. Examples subheadings include: *Presenting Concerns of the Patient, Sources of Information, Novel finding, Study Selection, PURPOSE OF REVIEW, Purpose of Review, Purpose of review* etc.

2. Edp: These documents included abstracts with and without subheadings, with a majority having titles. Total 45 subheadings with variations in case and word order were identified such as: *Case report, Observation, Subjects and Methods, Commentary, Pedagogical objectives, etc.*

<footnote n="9">⁹https://www.lissa.fr/dc/#env=lissa</footnote>

<footnote n="10">¹⁰https://www.istex.fr/</footnote>

<footnote n="11">¹¹https://crtt.univ-lyon2.fr/les-corpus-medicaux-du-crtt-613310.kjsp</footnote>

<footnote n="12">¹²http://statmt.org/2021</footnote>

<footnote n="13">¹³https://pypi.org/project/sentence-splitter/</footnote>
300

3. *Cochrane:* This corpus contains very structured summaries with 10 distinct subheadings according to the Cochrane nomenclature. These included: *abs selection criteria*, *abs search strategy*, *abs data collection*, *summary title* etc. <fnref n="14" />

305

For the test and development corpora, when the structure was available, a manual alignment was performed based on the sentence alignment file provided by WMT organizers. ¹⁵ Alignment operations involved inserting an empty line after each subsection, deleting unaligned sentences and merging sentences as per alignment information. Statistics regarding the numbers of structured documents in the train, test and development corpora are in Table 2.

## 4.2. Translation systems

310

Our translation systems mostly use the basic Transformer architecture [27]. They all rely on Meta's seq-2-seq library (fairseq) [53] with parameter settings borrowed from the transformer_wmt_de_en recipe. ¹⁶ The ReLU activation function was used in all encoder and decoder layers. We optimize with Adam [54], set up with a maximum learning rate of 0.0005 and an inverse square root decay schedule, as well as 4000 warmup updates. All corpora are segmented into subword units using Sentence Piece [55] with a vocabulary of 32K units. These units were learned on the union of all in-domain corpora. We share the decoder input and output embedding matrices.

315

Models are trained with mixed precision and a batch size of 4096 tokens on 4 V100 GPUs for 300k updates until convergence, where convergence is measured on the development set BLEU score. Final parameters are chosen based on the best score on the development set (Medline18, 19) and the corresponding scores for that checkpoint are reported on Medline20 test set. Translations were generated using a beam size of 4. For TRGTAG systems tags were removed before

320

<footnote n="14">¹⁴Recommandations for writing Cochrane's systematic reviews are in https://training.cochrane.org/handbook/.

¹⁵https://github.com/biomedical-translation-corpora/corpora

¹⁶https://fairseq.readthedocs.io/en/latest/models.html</footnote>
automatic evaluation. Evaluation was performed using SacreBleu [56] using BLEU [57], TER [58] and chrF [59] metrics.

For fine-tuned systems, the process starts with models converged systems. Training then resumes using a selected portion of the train data, with the same meta parameters and objective as for the base systems. In our results, names of fine-tuned systems are post-fixed with *-ft.

## 5. Document Structure Information and MT performance

To evaluate the impact of document structure information on NMT, we built systems with tags either on the source or target sides and compared them with baseline NMT systems. Scores show that incorporating tags substantially improves machine translation performance; this improvement is more prominent when tags are used on the source side to initialize the encoder input stream. These results are tabulated in Table 3 on two bio-medical test sets, Medline20 and Scielo for standard MT systems (All), source-tagged systems (SRCTAG) and destination-tagged systems (TRGTAG). Baseline systems use all general domain and biomedical corpora totalling 41.7M sentences for EN-FR and 40.9M sentences for FR-EN. Each of these systems is then further fine-tuned on bio-medical corpora (5.7M sentences for EN-FR, and 4.9M sentences for FR-EN).

We use three MT metrics to evaluate the systems in this section. These include BLEU which uses n-gram precision and Brevity penalty, TER which gives scores based on edit operations and chrF which uses F-score statistic for character n-gram matches that a hypothesis requires to match the reference translation. Since TER is an error rate, a decrease in score signifies improvement, while for BLEU and chrF increase denotes improvement. We see that on average the three metrics agree with each other in the general trends exhibited by the systems, but BLEU is more prominent, thus we will present further analysis based on BLEU scores.

A general trend is that systems with extra document information perform substantially better than vanilla NMT systems. We find that it seems better to
<table><caption>**Table 3:** BLEU, TER and chrF scores computed on Medline20 and Scielo test sets with and without tags, where we contrast tags in source (SRCTAG), reference target tags (TRGTAG with forced decoding) and predicted target tags (TRGTAG). +ft identifies the corresponding fine-tuned systems. Note that for TER, a decrease in score signifies improvement, while for BLEU and chrF increase denotes improvement. The best result in each column is in bold and best score for predicted vs. reference tags are italicized.</caption><thead><tr><th rowspan="3"></th><th colspan="6">Medline20</th><th colspan="6">Scielo</th></tr><tr><th colspan="3">EN-FR</th><th colspan="3">FR-EN</th><th colspan="3">EN-FR</th><th colspan="3">FR-EN</th></tr><tr><th>BLEU</th><th>TER</th><th>chrF</th><th>BLEU</th><th>TER</th><th>chrF</th><th>BLEU</th><th>TER</th><th>chrF</th><th>BLEU</th><th>TER</th><th>chrF</th></tr></thead><tbody><tr><td>All</td><td>39.3</td><td>50.2</td><td>65.5</td><td>40.5</td><td>49.0</td><td>67.0</td><td>36.4</td><td>52.3</td><td>63.4</td><td>36.3</td><td>53.2</td><td>65.6</td></tr><tr><td>+ft</td><td>40.3</td><td>48.9</td><td>66.3</td><td>43.1</td><td>46.5</td><td>68.7</td><td>36.0</td><td>52.0</td><td>63.3</td><td>36.5</td><td>52.6</td><td>66.0</td></tr><tr><td>SRCTAG</td><td>40.2</td><td>49.0</td><td>66.3</td><td>40.6</td><td>48.7</td><td>67.1</td><td>36.9</td><td>52.0</td><td>63.8</td><td>37.0</td><td>51.7</td><td>66.0</td></tr><tr><td>+ft</td><td>41.5</td><td>48.4</td><td>66.7</td><td>44.0</td><td>45.3</td><td>69.0</td><td>36.8</td><td>51.8</td><td>63.8</td><td>38.6</td><td>50.1</td><td>66.7</td></tr><tr><td>TRGTAG</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Reference</td><td>40.1</td><td>49.9</td><td>65.8</td><td>40.1</td><td>49.9</td><td>67.1</td><td>37.0</td><td>51.9</td><td>63.9</td><td>36.2</td><td>53.5</td><td>65.6</td></tr><tr><td>+ft</td><td>41.0</td><td>48.7</td><td>66.5</td><td>43.2</td><td>46.3</td><td>68.8</td><td>36.8</td><td>51.6</td><td>63.9</td><td>37.7</td><td>51.4</td><td>66.3</td></tr><tr><td>Predicted</td><td>39.6</td><td>50.2</td><td>65.6</td><td>40.0</td><td>50.1</td><td>67.0</td><td>36.1</td><td>52.3</td><td>63.3</td><td>35.1</td><td>54.8</td><td>65.3</td></tr><tr><td>+ft</td><td>40.3</td><td>49.2</td><td>66.3</td><td>42.6</td><td>46.9</td><td>68.6</td><td>35.6</td><td>52.3</td><td>63.2</td><td>35.9</td><td>53.5</td><td>65.8</td></tr></tbody></table>

introduce the tags on the source than on the target side, similar to [40, 41, 42]. We see an average improvement of 0.92 and 0.46 BLEU points on the source and target side tags respectively. Further, in TRGTAG setting, using reference tags is consistently better than predicting them, with more than 1 BLEU point difference between these settings for Scielo. These observations carry over for fine-tuned systems. We also see that the effect of fine-tuning is generally positive, with the exception for Scielo systems with TRGTAG for EN-FR.

## 5.1. Ablation Experiment

An ablation experiment was conducted in the SRCTAG setting for French-English direction to isolate the impact of document structure information. For this, we trained two models with one tag each: the first model with only domain tags, the second with only document structure (section) tags. This is in contrast to our experiments with the hierarchical tagging scheme where 3 dimensions of information are used to condition the translation output. The rationale behind this ablation experiment is to isolate the effect of other dimensions.

The section and domain tagged systems had a BLEU score of 36.7 and
<table><caption>**Table 4:** Ablation experiment: Output from model trained with only document structure tags (a) and only domain tags (b).</caption><tr><td>1 - src</td><td>Compte tenu du manque de renseignements concernant ce modèle modifié, nous avons procédé à une étude pour en déterminer la survie et le rendement à court et à moyen terme.</td></tr><tr><td>&lt;INT&gt;</td><td>Given the lack of information about this modified model, we conducted a study to determine short- and medium-term survival and performance.</td></tr><tr><td>&lt;M&gt;</td><td>Given the lack of information on this modified model, we conducted a study to determine its survival and performance in the short and medium term.</td></tr><tr><td>2 - src</td><td>S'il existe de nombreuses approches <b>chiropratiques</b>, deux types de <b>chiropracteurs</b> peuvent-être identifiés; ceux s'intéressant aux troubles musculo-squelettiques et ceux souhaitant prendre en charge aussi des troubles non musculo-squelettiques.</td></tr><tr><td>&lt;INT&gt;</td><td>While there are many <b>chiropractor</b> approaches, two types of <b>chiropractors</b> may be identified; those interested in musculoskeletal disorders and those <b>wishing to also manage non-mechanic disorders</b>.</td></tr><tr><td>&lt;M&gt;</td><td>If there are many <b>chiropractical</b> approaches, two types of <b>chiropractors</b> may be identified; those interested in musculoskeletal disorders and those <b>interested in the management of non-mechanic disorders as well</b>.</td></tr><tr><td>3 - src</td><td><b>Il est préoccupant de constater que</b> les étudiants qui adhèrent au modèle de la subluxation soient prêts à intégrer ces opinions dans leurs futures <b>prises en charge</b>; souhaitant proposer des ajustements <b>chiropratiques</b> aux patients asymptomatiques.</td></tr><tr><td>&lt;CON&gt;</td><td><b>Concerningly</b>, students who adhere to the <b>subluxe</b> model are prepared to <b>integrate</b> these views into their future <b>care</b>; wishing to propose <b>chiropractical</b> adjustments to asymptomatic patients.</td></tr><tr><td>&lt;M&gt;</td><td><b>It is worrying to note that</b> students who adhere to the <b>subluxation</b> model are prepared to <b>incorporate</b> these views into their future <b>management</b>; wishing to propose <b>chiropractical</b> adjustments to asymptomatic patients.</td></tr><tr><td>4 - src</td><td>Les symptômes courants sont des maux de tête, des troubles de la vue, des acouphènes pulsatiles et un oedème papillaire.</td></tr><tr><td>&lt;INT&gt;</td><td>Common symptoms are headaches, <b>visual</b> disorders, <b>perfumant</b> psychosis, and <b>hematoma</b> edema.</td></tr><tr><td>&lt;M&gt;</td><td>Common symptoms are headache, <b>vision</b> disorders, <b>puffing</b> drowsiness, and <b>facial</b> edema.</td></tr></table>
36.2 respectively (at 360M updates). Section tagging gave an improvement of 0.5 BLEU points. We observe the impact on output in Table 4 where sentences 1-2 display a common pattern: the difference in grammatical preference between models trained with only section information vs only domain information. Sentences 3-4 highlight variance in lexical choice for the two models. The difference in average sentence length of the output of the two systems was negligible (0.26 words). From these results, we conclude that compounded effect of hierarchical tagging is more pronounced as compared to using only one level of tagging.

370

375

## 6. Joint Prediction and Translation

In this section, we evaluate TRGTAG systems in detail. Recall that for each sentence the model has to successively predict three tags, corresponding respectively to the domain, corpus and the section, before generating the output translation. We thus systematically evaluate and contrast (i) the accuracy of the prediction of each tag, and (ii) the impact of having a correct vs. incorrect prediction on the BLEU score. To do so, we use the systems trained with target side tags as prefix of the reference translation. This setting allows us to study how well tags can be automatically predicted from the source sentence and the resulting impact on the final translation.

380

385

For this, we force decode each test sentence four times with four prefixes of increasing size: pre0 is the empty prefix condition, where all tags are automatically generated, while pre3 initializes the decoder with the three reference tags; pre1 and pre2 denote the two intermediary settings, respectively with one and two correct tags. The former two conditions respectively correspond to decoding with predicted or reference tags in Table 3, the latter two are novel.

390

### 6.1. Using reference vs. predicted tags

In our results, we bin the sentences based on tag predictions. Bin 000 contains sentences for which that all three tags are wrongly predicted; similarly

395
<table><caption>**Table 5:** Evaluating the difference in translation quality of sentences with correctly predicted sections vs. wrongly predicted sections (Domain, Corpus, Section). 000 indicates that the three tags are wrongly predicted, 111 that they are all correct.</caption><thead><tr><th colspan="2">All</th><th colspan="6">EN-FR: BLEU Scores (Percentage of sentences)</th></tr><tr><th>Prefix</th><th>Testsets<br>(# Sent.)</th><th>Global<br>Scores</th><th>000</th><th>001</th><th>100</th><th>101</th><th>110</th><th>111</th></tr></thead><tbody><tr><td>Pre0</td><td></td><td>36.8</td><td>31.0 (1.5%)</td><td>-</td><td>36.7 (66.5%)</td><td>38.1 (21%)</td><td>29.2 (0.3%)</td><td>37.6 (11%)</td></tr><tr><td>Pre1</td><td>scielo</td><td>36.8</td><td>-</td><td>-</td><td>36.6 (68%)</td><td>37.9 (21%)</td><td>29.2 (0.3%)</td><td>37.6 (11%)</td></tr><tr><td>Pre2</td><td>(6475)</td><td><strong>37.5</strong></td><td>-</td><td>-</td><td>-</td><td>-</td><td>35.8 (39%)</td><td>39.6 (61%)</td></tr><tr><td>Pre3</td><td></td><td><strong>37.5</strong></td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>37.5 (100%)</td></tr><tr><td>Pre0</td><td></td><td>41.0</td><td>44.7 (1.1%)</td><td>-</td><td>39.9 (59.5%)</td><td>44.5 (15.2%)</td><td>39.8 (7.7%)</td><td>47.2 (16.5%)</td></tr><tr><td>Pre1</td><td>medline20</td><td>41.0</td><td>-</td><td>-</td><td>39.8 (60%)</td><td>44.4 (15.5%)</td><td>39.8 (7.7%)</td><td>48.7 (17%)</td></tr><tr><td>Pre2</td><td>(997)</td><td><strong>41.4</strong></td><td>-</td><td>-</td><td>-</td><td>-</td><td>38.2 (55%)</td><td>46.0 (45%)</td></tr><tr><td>Pre3</td><td></td><td>40.1</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>40.1 (100%)</td></tr><tr><th colspan="2">+fine tuning on Biomed</th><th colspan="6"></th></tr><tr><td>Pre0</td><td></td><td>36.5</td><td>-</td><td>-</td><td>35.7 (49%)</td><td>38.5 (48%)</td><td>29.0 (0.5%)</td><td>39.4 (2.4%)</td></tr><tr><td>Pre1</td><td>scielo</td><td>36.5</td><td>-</td><td>-</td><td>35.7 (49%)</td><td>38.5 (48%)</td><td>29.0 (0.5%)</td><td>39.4 (2.4%)</td></tr><tr><td>Pre2</td><td>(6475)</td><td><strong>37.7</strong></td><td>-</td><td>-</td><td>-</td><td>-</td><td>35.3 (24.5%)</td><td>39.1 (75.5%)</td></tr><tr><td>Pre3</td><td></td><td>37.6</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>37.6 (100%)</td></tr><tr><td>Pre0</td><td></td><td>42.0</td><td>-</td><td>-</td><td>40.2 (45%)</td><td>45.8 (15%)</td><td>39.8 (10%)</td><td>46.2 (30%)</td></tr><tr><td>Pre1</td><td>medline20</td><td>42.0</td><td>-</td><td>-</td><td>40.2 (45%)</td><td>45.8 (15%)</td><td>39.8 (10%)</td><td>46.2 (30%)</td></tr><tr><td>Pre2</td><td>(997)</td><td><strong>42.4</strong></td><td>-</td><td>-</td><td>-</td><td>-</td><td>39.6 (49%)</td><td>45.6 (51%)</td></tr><tr><td>Pre3</td><td></td><td>41.9</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>41.9 (100%)</td></tr><tr><th colspan="2">All</th><th colspan="6">FR-EN:BLEU Scores/Percentage of sentences</th></tr><tr><th>Prefix</th><th>Testsets</th><th>Scores</th><th>000</th><th>001</th><th>100</th><th>101</th><th>110</th><th>111</th></tr><tr><td>Pre0</td><td></td><td>35.5</td><td>45.8 (0.2%)</td><td>-</td><td>35.6 (89.8%)</td><td>34.5 (10%)</td><td>-</td><td>-</td></tr><tr><td>Pre1</td><td>scielo</td><td>35.5</td><td>-</td><td>-</td><td>35.6 (89.8%)</td><td>34.5 (10%)</td><td>-</td><td>-</td></tr><tr><td>Pre2</td><td>(6475)</td><td><strong>36.8</strong></td><td>-</td><td>-</td><td>-</td><td>-</td><td>33.2(39%)</td><td>39.5(61%)</td></tr><tr><td>Pre3</td><td></td><td><strong>36.8</strong></td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>36.8 (100%)</td></tr><tr><td>Pre0</td><td></td><td>40.5</td><td>46.8 (0.4%)</td><td>-</td><td>40.4 (80%)</td><td>55.1 (2.6%)</td><td>31.5 (13%)</td><td>44.2 (4%)</td></tr><tr><td>Pre1</td><td>medline20</td><td>40.5</td><td>-</td><td>-</td><td>40.5(80%)</td><td>55.1 (2.6%)</td><td>31.5 (13%)</td><td>44.2 (4%)</td></tr><tr><td>Pre2</td><td>(997)</td><td>40.7</td><td>-</td><td>-</td><td>-</td><td>-</td><td>36.6 (60%)</td><td>47.0 (40%)</td></tr><tr><td>Pre3</td><td></td><td><strong>40.7</strong></td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>40.7 (100%)</td></tr><tr><th colspan="2">+fine tuning on Biomed</th><th colspan="6"></th></tr><tr><td>Pre0</td><td></td><td>36.6</td><td>-</td><td>-</td><td>37.0 (88.6%)</td><td>32.5 (10%)</td><td>36.4 (0.3%)</td><td>38.0 (1.5%)</td></tr><tr><td>Pre1</td><td>scielo</td><td>36.6</td><td>-</td><td>-</td><td>37.0 (88.6%)</td><td>32.5 (10%)</td><td>36.4 (0.3%)</td><td>38.0 (1.5%)</td></tr><tr><td>Pre2</td><td>(6475)</td><td>38.7</td><td>-</td><td>-</td><td>-</td><td>-</td><td>34.9 (23%)</td><td>40.6 (77%)</td></tr><tr><td>Pre3</td><td></td><td><strong>38.8</strong></td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>38.8(100%)</td></tr><tr><td>Pre0</td><td></td><td>43.6</td><td>-</td><td>-</td><td>42.6 (71.5%)</td><td>59.1 (4%)</td><td>35.4 (15%)</td><td>48.6 (9.5%)</td></tr><tr><td>Pre1</td><td>medline20</td><td>43.6</td><td>-</td><td>-</td><td>42.6 (71.5%)</td><td>59.1 (4%)</td><td>35.4 (15%)</td><td>48.6 (9.5%)</td></tr><tr><td>Pre2</td><td>(997)</td><td><strong>44.3</strong></td><td>-</td><td>-</td><td>-</td><td>-</td><td>39.1 (46%)</td><td>49.9 (54%)</td></tr><tr><td>Pre3</td><td></td><td><strong>44.3</strong></td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>44.3 (100%)</td></tr></tbody></table>
bin 111 contains sentences with three correct predictions, 101 those with a correct prediction for the first and third tags and so on. To measure the translation quality, BLEU scores are computed for each subset of sentences, excluding as before tags in the score computation. Condition pre0 is when we predict all three tags, while pre3, pre2 and pre1 involve some reference tag(s). Table 5 reports the BLEU scores and percentage of sentences in each bin.

400

We can make the following observations that apply for the two language directions. First, it turns out that predicting the first tag (domain) is easy which also explains the small gap in BLEU scores between pre0 and pre1. Having the second tag (corpus) right is the most challenging part, with an error rate higher than 75% for the non fine-tuned systems. This is because some of our in domain corpora are very close (eg. Cochrane and Medline). It remains difficult even after fine-tuning, with an error rate way above 60%. The third tag (section headings), is much easier — assuming the other tags are correct, we achieve accuracy higher than 50% (without fine-tuning) and higher than 60% with fine-tuning. When we get the corpus tag wrong, we are likely to fail for section prediction (see 100 vs. 101), i.e. if the system errs on corpus it is likely to err also on the section tag. The main reason for this is data imbalance with just 8.7% in-domain sentences having document structure information (see Table 2).

405

410

BLEU-wise, we see overall a general improvement when we move from predicted tags (**pre0**) to correct tags (**pre3**); when the corpus is known, section prediction is relatively easy explaining why **pre2** and **pre3** are always very close, and in some cases it even seems slightly better to let the system use predicted sections headings than use the correct ones.

415

***Structure vs No structure tags.*** As pre2 and pre3 scores are always very close, to better evaluate the impact of using reference section tags, we design another contrast and force decode all the test-sets with a 'generic' section tag instead of the actual IMRaD code. Results are in Table 6 and again contrast the effect of using source vs. target tags.

420
Having structure tags in source slightly improves BLEU for the fine-tuned systems on Medline (+0.6 and +0.7 respectively for EN-FR and FR-EN). All other comparisons show little, insignificant variations. The same small variations are observed for TRGTAG systems, where we even observe that the generic tag yields a tiny improvement over the other condition. These small differences

<table><caption>**Table 6:** Using specific or generic section tags</caption><thead><tr><th rowspan="2">Test set</th><th colspan="2">EN-FR</th><th colspan="2">FR-EN</th></tr><tr><th>Spec.</th><th>Gen.</th><th>Spec.</th><th>Gen.</th></tr></thead><tbody><tr><td><u>S<sub>RCTAG</sub></u></td><td></td><td></td><td></td><td></td></tr><tr><td>Scielo</td><td><b>37.6</b></td><td>37.5</td><td><b>37.6</b></td><td>37.5</td></tr><tr><td>+ft</td><td>37.6</td><td>37.6</td><td>39.7</td><td><b>39.8</b></td></tr><tr><td>Medline20</td><td><b>41.8</b></td><td>41.4</td><td>41.6</td><td>41.6</td></tr><tr><td>+ft</td><td><b>42.6</b></td><td>42.0</td><td><b>45.4</b></td><td>44.7</td></tr></tbody></table><table><thead><tr><th rowspan="2">TRGTAG</th><th colspan="2">EN-FR</th><th colspan="2">FR-EN</th></tr><tr><th>Spec.</th><th>Gen.</th><th>Spec.</th><th>Gen.</th></tr></thead><tbody><tr><td>Scielo</td><td>37.5</td><td>37.5</td><td><b>36.8</b></td><td>36.7</td></tr><tr><td>+ft</td><td><b>37.7</b></td><td>37.6</td><td><b>38.8</b></td><td>38.6</td></tr><tr><td>Medline20</td><td>41.0</td><td><b>41.4</b></td><td>40.7</td><td><b>40.8</b></td></tr><tr><td>+ft</td><td>41.9</td><td><b>42.3</b></td><td><b>44.3</b></td><td>44.0</td></tr></tbody></table>

may be due to the training data distribution, as 75% and 48% sentences of Medline and Scielo respectively had <ABS> as the section tag. Overall, these results do hint at a slight improvement in BLEU when the more specific section is known, but given the small difference and the data imbalance, we abstain to make any conclusive statement.

For this analysis, we report results on the Scielo testset, which among all the biomedical testsets, has the largest number of documents annotated with structure information (cf. Section 4.1.3). Confusion matrices for EN-FR are shown in Figure 3, and the corresponding metric scores and FR-EN confusion matrices are given in Appendix A.

Section prediction results as shown in Table 7 resonate with our human analysis of lexical and grammar usage in different sections shown in Table 8. We analysed 50 most frequent words per section along with their POS tags (using nltk). Some words are shown in Table 8 to demonstrate that the word itself is an indicator of its' most probable section and their usage is enriched in particular sections, e.g. the words like *however* and *may* exhibit higher probability to be
<figure><img src="image_2.png" /><figcaption>**Figure 3:** Confusion matrices for the Scielo testset (EN-FR): from left to right pre0, pre1 and pre2. First row baseline, Second row fine-tuned systems</figcaption></figure>

phrased in *Introduction* and *Conclusion* sections, whereas the illustrative and concrete diction exhibited by the words like *evidence* and *outcomes* have been mostly used in {Conclusion, Results} and {Objectives, Material and Methods, Conclusion} sections respectively. Interestingly the use of comparative words like *but* and *difference* was mostly in *Results* section. Words like *conclusion* and *controlled* have exhibited the highest probability of usage only in *Conclusion* and *Material and Methods* sections respectively. The confusion matrices and prediction scores show appreciable capability of the model for predicting the sections. We see that *Title* (<H1>) was predicted with the highest precision as titles are mostly content words. *Introduction* (<INT>) was the most difficult to predict as it typically presents the broader picture and has high lexical variation. {Objectives, Material and Methods, Conclusion} were predicted with precision above 88.
<table><caption>**Table 7:** Precision, Recall and F1 Score on Scielo baseline and fine-tuned systems for systems for section prediction (pre2).</caption><thead><tr><th rowspan="3">Section</th><th colspan="6">EN-FR</th><th colspan="6">FR-EN</th></tr><tr><th colspan="3">Baseline</th><th colspan="3">Fine-tuned</th><th colspan="3">Baseline</th><th colspan="3">Fine-tuned</th></tr><tr><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th></tr></thead><tbody><tr><td>&lt;H1&gt;</td><td>1.00</td><td>0.55</td><td>0.71</td><td>1.00</td><td>0.80</td><td>0.89</td><td>0.97</td><td>0.46</td><td>0.62</td><td>1.00</td><td>0.86</td><td>0.92</td></tr><tr><td>&lt;OBJ&gt;</td><td>1.00</td><td>0.71</td><td>0.83</td><td>0.99</td><td>1.00</td><td>0.76</td><td>1.00</td><td>0.57</td><td>0.72</td><td>0.99</td><td>0.74</td><td>0.85</td></tr><tr><td>&lt;INT&gt;</td><td>0.33</td><td>0.12</td><td>0.18</td><td>1.00</td><td>0.12</td><td>0.22</td><td>0.20</td><td>0.20</td><td>0.20</td><td>0.50</td><td>0.20</td><td>0.29</td></tr><tr><td>&lt;MaM&gt;</td><td>0.93</td><td>0.65</td><td>0.77</td><td>0.89</td><td>0.82</td><td>0.85</td><td>0.86</td><td>0.85</td><td>0.85</td><td>0.88</td><td>0.87</td><td>0.87</td></tr><tr><td>&lt;CON&gt;</td><td>0.96</td><td>0.34</td><td>0.51</td><td>0.88</td><td>0.53</td><td>0.66</td><td>0.99</td><td>0.32</td><td>0.49</td><td>0.88</td><td>0.53</td><td>0.67</td></tr><tr><td>&lt;RES&gt;</td><td>0.81</td><td>0.75</td><td>0.78</td><td>0.85</td><td>0.82</td><td>0.83</td><td>0.88</td><td>0.72</td><td>0.79</td><td>0.87</td><td>0.81</td><td>0.84</td></tr><tr><td>accuracy</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.61</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.61</td><td>-</td><td>-</td><td>0.77</td></tr><tr><td>macro avg</td><td>0.72</td><td>0.45</td><td>0.54</td><td>0.80</td><td>0.55</td><td>0.62</td><td>0.70</td><td>0.45</td><td>0.53</td><td>0.73</td><td>0.57</td><td>0.63</td></tr><tr><td>weighted avg</td><td>0.92</td><td>0.61</td><td>0.72</td><td>0.91</td><td>0.75</td><td>0.82</td><td>0.93</td><td>0.61</td><td>0.71</td><td>0.91</td><td>0.77</td><td>0.83</td></tr></tbody></table>

When we predict all tags (pre0), systems have precision above 0.95 for *Conclusion, Title, Material and Methods, Objectives* and 0.85 for *Results*. In terms of accuracy, we see an improvement from 0.32 to 0.50 for EN-FR after fine-tuning and a slight improvement for FR-EN. As seen earlier, section prediction is better when the first two tags are known, i.e pre2, however

ever title prediction remains good in FR-EN direction even for pre0 and pre1. *Introduction* remains the most difficult section to predict but pre2 systems learn to predict well. Accuracy for the baseline systems is 0.61, which increases for fine tuned systems to 0.75 and 0.77 for EN-FR and FR-EN respectively.

## 6.3. Qualitative Analysis: Tag conditioned translation variants

In this section, we study how tags affect the lexical and grammatical choice in automatic translations. For this analysis, we analysed the 1167 translations of all the titles (\<H1\> tags) in the Scielo corpus and compared two cases: (a)

<table><caption>**Table 8:** Word occurrence in sections from 50 most frequent words only. POS tags were generated using nltk pos tagger.</caption><thead><tr><th>Word</th><th>Occurrence in Sections</th></tr></thead><tbody><tr><td>conclusion_NNS</td><td>CON</td></tr><tr><td>controlled_JJ</td><td>MAM</td></tr><tr><td>but_CC</td><td>RES</td></tr><tr><td>difference_NN</td><td>RES</td></tr><tr><td>however_RB</td><td>INT CON</td></tr><tr><td>may_MD</td><td>INT CON</td></tr><tr><td>trial_NNS</td><td>MAM RES</td></tr><tr><td>evidence_NN</td><td>RES CON</td></tr><tr><td>people_NNS</td><td>H1 OBJ MAM</td></tr><tr><td>outcomes_NNS</td><td>OBJ MAM CON</td></tr><tr><td>studies_NNS</td><td>MAM RES CON</td></tr><tr><td>quality_NN</td><td>MAM RES CON</td></tr></tbody></table>
when the correct tag <H1> is used in source (b) when the generic tag (<ABS>)
is used instead. Titles are interesting because they tend to be more compact in
their wording than regular sentences and also differ in their syntactic structure
as they mostly consist in long nominal phrases devoid of verbs (some illustrations
in Table 9). As discussed above, titles can be predicted with some confidence
(in pre2 conditions).

490

Overall, we observed that the two outputs were very comparable and hardly
distinguishable in their BLEU scores (40.6 vs 40.9). In the majority of cases (931
titles), the two output translations are entirely identical. Out of the remaining
cases, the differences are always small, yet we can identify interesting differences
which illustrate the effect of these subsection tags. The first is that when using the
<H1> tag, target titles are actually slightly shorter than in the other condition:
the average length difference between the two outputs is 1.06 chars, with 77
cases of (a) being longer, and 131 cases where (b) is longer - these two situations
are illustrated in the first two examples of Table 9.17 There are various causes
for these length differences, which are sometimes as subtle as a change in a
determiner, see example 3 where translation (a) contains a collective use for the
singular determiner, which is very appropriate in scientific texts; and example 4
where (a) uses a noun group while (b) uses a verbal form. One syntactic pattern
however emerges from this analysis: outputs (a) tend to favour adjectives over
prepositional groups (see examples 5 and 6).

495

500

505

More examples of variations obtained with different tags are in appendix A
pendix B.

510

7. Related Work

*Document-level MT.* (DLMT) is loosely defined by contrast to "sentence-level"
MT (SLMT) and includes any technique aiming to handle translation phenomena
requiring a context extending beyond isolated sentences. The targeted phenomena

<footnote n="17">¹⁷Indeed the BLEU scores between these condition is mostly due to a lower brevity penalty when using <H1>.</footnote>
<table><caption>**Table 9:** Translation of Scielo test sets produced with fine-tuned systems from English into French. For each example, we display the source sentence, then the translations obtained respectively with the tags <H1> (a) and <ABS> (b). Note that for all these examples, the two French outputs are equally fluent and adequate. They illustrate small variations in wordings reflecting the style differences in titles vs. regular texts.</caption><tr><td>1 - src</td><td>Socioeconomic and geographic inequalities in infant mortality, 1990-2005</td></tr><tr><td>trg (a)</td><td>Inégalités socioéconomiques et géographiques de mortalité infantile, 1990-2005.</td></tr><tr><td>trg (b)</td><td>Inégalités socioéconomiques et géographiques en matière de mortalité infantile, 1990-2005.</td></tr><tr><td>2- src</td><td>The global burden of diarrhoeal disease, as estimated from studies published between 1992 and 2000.</td></tr><tr><td>trg (a)</td><td>La charge globale des maladies diarrhériques, telle qu'estimée à partir d'études publiées entre 1992 et 2000.</td></tr><tr><td>trg (b)</td><td>La charge mondiale des maladies diarrhériques, estimée d'après des études publiées entre 1992 et 2000.</td></tr><tr><td>3 - src</td><td>Homicide in children and adolescents: a case-control study in Recife, Brazil.</td></tr><tr><td>trg (a)</td><td>Homicide chez l'enfant et l'adolescent: étude cas-témoins à Recife, Brésil.</td></tr><tr><td>trg (b)</td><td>Homicide chez les enfants et les adolescents: étude cas-témoins à Recife, Brésil.</td></tr><tr><td>4 - src</td><td>Managing the effect of TRIPS on availability of priority vaccines.</td></tr><tr><td>trg (a)</td><td>Gestion de l'effet des ADPIC sur la disponibilité des vaccins prioritaires.</td></tr><tr><td>trg (b)</td><td>Gérer l'effet de l'ADPIC sur la disponibilité des vaccins prioritaires.</td></tr><tr><td>5 - src</td><td>Poverty, child undernutrition and morbidity: new evidence from India.</td></tr><tr><td>trg (a)</td><td>Pauvreté, sous-nutrition infantile et morbidité: nouvelles données indiennes.</td></tr><tr><td>trg (b)</td><td>Pauvreté, sous-nutrition infantile et morbidité: nouveaux éléments concernant l'Inde.</td></tr><tr><td>6 - src</td><td>Circulating vaccine-derived polioviruses: current state of knowledge.</td></tr><tr><td>trg (a)</td><td>Poliovirus circulants d'origine vaccinale: état actuel des connaissances.</td></tr><tr><td>trg (b)</td><td>Poliovirus circulants dérivés du vaccin: état actuel des connaissances.</td></tr></table>

515 are quite heterogeneous in nature, but notably encompass coreference issues, coherence issues, and discourse-level issues [4]. Coreference issues are mostly related to the consistency of pronoun use across languages, where the choice of a morphological variant for a pronoun (e.g., in gender or number) may be conditioned by its referent from previous sentence(s). Coherence corresponds to longer range phenomena that ensure that a translated text can be read as a whole, implying a consistent choice of terms, tense, style, and references throughout the text. Finally, DLMT also includes the generation of texts that correctly reproduce the argumentative structure of the source input. In the realm of neural architectures, DLMT has been mostly approached as "MT with long-range dependencies", fostering multiple approaches to integrate large spans of text in the translation context. These approaches range from simple extensions of SLMT where multiple sentences are translated as one unit [60, 61, 62], to more sophisticated proposals combining a short-term context (at the sentence
level) with a long-range context. This can be achieved with dual encoders [63], hierarchical architectures [64, 65, 66], or cache-based methods [12]. Widening the notion of context, topic models or even domain adaptation techniques can also be used to represent large documents [9]. A conclusion of several comparative studies [67, 15, 62] is that simple techniques (no context, or basic concatenation) are difficult to outperform. A review of DLMT is in [6]; more recent approaches based on large language models are documented in [68].

Tags. have been widely used in NMT to incorporate additional discrete conditioning factors in (1), and control the output language, the domain, or the level of formality and politeness [29, 30, 31], mitigate gender bias issues [69], and provide us with a simple and effective way to take structure in account.

Notably, [30, 40] use tags in multi-domain systems, an approach extended in [70] with multi-level; tags in [71, 72, 73, 31] inform systems about the level of formality or politeness of a sentence; finally, multilingual MT [74, 29, 75] use tags to select the desired target language. In the same line of studies, [76] use numerical tag values corresponding to length constraints, while [40] use a tag to enforce the decoding direction of a bidirectional system, in [35, 77] tags inform back-translated data, [78] use tags to control the readability level of NMT output, [42, 79] demonstrate zero-shot NMT capability using tags.

Tags have also proven useful for grammatical error correction using NMT [80] where tags are used with each word to indicate the operation e.g. keep, delete, replace etc. [81] use tags to enforce the decoder towards natural MT output, [82] used tags for named entity recognition, whereas tags have been used in [83] for entity projection for cross-lingual NER.

To our knowledge, there exists no prior work on using tags to represent the document structure in NMT other than [77], which only use tags on the source side. Closer to our work, one-level document structure-tags have been used for document quality prediction by [84], which reports a strong correlation with text classification. They used three tags (Title, Abstract and BodyText) and reported performance reduction when using smaller structure-tag set on all three
domains that they worked on.

In the context of Multilingual NMT, [42] shows language tags to significantly impact zero-shot translation quality. They report best scores by placing the target language tags on the encoder side which helps alleviate the off-target issue for such models. Similar analyses, for a variety of tags, are in [40, 41], and suggest that when tags are observed, having them on the source side yields better results. Our results also show better results with tags on the source side.

It is further possible to explicitly amplify this effect by injecting the tag information into the input representation of every token (eg. [30, 36] for domain information, or [37] for language information).

## 8. Conclusion and Discussion

Sentences in a document generally follow a typical style, for example the introductory sentences are clearly distinguishable from concluding sentences based on the lexical choice and sentence style etc. Whether or not can an MT system be trained to learn and use these sentence specific styles is the question that we have studied in this article.

We have tried to take advantage of the rigid structure of abstracts in the biomedical domain to assess the effect of structure on the translation quality. For this, we have carefully annotated a large subset of our training data with structural information, and trained models that were able to take this structure into account thanks to a hierarchical tag system that introduces an extra non-local context during translation. In addition to the elaboration of a new annotated resource, that will be useful for further works, our study has shown that (a) predicting the structural labels was possible from the sole source text, at least for fine tune systems exposed to structured documents; (b) based on this complex of tags, it was actually possible to improve our automatic evaluation scores, even though the actual effect of the sole structural tags on BLEU scores was found to be fairly limited in our test conditions. This highlights one clear limitation of our results, which mostly rely on automatic metrics such as BLEU or TER. It is
likely that such scores may not fully capture the fine-grained stylistic variations
that are implied by the use of document structure information.

In our future work, we would like to generalize this approach to other
structural labels that may be available for other documents and and domains,
as well as to combine this information with other, more fine grain, additional
contextual information that may be useful for document-level MT. Another
perspective is to explore how structural information is handled in NMT systems
based on large language models, which represent a competitive alternative to
the encoder-decoder models used in this study.

## Acknowledgements

This work was made possible thanks to the Saclay-IA computing platform.
It was granted access to the HPC resources of IDRIS under the allocation 2021-
[AD011011580R1, AD011011270R1, AD011011717] made by GENCI. The second
author was partly funded by the French "Agence Nationale de la Recherche"
(ANR) under grant ANR-22-CE23-0033 / MaTOS.

## References

[1] H. Hassan, A. Aue, C. Chen, V. Chowdhary, J. Clark, C. Federmann, X. Huang, M. Junczys-Dowmunt, W. Lewis, M. Li, et al., Achieving human parity on automatic Chinese to English news translation, arXiv preprint arXiv:1803.05567 (2018).

[2] M. Popel, M. Tomkova, J. Tomek, Ł. Kaiser, J. Uszkoreit, O. Bojar, Z. Žabokrtský, Transforming machine translation: a deep learning system reaches news translation quality comparable to human professionals, Nature Communications 11 (1) (2020) 4381. doi:10.1038/s41467-020-18073-9. URL https://doi.org/10.1038/s41467-020-18073-9

[3] E. Voita, P. Serdyukov, R. Sennrich, I. Titov, Context-aware neural machine translation learns anaphora resolution, in: Proceedings of the 56th Annual
615 Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Melbourne, Australia, 2018, pp. 1264–1274. doi:10.18653/v1/P18-1117. URL https://www.aclweb.org/anthology/P18-1117

[4] A. Popescu-Belis, Context in neural machine translation: A review of models and evaluations, arXiv preprint arXiv:1901.09115 (2019). URL http://arxiv.org/pdf/1901.09115

[5] R. Bawden, K. Bretonnel Cohen, C. Grozea, A. Jimeno Yepes, M. Kittner, M. Krallinger, N. Mah, A. Neveol, M. Neves, F. Soares, A. Siu, K. Verspoor, M. Vicente Navarro, Findings of the WMT 2019 biomedical translation shared task: Evaluation for MEDLINE abstracts and biomedical terminologies, in: Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2), Association for Computational Linguistics, Florence, Italy, 2019, pp. 29–53. doi:10.18653/v1/W19-5403. URL https://www.aclweb.org/anthology/W19-5403

[6] S. Maruf, F. Saleh, G. Haffari, A survey on document-level neural machine translation: Methods and evaluation, ACM Comput. Surv. 54 (2) (Mar. 2021). doi:10.1145/3441691. URL https://doi.org/10.1145/3441691

[7] P. Fernandes, K. Yin, E. Liu, A. Martins, G. Neubig, When does translation require context? a data-driven, multilingual exploration, in: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Toronto, Canada, 2023, pp. 606–626. URL https://aclanthology.org/2023.acl-long.36

[8] K. Hofmann, M. Tsagkias, E. Meij, M. De Rijke, The impact of document structure on keyphrase extraction, in: Proceedings of the 18th ACM conference on Information and knowledge management, 2009, pp. 1725–1728.
[9] S. Kuang, D. Xiong, W. Luo, G. Zhou, Modeling coherence for neural machine translation with dynamic and topic caches, in: Proceedings of the 27th International Conference on Computational Linguistics, Association for Computational Linguistics, Santa Fe, New Mexico, USA, 2018, pp. 596–606. URL https://www.aclweb.org/anthology/C18-1050

645

[10] J. Tiedemann, Y. Scherrer, Neural machine translation with extended context, in: Proceedings of the Third Workshop on Discourse in Machine Translation, Association for Computational Linguistics, Copenhagen, Denmark, 2017, pp. 82–92. doi:10.18653/v1/W17-4811.
URL https://www.aclweb.org/anthology/W17-4811

650

[11] J. Zhang, H. Luan, M. Sun, F. Zhai, J. Xu, M. Zhang, Y. Liu, Improving the Transformer translation model with document-level context, in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Brussels, Belgium, 2018, pp. 533–542. doi:10.18653/v1/D18-1049.
URL https://www.aclweb.org/anthology/D18-1049

655

[12] Z. Tu, Y. Liu, S. Shi, T. Zhang, Learning to remember translation history with a continuous cache, Transactions of the Association for Computational Linguistics 6 (2018) 407–420. doi:10.1162/tacl_a_00029.
URL https://www.aclweb.org/anthology/Q18-1029

660

[13] M. Junczys-Dowmunt, Microsoft translator at WMT 2019: Towards large-scale document-level neural machine translation, in: Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), Association for Computational Linguistics, Florence, Italy, 2019, pp. 225–233. doi:10.18653/v1/W19-5321.
URL https://www.aclweb.org/anthology/W19-5321

655

[14] Q. Guo, X. Qiu, P. Liu, Y. Shao, X. Xue, Z. Zhang, Star-transformer, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language

670
Technologies, Volume 1 (Long and Short Papers), Association for Com-
putational Linguistics, Minneapolis, Minnesota, 2019, pp. 1315–1325.
doi:10.18653/v1/N19-1133.

675

URL https://www.aclweb.org/anthology/N19-1133

[15] A. Lopes, M. A. Farajian, R. Bawden, M. Zhang, A. Martins, Document-level neural MT: A systematic comparison, in: 22nd Annual Conference of the European Association for Machine Translation, 2020, pp. 225–234.

680

[16] S. Ma, D. Zhang, M. Zhou, A simple and effective unified encoder for document-level machine translation, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Online, 2020, pp. 3505–3511. doi:10.18653/v1/2020.acl-main.321.

URL https://www.aclweb.org/anthology/2020.acl-main.321

685

[17] S. Abdul Rauf, F. Yvon, Document level contexts for neural machine translation, Research Report 2020-003, LIMSI-CNRS (Dec. 2020).
URL https://hal.archives-ouvertes.fr/hal-03687190

690

[18] K. Hashimoto, R. Buschiazzo, J. Bradbury, T. Marshall, R. Socher, C. Xiong, A high-quality multilingual dataset for structured documentation translation, in: Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers), Association for Computational Linguistics, Florence, Italy, 2019, pp. 116–127. doi:10.18653/v1/W19-5212.

URL https://aclanthology.org/W19-5212

695

[19] G. Hanneman, G. Dinu, How should markup tags be translated?, in: Proceedings of the Fifth Conference on Machine Translation, Association for Computational Linguistics, Online, 2020, pp. 1160–1173.
URL https://aclanthology.org/2020.wmt-1.138

[20] R. Dobreva, J. Zhou, R. Bawden, Document sub-structure in neural machine translation, in: Proceedings of the 12th Language Resources and Evaluation
700 Conference, European Language Resources Association, Marseille, France, 2020, pp. 3657–3667.
URL https://www.aclweb.org/anthology/2020.lrec-1.451

[21] B. Haddow, A. Birch, K. Heafield, Machine translation in healthcare, in: The Routledge Handbook of Translation and Health, Routledge, 2021, pp. 108–129.

[22] M. Zappatore, G. Ruggieri, Adopting machine translation in the healthcare sector: A methodological multi-criteria review, Computer Speech & Language 84 (2024) 101582. doi:https://doi.org/10.1016/j.csl.2023.101582.
URL https://www.sciencedirect.com/science/article/pii/S0885230823001018

[23] L. B. Sollaci, M. G. Pereira, The introduction, methods, results, and discussion (IMRAD) structure: a fifty-year survey., Journal of the Medical Library Association : JMLA 92 (3) (2004) 364–367.

[24] P. Koehn, Neural Machine Translation, Cambridge University Press, 2020.

[25] F. Stahlberg, Neural machine translation: A review, Journal of Artificial Intelligence Review 69 (2020) 343–418.

[26] G. Wiher, C. Meister, R. Cotterell, On decoding strategies for neural text generators (2022). doi:10.48550/ARXIV.2203.15721.
URL https://arxiv.org/abs/2203.15721

[27] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, in: I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, R. Garnett (Eds.), Advances in Neural Information Processing Systems 30, Curran Associates, Inc., 2017, pp. 5998–6008.
URL http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf
[28] D. Bahdanau, K. Cho, Y. Bengio, Neural machine translation by jointly learning to align and translate, CoRR abs/1409.0473 (2014).
URL http://arxiv.org/abs/1409.0473

730

[29] Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, et al., Google's neural machine translation system: Bridging the gap between human and machine translation (2016). arXiv:1609.08144.
URL https://arxiv.org/abs/1609.08144

735

[30] C. Kobus, J. Crego, J. Senellart, Domain control for neural machine translation, in: Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017, Varna, Bulgaria, 2017, pp. 372-378. doi:10.26615/978-954-452-049-6_049.
URL https://doi.org/10.26615/978-954-452-049-6_049

740

[31] A. Madaan, A. Setlur, T. Parekh, B. Poczos, G. Neubig, Y. Yang, R. Salakhutdinov, A. W. Black, S. Prabhumoye, Politeness transfer: A tag and generate approach, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Online, 2020, pp. 1869-1881. doi:10.18653/v1/2020.acl-main.169.
URL https://aclanthology.org/2020.acl-main.169

745

[32] R. C. Moore, Fast and accurate sentence alignment of bilingual corpora, in: S. D. Richardson (Ed.), Proc. AMTA'02, Lecture Notes in Computer Science 2499, Springer Verlag, Tiburon, CA, USA, 2002, pp. 135-144.
URL https://www.microsoft.com/en-us/research/publication/fast-and-accurate-sentence-alignment-of-bilingual-corpora/

750

[33] W. Peng, J. Liu, L. Li, Q. Liu, Huawei's NMT systems for the WMT 2019 biomedical translation task, in: Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2), Association for Computational Linguistics, Florence, Italy, 2019. doi:10.18653/v1/

755
760

[34] I. Caswell, C. Chelba, D. Grangier, Tagged back-translation, in: Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers), Association for Computational Linguistics, Florence, Italy, 2019, pp. 53–63. doi:10.18653/v1/W19-5206.

URL https://aclanthology.org/W19-5206

765

[35] B. Marie, R. Rubino, A. Fujita, Tagged back-translation revisited: Why does it really work?, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Online, 2020, pp. 5990–5997. doi:10.18653/v1/2020.acl-main.532.

URL https://aclanthology.org/2020.acl-main.532

770

[36] C. Chu, R. Dabre, S. Kurohashi, An empirical comparison of domain adaptation methods for neural machine translation, in: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), ACL 2017, Vancouver, Canada, 2017, pp. 385–391.

URL http://aclweb.org/anthology/P17-2061

775

[37] A. Conneau, G. Lample, Cross-lingual language model pretraining, in: H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, R. Garnett (Eds.), Advances in Neural Information Processing Systems 32, Curran Associates, Inc., 2019, pp. 7059–7069.

URL http://papers.nips.cc/paper/8928-cross-lingual-language-model-pretraining.pdf

780

[38] D. Britz, Q. Le, R. Pryzant, Effective domain mixing for neural machine translation, in: Proceedings of the Second Conference on Machine Translation, Association for Computational Linguistics, Copenhagen, Denmark, 2017, pp. 118–126.

785

URL http://aclweb.org/anthology/W17-4712
790

795

800

805

810

[39] J. Wuebker, S. Green, J. DeNero, S. Hasan, M.-T. Luong, Models and inference for prefix-constrained machine translation, in: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Berlin, Germany, 2016, pp. 66–75. doi:10.18653/v1/P16-1007.
URL https://aclanthology.org/P16-1007

[40] S. Takeno, M. Nagata, K. Yamamoto, Controlling target features in neural machine translation via prefix constraints, in: Proceedings of the 4th Workshop on Asian Translation (WAT2017), Asian Federation of Natural Language Processing, Taipei, Taiwan, 2017, pp. 55–63.
URL https://aclanthology.org/W17-5702

[41] M. Q. Pham, J. Crego, F. Yvon, Revisiting multi-domain machine translation, Transactions of the Association for Computational Linguistics 9 (0) (2021) 17–35.
URL https://transacl.org/index.php/tacl/article/view/2327

[42] L. Wu, S. Cheng, M. Wang, L. Li, Language tags matter for zero-shot neural machine translation, in: Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, Association for Computational Linguistics, Online, 2021, pp. 3001–3007. doi:10.18653/v1/2021.findings-acl.264.
URL https://aclanthology.org/2021.findings-acl.264

[43] A. Jimeno Yepes, A. Névéol, M. Neves, K. Verspoor, O. Bojar, A. Boyer, C. Grozea, B. Haddow, M. Kittner, Y. Lichtblau, P. Pecina, R. Roller, R. Rosa, A. Siu, P. Thomas, S. Trescher, Findings of the WMT 2017 biomedical translation shared task, in: Proceedings of the Second Conference on Machine Translation, Association for Computational Linguistics, Copenhagen, Denmark, 2017, pp. 234–247. doi:10.18653/v1/W17-4719.
URL https://www.aclweb.org/anthology/W17-4719

[44] M. Neves, A. J. Yepes, A. Névéol, The Scielo Corpus: a parallel corpus of scientific publications for biomedicine, in: Proceedings of the Tenth
815

International Conference on Language Resources and Evaluation (LREC'16),
European Language Resources Association (ELRA), Portorož, Slovenia,
2016, pp. 2942-2948.

URL https://www.aclweb.org/anthology/L16-1470

820

[45] J. Ivé, A. Max, F. Yvon, P. Ravaud, Diagnosing high-quality statistical
machine translation using traces of post-edition operations, in: International
Conference on Language Resources and Evaluation - Workshop on Transla-
tion Evaluation: From Fragmented Tools and Data Sets to an Integrated
Ecosystem (MT Eval 2016 2016), Portorož, Slovenia, 2016, p. 8.

URL http://www.lrec-conf.org/proceedings/lrec2016/workshops/LREC2016Workshop-MT%20Evaluation_Proceedings.pdf#page=65

825

[46] P. Lison, J. Tiedemann, OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles, in: Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), European Language Resources Association (ELRA), Portorož, Slovenia, 2016, pp. 923–929.

830

URL https://www.aclweb.org/anthology/L16-1147

[47] S. Abdul-Rauf, H. Schwenk, On the use of comparable corpora to improve SMT performance, in: Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009), Association for Computational Linguistics, Athens, Greece, 2009, pp. 16–23.

835

URL https://www.aclweb.org/anthology/E09-1003

[48] S. Naz, S. Abdul Rauf, N.-e. Hira, S. Ul Haq, FJWU participation for the WMT20 biomedical translation task, in: Proceedings of the Fifth Conference on Machine Translation, Association for Computational Linguistics, Online, 2020, pp. 849–856.

840

URL https://aclanthology.org/2020.wmt-1.92

[49] M. Neves, A. Jimeno Yepes, A. Névéol, C. Grozea, A. Siu, M. Kitterner, K. Verspoor, Findings of the WMT 2018 biomedical translation
845

shared task: Evaluation on Medline test sets, in: Proceedings of the
Third Conference on Machine Translation: Shared Task Papers, Asso-
ciation for Computational Linguistics, Belgium, Brussels, 2018, pp. 324–339.
doi:10.18653/v1/W18-6403.

URL https://aclanthology.org/W18-6403

850

[50] R. Bawden, K. Bretonnel Cohen, C. Grozea, A. Jimeno Yepes, M. Kittner, M. Krallinger, N. Mah, A. Neveol, M. Neves, F. Soares, A. Siu, K. Verspoor, M. Vicente Navarro, Findings of the WMT 2019 biomedical translation shared task: Evaluation for MEDLINE abstracts and biomedical terminologies, in: Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2), Association for Computational Linguistics, Florence, Italy, 2019, pp. 29–53. doi:10.18653/v1/W19-5403.
URL https://aclanthology.org/W19-5403

855

[51] D. Stojanovski, V. Hangya, M. Huck, A. Fraser, The LMU munich unsupervised machine translation system for WMT19, in: Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), Association for Computational Linguistics, Florence, Italy, 2019, pp. 393–399. doi:10.18653/v1/W19-5344.
URL https://www.aclweb.org/anthology/W19-5344

860

[52] F. Maniez, L'adjectif dénominal en langue de spécialité: étude du domaine de la médecine., Revue française de linguistique appliquée 14 (2) (2009) 117-130.

865

[53] M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, M. Auli, fairseq: A fast, extensible toolkit for sequence modeling, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), Association for Computational Linguistics, Minneapolis, Minnesota, 2019, pp. 48–53. doi:10.18653/v1/N19-4009.

URL https://www.aclweb.org/anthology/N19-4009

870
875

[54] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, in: Y. Bengio, Y. LeCun (Eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
URL http://arxiv.org/abs/1412.6980

880

[55] T. Kudo, J. Richardson, SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing, in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Association for Computational Linguistics, Brussels, Belgium, 2018, pp. 66–71. doi:10.18653/v1/D18-2012.
URL https://www.aclweb.org/anthology/D18-2012

885

[56] M. Post, A call for clarity in reporting BLEU scores, in: Proceedings of the Third Conference on Machine Translation: Research Papers, Association for Computational Linguistics, Brussels, Belgium, 2018, pp. 186–191. doi: 10.18653/v1/W18-6319.
URL https://www.aclweb.org/anthology/W18-6319

890

[57] K. Papineni, S. Roukos, T. Ward, W.-J. Zhu, BLEU: a method for automatic evaluation of machine translation, in: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, Stroudsburg, PA, USA, 2002, pp. 311–318. doi:10.3115/1073083.1073135.
URL https://www.aclweb.org/anthology/P02-1040

895

[58] M. Snover, B. Dorr, R. Schwartz, L. Micciulla, J. Makhoul, A study of translation edit rate with targeted human annotation, in: Proceedings of the seventh conference of the Association for Machine Translation in the America (AMTA), Boston, Massachusetts, USA, 2006, pp. 223–231.
URL http://www.mt-archive.info/AMTA-2006-Snover.pdf

900

[59] M. Popović, chrF: character n-gram F-score for automatic MT evaluation, in: Proceedings of the Tenth Workshop on Statistical Machine Translation,
Association for Computational Linguistics, Lisbon, Portugal, 2015, pp. 392–395. doi:10.18653/v1/W15-3049.

URL https://aclanthology.org/W15-3049

905

[60] Y. Scherrer, J. Tiedemann, S. Loáiciga, Analysing concatenation approaches to document-level NMT in two different domains, in: Proceedings of the Fourth Workshop on Discourse in Machine Translation (DiscoMT 2019), Association for Computational Linguistics, Hong Kong, China, 2019, pp. 51–61. doi:10.18653/v1/D19-6506.

URL https://www.aclweb.org/anthology/D19-6506

910

[61] Z. Ma, S. Edunov, M. Auli, A comparison of approaches to document-level machine translation, arXiv preprint arXiv:1910.07481 (2021). doi:10.11040.

915

[62] Z. Sun, M. Wang, H. Zhou, C. Zhao, S. Huang, J. Chen, L. Li, Rethinking document-level neural machine translation, in: Findings of the Association for Computational Linguistics: ACL 2022, Association for Computational Linguistics, Dublin, Ireland, 2022, pp. 3537–3548. doi:10.18653/v1/2022.findings-acl.279.

URL https://aclanthology.org/2022.findings-acl.279

920

[63] R. Bawden, R. Sennrich, A. Birch, B. Haddow, Evaluating discourse phenomena in neural machine translation, in: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), Association for Computational Linguistics, New Orleans, Louisiana, 2018, pp. 1304–1313. doi:10.18653/v1/N18-1118.

925

URL https://www.aclweb.org/anthology/N18-1118

[64] L. Miculicich, D. Ram, N. Pappas, J. Henderson, Document-level neural machine translation with hierarchical attention networks, in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Brussels, Belgium, 2018, pp.
2947-2954. doi:10.18653/v1/D18-1325.

URL https://www.aclweb.org/anthology/D18-1325

J. Chen, X. Li, J. Zhang, C. Zhou, J. Cui, B. Wang, J. Su, Modeling discourse structure for document-level neural machine translation, in: Proceedings of the First Workshop on Automatic Simultaneous Translation, Association for Computational Linguistics, Seattle, Washington, 2020, pp. 30–36. doi: 10.18653/v1/2020.autosimtrans-1.5.

URL https://www.aclweb.org/anthology/2020.autosimtrans-1.5

Z. Zheng, X. Yue, S. Huang, J. Chen, A. Birch, Towards making the most of context in neural machine translation, in: C. Bessiere (Ed.), Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, International Joint Conferences on Artificial Intelligence Organization, 2020, pp. 3983–3989. doi:10.24963/ijcai.2020/551.

URL https://doi.org/10.24963/ijcai.2020/551

Y. Kim, D. T. Tran, H. Ney, When and why is document-level context useful in neural machine translation?, in: Proceedings of the Fourth Workshop on Discourse in Machine Translation (DiscoMT 2019), Association for Computational Linguistics, Hong Kong, China, 2019, pp. 24–34. doi: 10.18653/v1/D19-6503.

URL https://www.aclweb.org/anthology/D19-6503

L. Wang, C. Lyu, T. Ji, Z. Zhang, D. Yu, S. Shi, Z. Tu, Document-level machine translation with large language models, in: H. Bouamor, J. Pino, K. Bali (Eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Singapore, 2023, pp. 16646–16661. doi:10.18653/v1/2023.emnlp-main.

1036.

955

URL https://aclanthology.org/2023.emnlp-main.1036

[69] D. Saunders, R. Sallis, B. Byrne, Neural machine translation doesn't translate gender coreference right unless you make it (2020). arXiv:2010.05332.
[70] E. Stergiadis, S. Kumar, F. Kovalev, P. Levin, Multi-domain adaptation in neural machine translation through multidimensional tagging, CoRR abs/2102.10160 (2021). arXiv:2102.10160. URL https://arxiv.org/abs/2102.10160

960

[71] R. Sennrich, B. Haddow, Linguistic input features improve neural machine translation, in: Proceedings of the First Conference on Machine Translation: Volume 1, Research Papers, Association for Computational Linguistics, Berlin, Germany, 2016, pp. 83–91. doi:10.18653/v1/W16-2209. URL https://www.aclweb.org/anthology/W16-2209

965

[72] X. Niu, M. Martindale, M. Carpuat, A study of style in machine translation: Controlling the formality of machine translation output, in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Copenhagen, Denmark, 2017, pp. 2814–2819. doi:10.18653/v1/D17-1299. URL https://www.aclweb.org/anthology/D17-1299

970

[73] X. Niu, S. Rao, M. Carpuat, Multi-task neural models for translating between styles within and across languages, in: Proceedings of the 27th International Conference on Computational Linguistics, Association for Computational Linguistics, Santa Fe, New Mexico, USA, 2018, pp. 1008–1021. URL https://aclanthology.org/C18-1086

975

[74] O. Firat, K. Cho, Y. Bengio, Multi-way, multilingual neural machine translation with a shared attention mechanism, in: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics, 2016, pp. 866–875. doi:10.18653/v1/N16-1101. URL http://www.aclweb.org/anthology/N16-1101

980

985

[75] M. Johnson, M. Schuster, Q. Le, M. Krikun, Y. Wu, Z. Chen, N. Thorat, F. a. Viégas, M. Wattenberg, G. Corrado, M. Hughes, J. Dean, Google's
multilingual neural machine translation system: Enabling zero-shot translation, *Transactions of the Association for Computational Linguistics* 5 (2017) 339–351.

URL https://transacl.org/ojs/index.php/tacl/article/view/1081

[76] Y. Kikuchi, G. Neubig, R. Sasano, H. Takamura, M. Okumura, Controlling output length in neural encoder-decoders, in: *Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing*, Association for Computational Linguistics, 2016, pp. 1328–1338.
doi:10.18653/v1/D16-1140.

URL http://aclweb.org/anthology/D16-1140

[77] J. Xu, M. Q. Pham, S. Abdul Rauf, F. Yvon, LISN @ WMT 2021, in: *Proceedings of the Sixth Conference on Machine Translation*, Association for Computational Linguistics, Online, 2021, pp. 232–242.

URL https://aclanthology.org/2021.wmt-1.22

[78] K. Marchisio, J. Guo, C.-I. Lai, P. Koehn, Controlling the reading level of machine translation output, in: *Proceedings of Machine Translation Summit XVII Volume 1: Research Track*, European Association for Machine Translation, Dublin, Ireland, 2019, pp. 193–2004.

URL https://www.aclweb.org/anthology/W19-6619

[79] Z. Mao, R. Dabre, Q. Liu, H. Song, C. Chu, S. Kurohashi, Exploring the impact of layer normalization for zero-shot neural machine translation (2023). arXiv:2305.09312.

[80] D. Liang, C. Zheng, L. Guo, X. Cui, X. Xiong, H. Rong, J. Dong, BERT enhanced neural machine translation and sequence tagging model for Chinese grammatical error diagnosis, in: *Proceedings of the 6th Workshop on Natural Language Processing Techniques for Educational Applications*, Association for Computational Linguistics, Suzhou, China, 2020, pp. 57–66.

URL https://aclanthology.org/2020.nlptea-1.8
[81] M. Freitag, D. Vilar, D. Grangier, C. Cherry, G. Foster, A natural diet: Towards improving naturalness of machine translation output, in: Findings of the Association for Computational Linguistics: ACL 2022, Association for Computational Linguistics, Dublin, Ireland, 2022, pp. 3340–3353. doi: 10.18653/v1/2022.findings-acl.263.
URL https://aclanthology.org/2022.findings-acl.263

1020

[82] A. Jain, B. Paranjape, Z. C. Lipton, Entity projection via machine translation for cross-lingual NER, in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Association for Computational Linguistics, Hong Kong, China, 2019, pp. 1083–1092. doi:10.18653/v1/D19-1100.
URL https://aclanthology.org/D19-1100

1025

[83] A. Berard, I. Calapodescu, C. Roux, Naver labs Europe's systems for the WMT19 machine translation robustness task, in: Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), Association for Computational Linguistics, Florence, Italy, 2019, pp. 526–532. doi:10.18653/v1/W19-5361.
URL https://aclanthology.org/W19-5361

1030

[84] G. Maillette de Buy Wenniger, T. van Dongen, E. Aedmaa, H. T. Kruitbosch, E. A. Valentijn, L. Schomaker, Structure-tags improve text classification for scholarly document quality prediction, in: Proceedings of the First Workshop on Scholarly Document Processing, Association for Computational Linguistics, Online, 2020, pp. 158–167. doi:10.18653/v1/2020.sdp-1.18.
URL https://aclanthology.org/2020.sdp-1.18

1035

1040
Appendix A. Precision and Recall scores for Confusion matrices

<figure><img src="image_3.png" /><figcaption>Figure A.4: FR-EN: from left to right pre0, pre1 and pre2. First row baseline, Second row fine-tuned systems</figcaption></figure>

<table><caption>Table A.10: Precision, Recall and F1 Score on Scielo baseline and fine-tuned systems without abstract in reference</caption><thead><tr><th rowspan="3">Prefix</th><th rowspan="3">Section</th><th colspan="9">EN-FR</th><th colspan="9">FR-EN</th></tr><tr><th colspan="3">Baseline</th><th colspan="3">Fine-tuned</th><th colspan="3">Baseline</th><th colspan="3">Fine-tuned</th></tr><tr><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th></tr></thead><tbody><tr><td rowspan="6">Pre0</td><td>&lt;CON&gt;</td><td>0.99</td><td>0.32</td><td>0.48</td><td>0.95</td><td>0.35</td><td>0.51</td><td>1.00</td><td>0.00</td><td>0.00</td><td>0.76</td><td>0.02</td><td>0.04</td></tr><tr><td>&lt;H1&gt;</td><td>1.00</td><td>0.01</td><td>0.02</td><td>1.00</td><td>0.51</td><td>0.68</td><td>0.98</td><td>0.44</td><td>0.60</td><td>0.99</td><td>0.37</td><td>0.53</td></tr><tr><td>&lt;INT&gt;</td><td>0.14</td><td>0.12</td><td>0.13</td><td>0.14</td><td>0.12</td><td>0.13</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td></tr><tr><td>&lt;MaM&gt;</td><td>0.98</td><td>0.39</td><td>0.56</td><td>0.97</td><td>0.55</td><td>0.70</td><td>0.85</td><td>0.02</td><td>0.04</td><td>0.95</td><td>0.07</td><td>0.12</td></tr><tr><td>&lt;OBJ&gt;</td><td>1.00</td><td>0.53</td><td>0.69</td><td>1.00</td><td>0.68</td><td>0.81</td><td>0.00</td><td>0.00</td><td>0.00</td><td>1.00</td><td>0.01</td><td>0.02</td></tr><tr><td>&lt;RES&gt;</td><td>0.85</td><td>0.38</td><td>0.52</td><td>0.89</td><td>0.47</td><td>0.61</td><td>0.82</td><td>0.04</td><td>0.07</td><td>0.81</td><td>0.07</td><td>0.13</td></tr><tr><td rowspan="3">Pre1</td><td>accuracy</td><td>-</td><td>-</td><td>0.32</td><td>-</td><td>-</td><td>0.50</td><td>-</td><td>-</td><td>0.10</td><td>-</td><td>-</td><td>0.11</td></tr><tr><td>macro avg</td><td>0.62</td><td>0.22</td><td>0.30</td><td>0.62</td><td>0.34</td><td>0.43</td><td>0.46</td><td>0.06</td><td>0.09</td><td>0.56</td><td>0.07</td><td>0.11</td></tr><tr><td>weighted avg</td><td>0.95</td><td>0.32</td><td>0.45</td><td>0.95</td><td>0.50</td><td>0.65</td><td>0.78</td><td>0.10</td><td>0.15</td><td>0.89</td><td>0.11</td><td>0.18</td></tr><tr><td rowspan="6">Pre2</td><td>&lt;CON&gt;</td><td>0.99</td><td>0.32</td><td>0.48</td><td>0.95</td><td>0.35</td><td>0.51</td><td>1.00</td><td>0.00</td><td>0.00</td><td>0.76</td><td>0.02</td><td>0.04</td></tr><tr><td>&lt;H1&gt;</td><td>1.00</td><td>0.01</td><td>0.02</td><td>1.00</td><td>0.51</td><td>0.68</td><td>0.98</td><td>0.44</td><td>0.61</td><td>0.99</td><td>0.37</td><td>0.53</td></tr><tr><td>&lt;INT&gt;</td><td>0.14</td><td>0.13</td><td>0.08</td><td>0.14</td><td>0.12</td><td>0.13</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td></tr><tr><td>&lt;MaM&gt;</td><td>0.96</td><td>0.39</td><td>0.56</td><td>0.97</td><td>0.55</td><td>0.70</td><td>0.85</td><td>0.02</td><td>0.04</td><td>0.95</td><td>0.07</td><td>0.12</td></tr><tr><td>&lt;OBJ&gt;</td><td>1.00</td><td>0.53</td><td>0.69</td><td>1.00</td><td>0.68</td><td>0.81</td><td>0.00</td><td>0.00</td><td>0.00</td><td>1.00</td><td>0.01</td><td>0.02</td></tr><tr><td>&lt;RES&gt;</td><td>0.85</td><td>0.38</td><td>0.53</td><td>0.89</td><td>0.47</td><td>0.61</td><td>0.82</td><td>0.04</td><td>0.07</td><td>0.81</td><td>0.07</td><td>0.13</td></tr><tr><td rowspan="3">Pre3</td><td>accuracy</td><td>-</td><td>-</td><td>0.32</td><td>-</td><td>-</td><td>0.50</td><td>-</td><td>-</td><td>0.10</td><td>-</td><td>-</td><td>0.11</td></tr><tr><td>macro avg</td><td>0.62</td><td>0.22</td><td>0.30</td><td>0.62</td><td>0.34</td><td>0.43</td><td>0.45</td><td>0.06</td><td>0.09</td><td>0.56</td><td>0.07</td><td>0.11</td></tr><tr><td>weighted avg</td><td>0.95</td><td>0.32</td><td>0.45</td><td>0.95</td><td>0.50</td><td>0.65</td><td>0.78</td><td>0.10</td><td>0.13</td><td>0.89</td><td>0.11</td><td>0.18</td></tr></tbody></table>
<table><caption>**Table A.11:** Precision, Recall and F1 Score on Scielo baseline and fine-tuned systems for systems with and without abstract in reference comparison</caption><thead><tr><th rowspan="3">Prefix</th><th rowspan="3">Section</th><th colspan="6">EN-FR</th><th colspan="6">FR-EN</th></tr><tr><th colspan="3">Baseline</th><th colspan="3">Fine-tuned</th><th colspan="3">Baseline</th><th colspan="3">Fine-tuned</th></tr><tr><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th></tr></thead><tbody><tr><td rowspan="10">Pre2</td><td>&lt;CON&gt;</td><td>0.96</td><td>0.34</td><td>0.51</td><td>0.88</td><td>0.53</td><td>0.66</td><td>0.99</td><td>0.32</td><td>0.49</td><td>0.88</td><td>0.53</td><td>0.67</td></tr><tr><td>&lt;H1&gt;</td><td>1.00</td><td>0.55</td><td>0.71</td><td>1.00</td><td>0.80</td><td>0.89</td><td>0.97</td><td>0.46</td><td>0.62</td><td>1.00</td><td>0.86</td><td>0.92</td></tr><tr><td>&lt;INT&gt;</td><td>0.33</td><td>0.12</td><td>0.18</td><td>1.00</td><td>0.12</td><td>0.22</td><td>0.20</td><td>0.20</td><td>0.20</td><td>0.50</td><td>0.20</td><td>0.29</td></tr><tr><td>&lt;MaM&gt;</td><td>0.93</td><td>0.65</td><td>0.77</td><td>0.89</td><td>0.82</td><td>0.85</td><td>0.86</td><td>0.85</td><td>0.85</td><td>0.88</td><td>0.87</td><td>0.87</td></tr><tr><td>&lt;OBJ&gt;</td><td>1.00</td><td>0.71</td><td>0.83</td><td>0.99</td><td>1.00</td><td>0.76</td><td>1.00</td><td>0.57</td><td>0.72</td><td>0.99</td><td>0.74</td><td>0.85</td></tr><tr><td>&lt;RES&gt;</td><td>0.81</td><td>0.75</td><td>0.78</td><td>0.85</td><td>0.82</td><td>0.83</td><td>0.88</td><td>0.72</td><td>0.79</td><td>0.87</td><td>0.81</td><td>0.84</td></tr><tr><td>accuracy</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>macro avg</td><td>0.72</td><td>0.45</td><td>0.54</td><td>0.80</td><td>0.55</td><td>0.62</td><td>0.70</td><td>0.45</td><td>0.53</td><td>0.73</td><td>0.57</td><td>0.63</td></tr><tr><td>weighted avg</td><td>0.92</td><td>0.61</td><td>0.72</td><td>0.91</td><td>0.75</td><td>0.82</td><td>0.93</td><td>0.61</td><td>0.71</td><td>0.91</td><td>0.77</td><td>0.83</td></tr><tr><td rowspan="10">Pre2 with &lt;ABS&gt;</td><td>&lt;ABS&gt;</td><td>0.48</td><td>0.82</td><td>0.61</td><td>0.62</td><td>0.78</td><td>0.70</td><td>0.48</td><td>0.85</td><td>0.61</td><td>0.64</td><td>0.76</td><td>0.70</td></tr><tr><td>&lt;CON&gt;</td><td>0.85</td><td>0.40</td><td>0.54</td><td>0.68</td><td>0.53</td><td>0.60</td><td>0.97</td><td>0.32</td><td>0.48</td><td>0.70</td><td>0.54</td><td>0.61</td></tr><tr><td>&lt;H1&gt;</td><td>1.00</td><td>0.09</td><td>0.17</td><td>1.00</td><td>0.80</td><td>0.89</td><td>0.96</td><td>0.46</td><td>0.62</td><td>0.99</td><td>0.86</td><td>0.92</td></tr><tr><td>&lt;INT&gt;</td><td>0.01</td><td>0.12</td><td>0.02</td><td>0.25</td><td>0.12</td><td>0.17</td><td>0.07</td><td>0.20</td><td>0.11</td><td>0.50</td><td>0.20</td><td>0.29</td></tr><tr><td>&lt;MaM&gt;</td><td>0.74</td><td>0.80</td><td>0.77</td><td>0.81</td><td>0.82</td><td>0.82</td><td>0.76</td><td>0.85</td><td>0.80</td><td>0.78</td><td>0.87</td><td>0.83</td></tr><tr><td>&lt;OBJ&gt;</td><td>0.99</td><td>0.71</td><td>0.83</td><td>0.99</td><td>0.76</td><td>0.86</td><td>1.00</td><td>0.57</td><td>0.72</td><td>0.99</td><td>0.74</td><td>0.84</td></tr><tr><td>&lt;RES&gt;</td><td>0.74</td><td>0.82</td><td>0.78</td><td>0.77</td><td>0.82</td><td>0.79</td><td>0.80</td><td>0.72</td><td>0.76</td><td>0.77</td><td>0.81</td><td>0.79</td></tr><tr><td>accuracy</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>macro avg</td><td>0.69</td><td>0.54</td><td>0.53</td><td>0.73</td><td>0.66</td><td>0.69</td><td>0.72</td><td>0.57</td><td>0.59</td><td>0.77</td><td>0.68</td><td>0.71</td></tr><tr><td>weighted avg</td><td>0.75</td><td>0.64</td><td>0.61</td><td>0.78</td><td>0.76</td><td>0.76</td><td>0.77</td><td>0.67</td><td>0.67</td><td>0.78</td><td>0.77</td><td>0.77</td></tr></tbody></table>

## Appendix B. Examples of variations incurred by tag differences

For this analysis, we generated translations of random documents using all possible tag combinations in source and target sides. A large set of these combinations were human analysed, some sample outputs for three tag combinations are in Table B.12 for SRCTAG vs TRGTAG with the corresponding BLEU, TER and chrF scores.

We observe variation in lexical choice as well as grammatical sentence formation, due to using SRCTAG vs TRGTAG. We see the preference of all SRCTAG combinations for certain lexical choices, e.g. SRCTAG system outputs always starts with the article *the*. Similarly, several phrase constructions appear to have been learned from out-of-domain data as these majorly appear in SRCTAG sentences having *any* tag from {<G> <WMT> <US>} .<fnref n="18" /> For instance, “death registration” and “has been” in second and third examples.

<footnote n="18">¹⁸This tag combination is used for all out-of-domain sentences in the training corpus.</footnote>
<figure><img src="image_4.png" /><figcaption>**Table B.12:** Example outputs with varying tag combinations. Lexical differences are marked in blue whereas grammatical changes are given in orange. Invalid tags in a particular combination are marked in red. BLEU, TER and chrF scores are computed for whole test-set.</figcaption></figure>

1055

An interesting example is the `<G> <PATR><fnref n="19" /> <ABS>` combination (last row in Table B.12), where two tags are wrong and the SRCTAG system used the word “formation”, “nearby” and “encoders” instead respectively of “training”, “outreach” and “coders” which are generated with the reference tags.

<footnote n="19">¹⁹PATR is a subpart of the Ufal corpus amounting to around 22% of bio-medical data without substructure information, was thus tagged with `<M> <PATR> <US>`.</footnote>
Appendix C. Number Similar and Different sentences selected in
each decoding setting

1060

<table><caption>Table C.13: Sentence Comparison, Percentage and BLEU scores</caption><thead><tr><th rowspan="2">Direction</th><th rowspan="2">Models</th><th rowspan="2">Testsets</th><th colspan="2">Pre0-Pre1</th><th colspan="2">Pre0-Pre2</th><th colspan="2">Pre1-Pre2</th><th colspan="2">Pre0-Pre3</th><th colspan="2">Pre1-Pre3</th><th colspan="2">Pre2-Pre3</th></tr><tr><th>Similar</th><th>Different</th><th>Similar</th><th>Different</th><th>Similar</th><th>Different</th><th>Similar</th><th>Different</th><th>Similar</th><th>Different</th><th>Similar</th><th>Different</th></tr></thead><tbody><tr><td rowspan="10">en-fr</td><td rowspan="3">tagall</td><td>medline20</td><td>99.16%</td><td>0.84%</td><td>60.00%</td><td>40.00%</td><td>60.00%</td><td>40.00%</td><td>30.80%</td><td>69.20%</td><td>30.80%</td><td>69.20%</td><td>37.87%</td><td>62.13%</td></tr><tr><td></td><td>(39.2)</td><td>(45.2 - 45.3)</td><td>(44.9)</td><td>(38.3 - 39.6)</td><td>(44.9)</td><td>(38.3 - 39.5)</td><td>(46.6)</td><td>(38.8 - 39.6)</td><td>(46.6)</td><td>(38.8 - 39.6)</td><td>(46.3)</td><td>(39.4 - 39.1)</td></tr><tr><td></td><td>(98.67%)</td><td>1.33%</td><td>56.93%</td><td>43.07%</td><td>57.29%</td><td>42.71%</td><td>32.98%</td><td>67.02%</td><td>33.26%</td><td>66.80%</td><td>36.84%</td><td>63.96%</td></tr><tr><td></td><td>scielo</td><td>(36.0)</td><td>(32.4 - 33.2)</td><td>(38.7)</td><td>(34.2 - 35.1)</td><td>(38.8)</td><td>(34.2 - 35.1)</td><td>(43.2)</td><td>(35.0 - 36.2)</td><td>(43.2)</td><td>(35.0 - 36.1)</td><td>(41.7)</td><td>(35.6 - 36.1)</td></tr><tr><td rowspan="3">edp17</td><td></td><td>98.71%</td><td>1.29%</td><td>40.03%</td><td>59.97%</td><td>40.19%</td><td>59.81%</td><td>31.27%</td><td>68.73%</td><td>31.43%</td><td>68.57%</td><td>42.88%</td><td>57.12%</td></tr><tr><td></td><td>(31.4)</td><td>(29.3 - 30.6)</td><td>(38.0)</td><td>(30.3 - 33.7)</td><td>(38.0)</td><td>(30.3 - 33.7)</td><td>(39.5)</td><td>(30.9 - 34.2)</td><td>(39.9)</td><td>(30.9 - 34.2)</td><td>(37.6)</td><td>(33.5 - 33.9)</td></tr><tr><td></td><td>100%</td><td>0%</td><td>53.69%</td><td>46.31%</td><td>53.69%</td><td>46.31%</td><td>50.84%</td><td>49.96%</td><td>50.84%</td><td>49.96%</td><td>84.64%</td><td>15.36%</td></tr><tr><td></td><td>medline20</td><td>(40.2)</td><td>-</td><td>(43.1)</td><td>(38.7 - 40.3)</td><td>(43.1)</td><td>(38.7 - 40.3)</td><td>(44.4)</td><td>(38.4 - 39.4)</td><td>(44.4)</td><td>(38.4 - 39.4)</td><td>(42.4)</td><td>(37.3 - 35.6)</td></tr><tr><td></td><td></td><td>scielo</td><td>100%</td><td>0%</td><td>53.40%</td><td>46.60%</td><td>53.40%</td><td>46.60%</td><td>52.54%</td><td>47.46%</td><td>52.54%</td><td>47.46%</td><td>8.86%</td><td>91.14%</td></tr><tr><td rowspan="3">tagall-f-biomed</td><td></td><td>(35.5)</td><td>-</td><td>(38.4)</td><td>(34.1 - 36.0)</td><td>(38.4)</td><td>(34.1 - 36.0)</td><td>(38.2)</td><td>(34.2 - 36.0)</td><td>(38.2)</td><td>(34.2 - 36.0)</td><td>(37.1)</td><td>(34.4 - 33.8)</td></tr><tr><td></td><td>100%</td><td>0%</td><td>43.95%</td><td>56.05%</td><td>43.95%</td><td>56.05%</td><td>43.58%</td><td>56.42%</td><td>43.58%</td><td>56.42%</td><td>95.00%</td><td>5.00%</td></tr><tr><td></td><td>(34.4)</td><td>-</td><td>(42.1)</td><td>(32.7 - 36.8)</td><td>(42.1)</td><td>(32.7 - 36.8)</td><td>(41.8)</td><td>(32.8 - 36.8)</td><td>(41.8)</td><td>(32.8 - 36.8)</td><td>(37.9)</td><td>(37.2 - 35.7)</td></tr><tr><td rowspan="10">fr-en</td><td rowspan="3">tagall</td><td>medline20</td><td>99.85%</td><td>0.15%</td><td>64.41%</td><td>35.59%</td><td>64.33%</td><td>35.67%</td><td>58.48%</td><td>41.52%</td><td>58.49%</td><td>41.60%</td><td>83.50%</td><td>16.60%</td></tr><tr><td></td><td>(40.0)</td><td>(43.3 - 47.4)</td><td>(40.9)</td><td>(38.1 - 39.2)</td><td>(41.0)</td><td>(39.2 - 39.2)</td><td>(41.4)</td><td>(38.9 - 39.1)</td><td>(41.5)</td><td>(38.9 - 39.1)</td><td>(41.0)</td><td>(35.9 - 36.2)</td></tr><tr><td></td><td>(99.86%)</td><td>0.14%</td><td>52.77%</td><td>47.23%</td><td>52.78%</td><td>47.22%</td><td>52.27%</td><td>47.73%</td><td>52.28%</td><td>47.72%</td><td>92.33%</td><td>7.67%</td></tr><tr><td></td><td>scielo</td><td>(35.2)</td><td>(44.2 - 43.1)</td><td>(38.9)</td><td>(33.5 - 35.0)</td><td>(38.9)</td><td>(33.5 - 35.0)</td><td>(39.0)</td><td>(33.5 - 35.0)</td><td>(39.0)</td><td>(33.5 - 35.0)</td><td>(36.7)</td><td>(32.6 - 32.7)</td></tr><tr><td rowspan="3">edp17</td><td></td><td>99.57%</td><td>0.43%</td><td>59.38%</td><td>40.62%</td><td>59.43%</td><td>40.57%</td><td>58.46%</td><td>41.54%</td><td>58.41%</td><td>41.59%</td><td>92.37%</td><td>7.63%</td></tr><tr><td></td><td>(32.2)</td><td>(41.6 - 42.1)</td><td>(34.6)</td><td>(30.7 - 31.0)</td><td>(34.6)</td><td>(30.7 - 31.0)</td><td>(34.7)</td><td>(30.7 - 31.0)</td><td>(34.6)</td><td>(30.7 - 31.1)</td><td>(32.9)</td><td>(28.4 - 28.7)</td></tr><tr><td></td><td>100%</td><td>0%</td><td>66.31%</td><td>33.69%</td><td>66.31%</td><td>33.69%</td><td>60.23%</td><td>39.77%</td><td>60.23%</td><td>39.77%</td><td>84.41%</td><td>15.59%</td></tr><tr><td></td><td>medline20</td><td>(42.7)</td><td>-</td><td>(43.6)</td><td>(41.6 - 42.7)</td><td>(43.6)</td><td>(41.6 - 42.7)</td><td>(44.4)</td><td>(41.0 - 42.0)</td><td>(44.4)</td><td>(41.0 - 42.0)</td><td>(44.3)</td><td>(37.7 - 37.7)</td></tr><tr><td rowspan="3">tagall-f-biomed</td><td></td><td>scielo</td><td>100%</td><td>0%</td><td>46.76%</td><td>53.24%</td><td>46.76%</td><td>53.24%</td><td>46.56%</td><td>53.44%</td><td>46.56%</td><td>53.44%</td><td>91.15%</td><td>8.85%</td></tr><tr><td></td><td>(36.0)</td><td>-</td><td>(41.0)</td><td>(34.4 - 36.7)</td><td>(41.0)</td><td>(34.4 - 36.7)</td><td>(41.3)</td><td>(34.4 - 36.7)</td><td>(41.3)</td><td>(34.4 - 36.7)</td><td>(38.4)</td><td>(33.2 - 33.7)</td></tr><tr><td></td><td>100%</td><td>0%</td><td>55.83%</td><td>44.17%</td><td>55.83%</td><td>44.17%</td><td>56.18%</td><td>43.90%</td><td>56.18%</td><td>43.90%</td><td>90.22%</td><td>9.78%</td></tr><tr><td></td><td></td><td>edp17</td><td>(34.1)</td><td>-</td><td>(38.4)</td><td>(31.7 - 33.6)</td><td>(38.4)</td><td>(31.7 - 33.6)</td><td>(38.6)</td><td>(31.5 - 33.2)</td><td>(38.6)</td><td>(31.5 - 33.2)</td><td>(36.0)</td><td>(29.8 - 28.7)</td></tr></tbody></table>