# The SPECTRANS System Description for the WMT22 Biomedical Task

Nicolas Ballier, Jean-Baptiste Yunès, Guillaume Wisniewski, Lichao Zhu, Maria Zimina-Poirot

► To cite this version:

Nicolas Ballier, Jean-Baptiste Yunès, Guillaume Wisniewski, Lichao Zhu, Maria Zimina-Poirot. The SPECTRANS System Description for the WMT22 Biomedical Task. 7th conference on machine translation (WMT22), EMNLP 2022, Dec 2022, Abu Dhabi, United Arab Emirates. . hal-03895294

HAL Id: hal-03895294

https://hal.science/hal-03895294

Submitted on 12 Dec 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# The SPECTRANS System Description for the WMT22 Biomedical Task

<figure><img src="image_2.png" /></figure>

Nicolas Ballier¹, Jean-Baptiste Yunès², Guillaume Wisniewski³, Lichao Zhu¹, Maria Zimina-Poirot¹

¹CLILLAC-ARP, ²IRIF, ³LLF, Université Paris Cité, F-75013 Paris, France
{nicolas.ballier, guillaume.wisniewski, jean-baptiste.yunes, lichao.zhu, maria.zimina-poirot}@u-paris.fr

## 1. Our approaches to the task

* to use open source NMT toolkit JoeyNMT and Systran Pure Neural Server® to train and fine-tune biomedical NMT models (en↔fr) based on specialized biomedical corpora (UFAL, WMT, Khresmoi);
* to compare and analyze translations predicted by the models;
* to interpret and understand performances of the models.

<table><caption>## 2. Data and training for baseline model and fine-tuning</caption><thead><tr><th rowspan="2">Technique</th><th rowspan="2">Run</th><th rowspan="2">Corpus</th><th colspan="2">BLEU</th><th colspan="2">sacreBLEU</th><th rowspan="2">Toolkit</th></tr><tr><th>fr→en</th><th>en→fr</th><th>fr→en</th><th>en→fr</th></tr></thead><tbody><tr><td>Baseline model (in-domain)</td><td>run 1</td><td>UFAL</td><td>0.258</td><td>0.207</td><td>61.01</td><td>59.23</td><td>JoeyNMT</td></tr><tr><td>SYSTRAN Pure Neural Server</td><td>run 2</td><td>SYSTRAN in-house corpora</td><td>0.401</td><td>0.316</td><td>/</td><td>/</td><td>SYSTRAN</td></tr><tr><td>ModelStudio Light</td><td>run 3</td><td>in-house data</td><td>0.259</td><td>0.073</td><td>/</td><td>/</td><td>SYSTRAN</td></tr><tr><td>Baseline model (out-of-domain) + fine-tuning (in-domain)</td><td>run 4</td><td>Europarl 7 + UFAL</td><td>0.097</td><td>0.203</td><td>18.60</td><td>21.13</td><td>JoeyNMT</td></tr><tr><td></td><td></td><td>Europarl 7 + WMT and Khresmoi</td><td>/</td><td>/</td><td>54.8</td><td>38.4</td><td>JoeyNMT</td></tr></tbody></table>

## 3. Analysis and metrics of the translations

Comparison of occurrences of "must" and "should" in SYSTRAN and JoeyNMT translations with *iTrameur*

<table><thead><tr><th>Cooc</th><th>FqCooc total</th><th>FqCooc contexte</th><th>IndSP</th></tr></thead><tbody><tr><td>must</td><td>7</td><td>14</td><td>27</td></tr><tr><td>surgery</td><td>2</td><td>4</td><td>7</td></tr><tr><td>sleep</td><td>5</td><td>4</td><td>6</td></tr><tr><td>be</td><td>30</td><td>8</td><td>5</td></tr></tbody></table>

<table><thead><tr><th>N°</th><th>Systran Translation</th><th>JoeyNMT Translation</th></tr></thead><tbody><tr><td>1</td><td>It is these human traits that a rational organization of research <strong>must</strong> try to promote and exploit.</td><td>this is therefore those of human interest that a rational research organization <strong>should</strong> attempt to encourage and operate.</td></tr><tr><td>2</td><td>Other parasomias, presenting Dreams or fragments of dysphoric dreams, <strong>must be</strong> distinguished from nightmares, and their management is different: These are mainly night terror, <strong>sleep</strong>-related hallucinations and behavioral disorder In REM <strong>sleep</strong>.</td><td>other parasomias, presenting <strong>dreams</strong> or fragments of dysphoric <strong>dreams</strong>, are indistinguishable from nightmare, and are mainly nocturnal ground, hallucinations related to sleep and rem behavioral disorder.</td></tr><tr><td>3</td><td>Manufacturers of medical devices <strong>must</strong> demonstrate, often through clinical trials, the safety, performance and clinical benefit of their products.</td><td>manufacturers of medical devices <strong>should</strong> show, often using clinical trials, safety, performance and clinical benefit of the products.</td></tr><tr><td>4</td><td>The treatment of pvh <strong>must be</strong> comprehensive, it requires taking into account all these aspects, medical, psychic, social, and involving patients.</td><td>consideration <strong>should</strong> be given to managing the conditions of all such aspects, medical, psyche, social, and patient management.</td></tr><tr><td>5</td><td>Parkinson's syndrome is then associated with other symptoms called "red flags", which <strong>must be</strong> sought during interrogation and physical examination.</td><td>parkinsonian syndrome is then associated with other symptoms called " red flags ", which <strong>should</strong> be considered for interpreting and physical examination.</td></tr><tr><td>6</td><td>Titration remains necessary and maximum tolerated doses <strong>must be</strong> reached.</td><td>titration remains necessary and maximum tolerated doses <strong>should</strong> be reached.</td></tr><tr><td>7</td><td>A multidisciplinary approach <strong>must</strong> involve expertise in orthopedic <strong>surgery</strong>, musculoskeletal imaging and nuclear medicine, infectious diseases, as well as plastic or <strong>vascular surgery</strong> for cases with soft tissue loss or vascularization defect.</td><td>a multidisciplinary approach <strong>should</strong> include specialists for orthopedic <strong>surgery</strong>, musculoskeletal and nuclear medicine imaging, infectious diseases, and in plastic or <strong>vascular surgery</strong> for cases with loss of soft tissue or <strong>vascular</strong> defects.</td></tr><tr><td>8</td><td>The treatment of pvh <strong>must be</strong> comprehensive, it requires taking into account all these aspects, medical, psychic, social, and involving patients.</td><td>consideration <strong>should</strong> be given to managing the conditions of all such aspects, medical, psyche, social, and patient management.</td></tr><tr><td>9</td><td>Other parasomias, presenting Dreams or fragments of dysphoric dreams, <strong>must be</strong> distinguished from nightmares, and their management is different: These are mainly night terror, <strong>sleep</strong>-related hallucinations and behavioral disorder In REM <strong>sleep</strong>.</td><td>other parasomias, presenting <strong>dreams</strong> or fragments of dysphoric <strong>dreams</strong>, are indistinguishable from nightmare, and are mainly nocturnal ground, hallucinations related to sleep and rem behavioral disorder.</td></tr><tr><td>10</td><td>Titration remains necessary and maximum tolerated doses <strong>must be</strong> reached.</td><td>titration remains necessary and maximum tolerated doses <strong>should</strong> be reached.</td></tr></tbody></table>

* overuses of "must" in the SYSTRAN translation (IndSP = +5) and "should" in JoeyNMT
* large frequency differences of both words in training data : 18,462 ("should") vs. 4,061 ("must"); occurrence preponderances seemingly unbalanced the translation predictions

## 4. Vocabulary comparisons

Comparison of Vocabulary Growth Curves in SYSTRAN PNS and mBART translations

<figure><img src="image_3.png" /></figure>

SYSTRAN did slightly better than mBART, with less word variations.

Comparison of Specificity Vocabulary Growth Curves in Systran PNS and mBART translations

<figure><img src="image_4.png" /></figure>

The French feminine determiner "la" before "COVID" is more correctly predicated by PNS that mBART.

## 5. Conclusion

We used a variety of strategies, toolkit comparison and fine-tuning to compare outcomes of different NMT systems in biomedical translation. Our contribution lies in the textometric analysis of the output. This allowed us to raise the issue of the role of the variability observed for the gender of COVID in French or for technical terms like "keloids".

## 6. Acknowledgements

The SPECTRANS project is funded under the 2020 UPCité émergence research project under the ANR grant ANR-18-IDEX-0001,and supported by the PAPTAN Platform. We gratefully acknowledge support from the CNRS/TGIR HUMA-NUM and IN2P3 Computing Center (Lyon - France)