# Lingua Custodia's participation at the WMT 2025 Terminology shared task

Jingshu Liu, Raheel Qader, Gaëtan Caillaut, Mariam Nakhlé

<figure><img src="image_1.png" /></figure>

## To cite this version:

Jingshu Liu, Raheel Qader, Gaëtan Caillaut, Mariam Nakhlé. Lingua Custodia's participation at the WMT 2025 Terminology shared task. 2025. hal-05502287

HAL Id: hal-05502287

https://hal.science/hal-05502287v1

Preprint submitted on 10 Feb 2026

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
arXiv:2510.17504v1 [cs.CL] 20 Oct 2025

# Lingua Custodia's participation at the WMT 2025 Terminology shared task

Jingshu Liu

Raheel Qader

Gaëtan Caillaut

Mariam Nakhlé

Lingua Custodia, France

{jingshu.liu,mariam.nakhle,gaetan.caillaut,raheel.qader}@linguacustodia.com

## Abstract

This paper presents Lingua Custodia's submission to the WMT25 shared task on Terminology shared task targeting three language pairs: English-German, English-Spanish, and English-Russian. Our approach builds on two recent instruction-tuned large language models *Qwen3-4b* and *Gemma3-4b-it* which we further finetune in a two-stage process. In the first stage, we perform supervised fine-tuning (SFT) on parallel data augmented with explicit terminology constraints. In the second stage, we apply Group Relative Policy Optimization (GRPO) with a custom reward function designed to encourage correct terminology usage while preserving general translation quality. Experimental results on English-German and English-Spanish language pairs show >= 10 points of accuracy gain in term match compared to their baseline counterparts and >= 5 points for English-Russian, while maintaining similar Comet-Kiwi scores.

## 1 Introduction

Nowadays both classical encode-decoder (Sutskever et al., 2014; Bahdanau et al., 2014; Luong et al., 2015; Vaswani et al., 2017) and decoder only LLM based neural machine translation systems (Alves et al., 2024; Cheng et al., 2025; Gwen et al., 2025) have demonstrated strong performance across a wide range of translation tasks, with the latter increasingly favored due to their scalability, multilingual generalization, and ability to leverage instruction-tuning for domain adaptation.

We leverage LLM-based MT systems due to their ease of fine-tuning and scalability, which make them well-suited for rapidly adapting to new domains and tasks. This paper describes our submission to the WMT25 Terminology translation task in English to German, English to Spanish and English to Russian. The task aims to assess the

extent to which machine translation models can utilize additional information regarding the translation of terminologies. Previous works on machine translation with terminology control can be grouped into two categories according to whether the method needs training the model with terminology information. One group incorporates the constraint during the inference time (Hokamp and Liu, 2017; Post and Vilar, 2018; Susanto et al., 2020). An LLM trained without any terminology constraint can also be seen as this category with zero or few shot prompting during the inference. These methods can typically satisfy most of the constraints but suffer from high computational cost and sometimes low translation quality because they always try to strictly apply the terminology constraint regardless of the correctness of the whole sentence. The other group integrates lexical constraints during training (Dinu et al., 2019; Crego et al., 2016; Song et al., 2019; Alves et al., 2024) by annotating terminology in order to guide the model to learn the enforcement of the translation constraints. The main disadvantage of these methods is the lack of guarantee of all constraints in the translations. Another limitation of these works is that they usually requires extra resources such as a term dictionary, an annotator (a big LLM) to augment the data.

Our work extends the prior approaches of (Ailem et al., 2021) and (Liu et al., 2023), which follow the second line of methods. Furthermore, we build our systems by finetuning two of the most prominent open large language models: *Qwen3* (Qwen et al., 2025) and *Gemma3* (Team et al., 2025).

We evaluate our work on the WMT25 terminology track1 testsets. Since the reference is not released by the time of writing this paper, we evaluate our system by a simple naive strict match with respect to the target constraint completed by the Comet-Kiwi (Rei et al., 2022) score to make sure that the translation ability is sustained. Results
where *M* is the set of source–target terminology pairs extracted from the prompt, and *M*<sub>target</sub> is the set of target terms. If no terminology is specified in the prompt, we assign a default reward of 1.0.

These rewards are computed per sample and can
be combined in a weighted manner during GRPO
training to simultaneously improve general transla-
tion quality and enforce terminology constraints.

3 Experiments

Our experiments are designed to compare the LLM
baselines with our fine-tuned Qwen3 and Gemma3
models, measuring both translation quality perfor-
mance and terminology adherence on the WMT25
benchmarks. The following sections detail the
training datasets, the experimental settings, and
the results obtained.

3.1 Data

We apply LaBSE (Feng et al., 2022) on the Com-
mon Crawl and WMT25 English-Russian collected
datasets. Pairs with a similarity score below 0.9 are
discarded. From the remaining data, we randomly
sample 10,000 parallel sentences for each language
pair to construct the supervised fine-tuning dataset.
For the GRPO training phase, we sample an addi-
tional 1,000 parallel sentences per language pair
from the filtered data. Hence our final training
dataset contains 33,000 samples. The dataset is
then automatically annotated with terminology con-
trol information with our unsupervised dictionaries.

An example of a training sample:

User prompt:

Generate a Spanish translation that accurately
reflects the terminology specified in
clause → cláusula.

Text to translate: To this end, it must comply
with the WTO requirements and, in particular,
with the GATT enabling clause cláusula of 1979.

**Assistant response:** A tal fin, debe cumplir con
los requisitos de la OMC y, en especial, con la
cláusula de habilitación del GATT de 1979.

<caption>Figure 1: Example of a training sample with terminol-
ogy control. Only the bleu part is calculated for the loss
during the trainig. We vary the user prompt using the
10 templates previously generated.</caption>

## 3.2 Settings

For the supervised finetuning (SFT) phase, we train
Qwen3-4B and Gemma3-4B-IT using the AdamW
optimizer with a learning rate of 5 × 10⁻⁶, weight

decay of 0.01, and 1000 warmup steps. Each train-
ing batch per device contains 16 samples, and the
maximum sequence length is set to 2048 tokens.
Training is performed in bf16 mixed-precision
mode for computational efficiency. Gradient clip-
ping is applied at a maximum norm of 1.0. The
finetuning runs on 4 H100 GPUs using FSDP2
(Zhang et al., 2024) for memory efficiency.

For the GRPO phase, the per-device batch size is
reduced to 8, and for each sample 16 distinct com-
pletions are generated to compute rewards. The re-
ward function combines a BLEU-based score and a
terminology adherence score with equal weighting
(see Section2.3). GRPO training is carried out on 4
nodes of 4 H100 GPU machines using DeepSpeed
ZeRO-3 (Rajbhandari et al., 2020), bf16 precision,
and the AdamW optimizer with the same learning
rate and weight decay as in SFT.

3.3 Results

We evaluate our systems on the translation con-
straint success rate by a simple strict match because
by the time of our naive evaluation, the reference
was not available. We report our results obtained
with baseline, SFT finetuned and SFT+GRPO mod-
els on the WMT25 testset with the these settings of
WMT25:

*   **No terminology**: the system is only provided with input sentences/documents.

*   **Proper terminology**: the system is provided with input texts and terminology dictionaries.

*   **Random terminology**: the system is pro-
    provided with input texts and translation dictio-
    naries that contain words randomly drawn
    from input texts.

Overall, we observe that baseline instruction-
tuned models (Qwen3-4b, Gemma3-4b-it) achieve
competitive COMET scores across all settings,
with slightly higher scores in the *no terminol-
ogy* condition which is expected because the
task is classical translation without any con-
straint. In terminology-constrained settings, *proper
terminology* leads to moderate term accuracy
(around 0.40–0.55), whereas *random terminology*
consistently yields much higher term accuracy
(0.77–0.96), as the random words from the input
text are usually general meaning words and some-
times stop words.

After the first finetuning phase (Qwen3_SFT,
Gemma3_SFT), term accuracy generally improves
<table><caption>Table 1: Results for language pairs: EN→DE, EN→ES, and EN→RU. We report Comet-Kiwi and terminology match accuracy (denoted by term%). Qwen3_SFT and Gemma3_SFT are the models after the first finetuning phase, and Qwen3_SFT_GRPO and Gemma3_SFT_GRPO the models after the GRPO phase.</caption><thead><tr><th rowspan="2">Model</th><th rowspan="2">Setting</th><th colspan="2">EN→DE</th><th colspan="2">EN→ES</th><th colspan="2">EN→RU</th></tr><tr><th>comet-kiwi</th><th>term%</th><th>comet-kiwi</th><th>term%</th><th>comet-kiwi</th><th>term%</th></tr></thead><tbody><tr><td rowspan="3">Qwen3-4b</td><td>proper</td><td><strong>0.7979</strong></td><td>0.5525</td><td>0.8106</td><td>0.5242</td><td>0.8035</td><td>0.4280</td></tr><tr><td>random</td><td><strong>0.7917</strong></td><td>0.7724</td><td>0.8081</td><td>0.8202</td><td><strong>0.7959</strong></td><td>0.7228</td></tr><tr><td>noterm</td><td>0.8022</td><td>-</td><td>0.8183</td><td>-</td><td>0.8064</td><td>-</td></tr><tr><td rowspan="3">Gemma3-4b-it</td><td>proper</td><td>0.7887</td><td>0.5562</td><td>0.8106</td><td>0.5465</td><td>0.8025</td><td>0.4008</td></tr><tr><td>random</td><td>0.7831</td><td>0.7659</td><td>0.8005</td><td>0.8063</td><td>0.7893</td><td>0.7120</td></tr><tr><td>noterm</td><td>0.8000</td><td>-</td><td>0.8219</td><td>-</td><td>0.8094</td><td>-</td></tr><tr><td rowspan="3">Qwen3_SFT</td><td>proper</td><td>0.7755</td><td>0.5543</td><td>0.8060</td><td>0.5892</td><td>0.7942</td><td>0.4144</td></tr><tr><td>random</td><td>0.7741</td><td>0.8065</td><td>0.8035</td><td>0.8709</td><td>0.7872</td><td>0.7246</td></tr><tr><td>noterm</td><td>0.7901</td><td>-</td><td>0.8147</td><td>-</td><td>0.8030</td><td>-</td></tr><tr><td rowspan="3">Gemma3_SFT</td><td>proper</td><td>0.7924</td><td>0.5561</td><td><strong>0.8148</strong></td><td>0.5725</td><td><strong>0.8038</strong></td><td>0.3988</td></tr><tr><td>random</td><td>0.7893</td><td>0.7854</td><td><strong>0.8149</strong></td><td>0.8691</td><td>0.7887</td><td>0.7246</td></tr><tr><td>noterm</td><td>0.7971</td><td>-</td><td><strong>0.8237</strong></td><td>-</td><td>0.8107</td><td>-</td></tr><tr><td rowspan="3">Qwen3_SFT_GRPO</td><td>proper</td><td>0.7062</td><td><strong>0.7053</strong></td><td>0.7631</td><td><strong>0.6208</strong></td><td>0.7388</td><td><strong>0.5545</strong></td></tr><tr><td>random</td><td>0.6905</td><td><strong>0.9577</strong></td><td>0.7413</td><td><strong>0.9525</strong></td><td>0.7034</td><td><strong>0.9656</strong></td></tr><tr><td>noterm</td><td>0.7755</td><td>-</td><td>0.8118</td><td>-</td><td>0.7815</td><td>-</td></tr><tr><td rowspan="3">Gemma3_SFT_GRPO</td><td>proper</td><td>0.7873</td><td>0.6980</td><td>0.8017</td><td>0.6097</td><td>0.8008</td><td>0.4669</td></tr><tr><td>random</td><td>0.7861</td><td>0.9106</td><td>0.8023</td><td>0.9337</td><td>0.7813</td><td>0.8351</td></tr><tr><td>noterm</td><td><strong>0.8058</strong></td><td>-</td><td>0.8023</td><td>-</td><td><strong>0.8120</strong></td><td>-</td></tr></tbody></table>

for both the *proper* and *random* terminology settings, while maintaining similar Comet-Kiwi scores. Finally, the GRPO phase produces a substantial boost in term accuracy for *proper* and *random terminology* conditions. For example, Qwen3_SFT_GRPO achieves over 0.95 term accuracy in the *random* setting across all language pairs. However, this improvement in terminology adherence comes at the cost of a moderate drop in COMET, particularly for Qwen3 models. Across models, Gemma3 variants maintain Comet-Kiwi more effectively after GRPO than Qwen3 variants, suggesting a better trade-off between translation quality and terminology accuracy. Therefore a good compromise model could be Gemma3_SFT_GRPO since it has similar term accuracy except for the English-Russian pair but holds solid Comet-Kiwi scores.

Language pair differences are also notable: EN→ES generally attains slightly higher Comet-Kiwi scores than EN→DE and EN→RU, regardless of the setting, while EN→RU tends to show the lowest term accuracy in *proper terminology* for the base models (down to 0.40 for Gemma3-

4b-it). This outcome is unsurprising, as the baseline systems already show weaker performance on EN→RU.

In summary, the proposed GRPO is highly effective when maximizing terminology adherence is the primary goal. However, if the aim is to balance translation quality and term accuracy, SFT-only models or Gemma3_SFT_GRPO provide a more favorable trade-off, as they retain higher Comet-Kiwi while still improving term accuracy.

## 4 Conclusion

In this work, we present our systems for the WMT25 Terminology Shared Task. We fine-tuned Qwen3-4b and Gemma3-4b-it in a two-stage process, combining supervised fine-tuning (SFT) with Group Relative Policy Optimization (GRPO). The fine-tuned models achieved substantial improvements in terminology match accuracy while maintaining strong COMET-Kiwi scores across all three evaluated language pairs. These results demonstrate the effectiveness of integrating terminology-aware training objectives into both SFT and reinforcement learning optimization, enabling models
to better adhere to lexical constraints without sacrificing overall translation quality. Future work will explore extending terminology-aware training to additional language pairs and low-resource scenarios, investigating more sophisticated reward functions tailored for terminology adherence. Additionally, we plan to study the interaction between terminology control and other translation tasks such as inline tags to build an "all-in-one" multilingual and multi-task translation model.

## References

Melissa Ailem, Jingshu Liu, and Raheel Qader. 2021. Encouraging neural machine translation to satisfy terminology constraints. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1450–1455, Online. Association for Computational Linguistics.

Duarte M. Alves, José Pombal, Nuno M. Guerreiro, Pedro H. Martins, João Alves, Amin Farajian, Ben Peeters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, Pierre Colombo, José G. C. de Souza, and André F. T. Martins. 2024. Tower: An open multilingual large language model for translation-related tasks. Preprint, arXiv:2402.17733.

Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2016. Learning principled bilingual mappings of word embeddings while preserving monolingual invariance. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP’16), pages 2289–2294, Austin, TX, USA.

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben-gio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.

Shanbo Cheng, Yu Bao, Qian Cao, Luyang Huang, Liyan Kang, Zhicheng Liu, Yu Lu, Wenhao Zhu, Jing-wen Chen, Zhichao Huang, Tao Li, Yifu Li, Huiying Lin, Sitong Liu, Ningxin Peng, Shuaijie She, Lu Xu, Nuo Xu, Sen Yang, and 7 others. 2025. Seed-x: Building strong multilingual translation llm with 7b parameters. Preprint, arXiv:2507.13618.

Common Crawl. 2023. Common crawl - open repository of web crawl data. https://commoncrawl.org. Retrieved: 22 November 2023.

Josep Crego, Jungi Kim, Guillaume Klein, Anabel Re-bollo, Kathy Yang, Jean Senellart, Egor Akhanov, Patrice Brunelle, Aurelien Coquard, Yongchao Deng, and 1 others. 2016. Systran’s pure neural machine translation systems. arXiv preprint arXiv:1610.05540.

Georgiana Dinu, Prashant Mathur, Marcello Federico, and Yaser Al-Onaizan. 2019. Training neural machine translation to apply terminology constraints.

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, page 3063–3068.

Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Ari-vazhagan, and Wei Wang. 2022. Language-agnostic BERT sentence embedding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 878–891, Dublin, Ireland. Association for Computational Linguistics.

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schel-ten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mi-tra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.

Amir Hazem and Emmanuel Morin. 2016. Efficient data selection for bilingual terminology extraction from comparable corpora. In Proceedings of the 26th International Conference on Computational Linguistics (COLING’16), pages 3401–3411, Osaka, Japan.

Chris Hokamp and Qun Liu. 2017. Lexically constrained decoding for sequence generation using grid beam search. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, page 1535–1546.

Jingshu Liu, Emmanuel Morin, and Sebastián Peña Sal-darriaga. 2018. Towards a unified framework for bilingual terminology extraction of single-word and multi-word terms. In Proceedings of the 27th International Conference on Computational Linguistics (COLING’18), pages 2855–2866, Santa Fe, NM, USA.

Jingshu Liu, Mariam Nakhlé, Gaëtan Caillout, and Ra-heel Qadar. 2023. Lingua custodia’s participation at the WMT 2023 terminology shared task. In Proceedings of the Eighth Conference on Machine Translation, pages 897–901, Singapore. Association for Computational Linguistics.

Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, page 1412–1421.

Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191, Belgium, Brussels. Association for Computational Linguistics.

Matt Post and David Vilar. 2018. Fast lexically constrained decoding with dynamic beam allocation for neural machine translation. Proceedings of NAACL-HLT 2018, page 1314–1324.
Qwen, ;, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. Qwen3-mt. https://modelscope.cn/studios/Qwen/Qwen3-MT-demo.

Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. Preprint, arXiv:1910.02054.

Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. T. Martins. 2022. CometKiwi: IST-unbabel 2022 submission for the quality estimation shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 634–645, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

Kai Song, Yue Zhang, Heng Yu, Weihua Luo, Kun Wang, and Min Zhang. 2019. Code-switching for enhancing nmt with pre-specified translation. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), page 449–459.

Raymond Hendy Susanto, Shamil Chollampatt, and Lil- ing Tan. 2020. Lexically constrained neural machine translation with levenshtein transformer. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, page 3536–3543.

Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. arXiv preprint arXiv:1409.3215.

Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, and 197 others. 2025. Gemma 3 technical report. Preprint, arXiv:2503.19786.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008.

An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.

Ruisi Zhang, Tianyu Liu, Will Feng, Andrew Gu, San- ket Purandare, Wanchao Liang, and Francisco Massa. 2024. Simplefsdp: Simpler fully sharded data paral- lel with torch.compile. Preprint, arXiv:2411.00284.