Cultivar: A Contrastive and Locale-Oriented Translation Benchmark for Investigating Contamination and Localisation Robustness
Authors: name: Pinzhen Chen, name: Koel Dutta Chowdhury, name: Xiaoya Xu, name: David Tan, name: Doreen Osmelak, name: Ona de Gibert, name: Ariun-Erdene Tumurchuluun, name: Ashok Urlana, name: Fedor Sizov, name: Hale Sirin, name: Jesujoba Alabi, name: Karrar Talib Abed, name: Mateusz Klimaszewski, name: Nikolay Bogoychev, name: Niyati Bafna, name: Patricia Schmidtova, name: Preksha Manjunath Shanbhag, name: Sherrie Shen, name: Vilem Zouhar, name: Vivek Iyer, name: Yasser Hamidullah, name: Yusser Al Ghussin, name: Zheng Zhao
Organization: Queen's University Belfast

Multilingual translation benchmarks are typically sourced in English and translated into other languages, treating language pairs as the unit of evaluation---a design that is prone to contamination over time and overlooks locale and cultural considerations. We therefore advocate for source-contrastive evaluation and instantiate it with Cultivar, a localised subset of FLORES, which enables locale-specific translation evaluation. When paired with unlocalised counterparts, performance discrepancy allows the probing of data contamination and localisation robustness. We benchmark 32 open-weight models and find that MT-specialised models are less robust, a few models potentially overfit FLORES, and models tend to translate US content better than that of other locales, regardless of language.