Contrastive Preference Optimization: Pushing the Boundaries of LLM
  Performance in Machine Translation
Authors: name: Haoran Xu, name: Amr Sharaf, name: Yunmo Chen, name: Weiting Tan, name: Lingfeng Shen, name: Benjamin Van Durme, name: Kenton Murray, name: Young Jin Kim

Moderate-sized large language models (LLMs) -- those with 7B or 13B
parameters -- exhibit promising machine translation (MT) performance. However,
even the top-performing 13B LLM-based translation models, like ALMA, does not
match the performance of state-of-the-art conventional encoder-decoder
translation models or larger-scale LLMs such as GPT-4. In this study, we bridge
this performance gap. We first assess the shortcomings of supervised
fine-tuning for LLMs in the MT task, emphasizing the quality issues present in
the reference data, despite being human-generated. Then, in contrast to SFT
which mimics reference translations, we introduce Contrastive Preference
Optimization (CPO), a novel approach that trains models to avoid generating
adequate but not perfect translations. Applying CPO to ALMA models with only
22K parallel sentences and 12M parameters yields significant improvements. The
resulting model, called ALMA-R, can match or exceed the performance of the WMT
competition winners and GPT-4 on WMT'21, WMT'22 and WMT'23 test datasets.