SimulPL: Aligning Human Preferences in Simultaneous Machine Translation

Authors: Donglei Yu, Yang Zhao, Jie Zhu, Yangyifan Xu, Yu Zhou, Chengqing Zong
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: simultaneous machine translation, simultaneous preference optimization, human preferences

Abstract:
Simultaneous Machine Translation (SiMT) generates translations while receiving streaming source inputs. This requires the SiMT model to learn a read/write policy, deciding when to translate and when to wait for more source input. Numerous linguistic studies indicate that audiences in SiMT scenarios have distinct preferences, such as accurate translations, simpler syntax, and no unnecessary latency. Aligning SiMT models with these human preferences is crucial to improve their performances. However, this issue still remains unexplored. Additionally, preference optimization for SiMT task is also challenging. Existing methods focus solely on optimizing the generated responses, ignoring human preferences related to latency and the optimization of read/write policy during the preference optimization phase. To address these challenges, we propose Simultaneous Preference Learning (SimulPL), a preference learning framework tailored for the SiMT task. In the SimulPL framework, we categorize SiMT human preferences into five aspects: **translation quality preference**, **monotonicity preference**, **key point preference**, **simplicity preference**, and **latency preference**. By leveraging the first four preferences, we construct human preference prompts to efficiently guide GPT-4/4o in generating preference data for the SiMT task. In the preference optimization phase, SimulPL integrates **latency preference** into the optimization objective and enables SiMT models to improve the read/write policy, thereby aligning with human preferences more effectively. Experimental results indicate that SimulPL exhibits better alignment with human preferences across all latency levels in Zh$\rightarrow$En, De$\rightarrow$En and En$\rightarrow$Zh SiMT tasks.  Our data and code will be available at https://github.com/EurekaForNLP/SimulPL.

Decision: accept (poster)
Reviewer scores: 8, 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1730148158409

thread_type:
official_review

<thread_object>
creator: reviewer_1tvx

<content>
summary:
This paper introduces a new approach to simultaneous interpretation with low latency.

strengths:
The experimental results look impressive.

I think there will also be interest in the experimental materials.  Your arguments about offline machine
translation (OMT)  are convincing.  That said, I didn't see any discussion of sharing those materials.

I like the idea of offering the user choices over preferences, but do we need to restrict ourselves to a single choice?  Could you imagine a speech-to-text scenario where you could offer multiple text windows so the user can look at one window when latency is a priority and another window when BLEU is a priority?

weaknesses:
I have never been that happy with the standard discussion of latency and BLEU scores in this literature.  I have quite a bit of experience with professional interpreters as well as automatic solutions.  According to a number of standard metrics, the automatic solutions are better than professional interpreters, but I know that I am more engaged in the meeting when I have access to an interpreter.  They tell me what I need to know when I need to know it.  In addition, sometimes the speaker uses a metaphor that requires more explanation.    The professionals understand when then need to do more than just translate what the speaker said.

I never liked average latency.  I can't fault you for doing what others have done, but sometimes latency matters and sometimes it doesn't.  The professionals understand this.  They speed up when necessary and slow down when latency is less of a priority.  Moreover, I probably care more about worst case latency or RMS latency than average latency.

This literature would benefit by including more input from professional interpreters that have a lot more experience in this area than we have.

I would really like to see some data from schools that teach professional interpreters. We should be able to record what students say in the booth when they learn how to do this task.

questions:
Do you have any plans to share your benchmark?

Do you have a github that will make it easy for others to replicate your work?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730397433000

thread_type:
official_review

<thread_object>
creator: reviewer_qwwc

<content>
summary:
This paper studies the problem of Simultaneous Machine Translation (SiMT). To achieve preference alignment for SiMT models, this paper categorizes human preferences in SiMT scenarios and focus on five aspects: translation quality preference, monotonicity preference, key point preference, simplicity preference, and latency preference. Based on this catogorization, this pape rconstructs human preference prompts with GPT to generate preference data for SiMT. After constructing the data, this paper designs a multi-task supervised fine-tuning stage and a simultaneous direct preference optimization stage to achieve preference alignment for the SiMT model. Experiments on text-to-text SiMT tasks show the effectiveness of the proposed approach.

strengths:
1. The proposed approach shows performance improvement on the machine translation datasets.
2. The proposed five aspects for SiMT preference alignment could be useful.
3. The presentation is clear and easy to understand.

weaknesses:
1. Heavy reliance on the quality of the GPT-generated preference data, and few effort is done to verify the quality/correctness/trustworthiness of the generated data.
2. Overall limited novelty for the whole framework, the idea of length penalty term |y| in the proposed SimulDPO loss function come from [1], and nothing much novel is added here.
3. Scope is very limited to SiMT only, and this method does not seem applicable to more general LLM preference alignment cases.
4. Some design is arbitrary. For example, in Confidence-Based Policy During Inference, why set the confidence c_t as exactly 0.5? What if your confidence estimation model is biased, e.g., $\mathbb{E}(c_t)\neq0.5$?
5. More experiments and analysis should be done to conduct a comprehensive evaluation. In the current form, experiment analysis are limited.


References:
[1] Park, Ryan, et al. "Disentangling length from quality in direct preference optimization." arXiv preprint arXiv:2403.19159 (2024).

questions:
Why most experiments are done between translation of Zh and En? Should possibly incorporate more languages to demonstrate the effectiveness of the approach.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730719650712

thread_type:
official_review

<thread_object>
creator: reviewer_tojp

<content>
summary:
The paper introduces Simultaneous Preference Learning (SimulPL) for the task of Simultaneous Machine Translation. The approach encompasses not only data construction but also model training and inference strategies. However, the novelty of the proposed method appears to be insufficient.

strengths:
The paper presents a comprehensive approach to Simultaneous Machine Translation, addressing data construction, model training, and inference.

weaknesses:
1. The paper lacks a clear emphasis on the innovative aspects of the proposed method. Highlighting specific novel contributions would strengthen the paper.

2. It remains unclear how the quality of the automatic labeling by GPT-4 is ensured, and the consistency rate with human labeling is not provided.

3. The distinction between the uppercase X and lowercase x in equations 6 and 7 needs clarification.

4. According to Figure 6, the improvement of SimulPL over only MSFT is not substantial, raising questions about the effectiveness of the proposed method.

questions:
please see weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730756340883

thread_type:
official_review

<thread_object>
creator: reviewer_kp65

<content>
summary:
This works present a preference learning framework for simultaneous tasks like SiMT. Particularly, the proposed SimulPL framework categorizes SiMT human preferences into five aspects: translation quality preference, monotonicity preference, key point preference, simplicity preference, and latency preference.

strengths:
1. This is an intriguing question and a useful framework for practical SiMT scenarios. While SiMT has been extensively studied in the research community, most papers have focused on automatic metrics rather than real human preferences. This work proposed 5 human preferences that are closely related to the end-users of SiMT.

2. The proposed optimization objective aligns well with the proposed human preferences. For example, the output length constraint is applied for latency preference. The derived final objective in Eq(9) is further to improve its read/write policy.

weaknesses:
1. The data construction relies on GPT-4/4o. What is the cost of construction the training data? Does it mean GPT-4/4o is the upper bound of human preference alignment? Or this work is to GPT-4 preference alignment?  Figure 2 only shows the win-rate between GPT-4 translation and original one. Is there a comparison between GPT-4 and professional interpreter?

2. In the experiments, you only tune the hyper-parameter $\alpha$, and set $c_t$ as 0.5 to control the read/write. In general, tuning the thresholds of $c_t$ may introduce various read/write strategies. Did you try other threshold?

questions:
1. Since real-time translation is a key requirement of SiMT, LLM-based SiMT may need to re-translate each partial source sentence, as the decoder-only architecture of LLMs does not permit the insertion of additional source tokens without disrupting the positional indexes of the already translated target tokens. I want the authors to justify this issue.

2. Continued on Weaknesses 1, given some GPT-4/4o translations, is that possible to ask the professional interpreter to select the high quality translation? Then a classifier or reward model can be implemented to filter out the translations align most with human.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732088419484

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response: Further Evaluation of GPT-generated Data. (Part II)

comment:
**For human evaluation**, following Kocmi et al. [1] and Xu et al. [2], we randomly sample 200 sentences from the test set and employ professional interpreters, who are not involved in the revision process, to manually assess their GPT-generated translations and manually-revised translations. Our evaluation criteria are as follows:

- **0**: The translation is poor and fails to convey any meaningful information.
- **2**: The translation conveys some of the speaker’s information but misses key points. It also includes unnecessary reordering, leading to unnecessary delays in real-time scenarios, and uses less common expressions that make it harder for the audience to quickly understand.
- **4**: The translation conveys all the important information with minimal unnecessary reordering. It uses simple expressions that generally meet the audience's needs, though there are still some minor issues.
- **6**: The translation is a perfect simultaneous interpretation, accurately conveying the speaker's key points while omitting unnecessary details. It uses expressions that align with spoken language conventions, significantly reflecting human preferences in simultaneous interpretation.

Due to time constraints, we conduct this evaluation only on the Zh$\rightarrow$En test set. We calculate their average scores, win-tie-loss rates, and the distribution of their scores. The results are as follows:

| References       | Average Score | Win-Tie-Lose |            |           | Distribution of Scores |       |        |        |
| ---------------- | ------------- | ------------ | ---------- | --------- | ---------------------- | ----- | ------ | ------ |
|                  |               | win ratio    | lose ratio | tie ratio | 0                      | 2     | 4      | 6      |
| GPT-generated    | 5.37          | 6.00%        | 12.00%     | 82.00%    | 0.50%                  | 5.50% | 19.00% | 75.00% |
| Manually Revised | 5.57          | 12.00%       | 6.00%      | 82.00%    | 0.00%                  | 1.00% | 19.50% | 79.50% |

As shown in the table, the scores of GPT-generated translations are very close to those are manually revised, with only a 12% loss ratio. The score distribution shows that 94% of the GPT-generated translations scored 4 or higher. These results suggest that the GPT-generated data aligns well with SiMT human preferences.

**Reference**

[1] Tom Kocmi, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Novák, Martin Popel, and Maja Popović. 2022. Findings of the 2022 Conference on Machine Translation (WMT22). In *Proceedings of the Seventh Conference on Machine Translation (WMT)*, pages 1–45, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

[2] Xu, Haoran, et al. "Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation." *Forty-first International Conference on Machine Learning*.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732088487918

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response: Further Evaluation of GPT-generated Data. (Part I)

comment:
The references in our test sets are manually revised by professional interpreters, which is mentioned in lines 221-222 on Page 5 of our paper. To further validate the quality of our constructed dataset, we add an analysis of GPT-generated translations and those manually revised by interpreters on the test set. The experiments include both automated evaluation and human evaluation. 

**For automated evaluation**, we compare the multi-aspect evaluation results of GPT-generated translations and the manually revised translations. To facilitate comparison, we also provide the multi-aspect evaluation results for the original references of the test set. These results are shown as follows:

| references       | Zh$\rightarrow$En   |                 |                  |                | De$\rightarrow$En   |                 |                  |                | En$\rightarrow$Zh   |                 |                  |                |
| ---------------- | ------------------- | --------------- | ---------------- | -------------- | ------------------- | --------------- | ---------------- | -------------- | ------------------- | --------------- | ---------------- | -------------- |
|                  | NIR(%) $\downarrow$ | DD $\downarrow$ | SLR $\downarrow$ | Ref-free COMET | NIR(%) $\downarrow$ | DD $\downarrow$ | SLR $\downarrow$ | Ref-free COMET | NIR(%) $\downarrow$ | DD $\downarrow$ | SLR $\downarrow$ | Ref-free COMET |
| GPT-generated    | 6.48                | 6.55            | 0.49             | 78.32          | 5.49                | 5.71            | 0.92             | 80.86          | 10.65               | 5.63            | 0.88             | 81.24          |
| Manually Revised | 6.71                | 6.69            | 0.52             | 79.05          | 5.52                | 5.84            | 0.95             | 81.89          | 10.47               | 5.63            | 0.88             | 81.42          |
| Origin           | 13.21               | 7.53            | 0.74             | 79.05          | 7.80                | 6.29            | 1.09             | 80.78          | 12.89               | 6.19            | 0.89             | 75.77          |

As shown in the table, the translations generated by GPT-4/4o are either clearly superior to or comparable with the original references in terms of monotonicity, key points, simplicity, and translation quality. Moreover, these results are very close to the manually revised translations. This indicates that the quality of the GPT-generated data is both reliable and aligned well with human preferences.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732628353160

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to Reviewer Feedback and Updates to Our Paper.

comment:
We would like to thank all the reviewers for their valuable feedback!

We are pleased to hear that the reviewers found the study of human preferences in SiMT scenarios **a fascinating topic** (Reviewer kP65, Reviewer 1TVX); that **our SimulPL framework is considered practical for SiMT scenarios** (Reviewer kP65, Reviewer TojP); that **our categorized five aspects of human preferences  for SiMT tasks resonate with users** (Reviewer kP65, Reviewer qWwc); that **our experimental results are comprehensive and convincing** (Reviewer TojP, Reviewer qWwc, Reviewer 1TVX); and that the paper is **clear and easy to understand** (Reviewer qWwc).

We have addressed all the reviewers' comments and revised the paper accordingly. The revised version primarily includes the following significant updates, with modified sections marked in red.

1. In the Introduction, we **clarified our contributions more clearly** (Reviewer TojP).
2. We have further clarified **the distinction between $\mathrm{X}$ and  $\mathrm{x}$  in Equations 6 and 7** (Reviewer TojP).
3. In Section 5.6, we **discussed the generalization of SimulPL to other tasks** (Reviewer qWwc).
4. In Appendix A.2, we **added further evaluation of GPT-generated data** (Reviewer kP65, Reviewer TojP, Reviewer qWwc).
5. In Appendix C, we **explained the difference between our length-based constraints for latency preference and existing methods** (Reviewer qWwc).
6. In Appendix D, we **analyzed the impact of the confidence threshold on SimulPL’s performance during inference** (Reviewer kP65, Reviewer qWwc).
7. In Appendix G, we **provided additional analysis experiments for De$\rightarrow$En task** (Reviewer qWwc).

We hope that our revisions address the reviewers' concerns. If so, we sincerely hope that the reviewers might consider raising the scores in light of the updates. We promise to further improve the paper in our final revision.

**Finally, we sincerely thank all the reviewers, ACs, and PCs for their time and effort!**
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a framework for simultaneous machine translation (SiMT) that aligns with human preferences, categorized into five aspects: translation quality, monotonicity, key points, simplicity, and latency.

The paper's strengths include its comprehensive approach to addressing human preferences in SiMT, the innovative use of GPT-4/4o for generating preference data, and the convincing experimental results demonstrating significant improvements in alignment with human preferences. The paper is well-written and easy to understand, making complex ideas accessible to the reader.

The weaknesses include the reliance on GPT-4/4o for data construction, raising questions about the cost and quality of the generated data, and the lack of detailed comparisons with professional interpreters. The presentation could be improved, particularly in sections discussing the empirical validation and the logical flow between different parts of the paper.

All the reviewers are positive about the paper. I recommend accepting it.