Learning from others' mistakes: Finetuning machine translation models with span-level error annotations
Authors: Lily H Zhang
Hamid Dadkhahi
Mara Finkelstein
Firas Trabelsi
Jiaming Luo
Markus Freitag
Venue: ICML 2025 poster (2025)
Keywords: machine translation
fine-grained annotations
multidimensional quality metrics
Primary area: applications->language_speech_and_dialog
TL;DR: We propose Training with Annotations (TWA) to improve the quality of machine translation models with imperfect examples that include detailed error annotations.

Abstract: Despite growing interest in incorporating feedback to improve language models, most efforts focus only on sequence-level annotations. In this work, we explore the potential of utilizing fine-grained span-level annotations from offline datasets to improve model quality. We develop a simple finetuning algorithm, called Training with Annotations (TWA), to directly train machine translation models on such annotated data. TWA utilizes targeted span-level error information while also flexibly learning what to penalize within a span. Moreover, TWA considers the overall trajectory of a sequence when deciding which non-error spans to utilize as positive signals. Experiments on English-German and Chinese-English machine translation show that TWA outperforms baselines such as supervised finetuning on sequences filtered for quality and Direct Preference Optimization on pairs constructed from the same data.

Learning from others' mistakes:
Finetuning machine translation models with span-level error annotations
Lily H. Zhang 12 Hamid Dadkhahi² Mara Finkelstein Firas Trabelsi
Jiaming Luo? Markus Freitag²
Abstract
Most efforrs, however, consider only sequence-level labels, usually in the form of scalar score assigned to the en- tire output. In contrast, this work investigates the poten- tial of using fine- fine-grained span-level annotations from of- fline datasets to enhance language model training. Unlike sequence- level annotations, span-level annotations provide information about specific segments within sequence, of- fering more detailed information for model learning. More- over, in many situations, collecting fine-grained information is similar effort to collecting sequence-level labels (Wu et al., 2023), making the former a practical form of data for im- proving model performance given a method that can take advantage of the information.
Despite growing interest in incorporating feed- back to improve language models, most efforts focus only on sequence-level annotations. In this work, we explore the potential of utilizing fine-grained span-level annotations from offline datasets improve model quality. We develop simple finetuning algorithm, called Training with Annotations (TWA), to directly train ma- chine translation models on this annotated data. TWA utilizes targeted span-level error informa- tion while also flexibly learning what to penal- ize within a span. Moreover, TWA considers the overall trajectory of sequence when decid- ing which non-error spans to utilize as positive signals. Experiments on English- German and Chinese-English machine translation show that TWA outperforms baselines such as supervised finetuning on sequences filtered for quality and Di- rect Preference Optimization on pairs constructed from the same data.
To explore the potential of fine-grained annotations, fo- cus on the setting of machine translation, an application where substantial amounts of fine-grained information is already readily available, namely Multidimensional Quality Metrics (MQM) data from previous Workshop on Machine Translation (WMT) Shared Tasks (Freitag et al., 2021a). This data, used to evaluate the quality of machine trans- lation systems, contains span-level annotations of the er- present in given translation well their category (e.g., fluency, accuracy) and severity (e.g., major and mi- nor). While MQM data has previously been used to develop auxiliary reward or metrics models (Juraska et al., 2023; Rei et al., 2022), it has not been directly employed for training machine translation (MT) models. Positive results in this set- ting can motivate the collection of fine-grained annotations to improve text generation tasks in other settings.
English-German
1. Introduction
Language models have advanced to the point where it is often difficult to improve them substantially via super- vised finetuning on high-quality human-written examples alone; instead, recent efforts to improve language model or sequence-to-sete toesuueecc model performance have largely lied on annotations of model generations, from preferences to per-sequence scores (Bai et al., 2022; Ethayarajh et al., 2022; Lambert et al., 2023; Kopf et al., 2023). Such data, coupled with techniques to learn from it (Christiano et al., 2017; Rafailov et al., 2023; Gulcehre et al., 2023; Dong al., 2023), have yielded impressive results for many top language models.
To directly utilize these translations and their span-level annotations to finetune an MT model, we introduce a new algorithm called Training with Annotations (TWA). TWA utilizes span-level information from the annotations to treat error and non-error spans differently. For error spans, the TWA loss seeks to decrease the probability of the span given the context while allowing the model to learn which tokens in the span to penalize do so. For non-error tokens, TWA takes into account the overall sequence trajectory when deciding which spans should be treated as positive signals. A high-level summary of TWA can found in Figure I.
'Center for Data Science, New York University Google. respondence to: Lily H. Zhang <lily.h. @ @ny..edu>.
"Googl Cor-
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Experiments on Inglish-German and Chinese-Englise ma- chine translation demonstrate that TWA yields significant

---

Learning from others' mistakes: Finctuning machine translation models with span-level error annotations
Give me an example of blessing in adversity.
Minor error
Give me
story about
blessing in disguise
Major error
Give
a
story
about
blessing
dis
guise
1
-1
-1
0
-5
-5
X
X
Crass entropy
Lspan( -- W| log 1
exp
log p+)
Unlikellhodd loss
tEspan
Ignore off-trajectory non-error tokens
Figure Overview of Training with Annotations (TWA). TWA proceeds by tokenizing oulput text and annotations. Then, a weighted span-level unlikelihood loss is applied to each error span to allow the model to learn what parts of the error span to penalize and non-error tokens following an error span are ignored as they are off-trajectory. All other tokens (i.c., non-crror tokens preceding an error span) are trained with cross entropy loss.
improvements over baselines which either do not consider annotation information only utilize the information at the sequence level. Specifically, TWA can outperform meth- ods such as supervised finetuning on sequences filtered for quality and Direct Preference Optimization (DPO) on pref- erence pairs constructed from the same data. These results highlight the effectiveness of taking advantage of span-level annotations improve model performance.
MQM annotations have been used to evaluate MT systems, described above, but not additional training signal to finetune MT models. Utilizing these annotations during training requires developing method that can take this information into account. We describe our proposed method, Training with Annotations, in Section 4.
3. Related Work
First, we describe the MQM data and the information pro- vided in the span-level annotations (Section 2). Then, we discuss existing work which either utilizes the MQM data or the fine-grained annotations (Section 3). Then, we introduce our method, Training with Annotations (TWA), in Section 4. We outline our experimental setup Section and present the results in Section 6. Finally, we conclude with a discus- sion of our findings and future work in Section 7.
Utilizing MQM data. TWA is the first method span-level MQM data to directly finetune machine trans- lation models, but there exist other methods which utilize MQM data to train other models, including automated met- rics and instruction-tunel language models. Namely, exist- ing automated metrics in machine translation such Met- rieX (Juraska et al., 2023) utilize MQM scores labels for training data, methods which utilize these neural-basse based automated metrics indirectly benefit from MQM data. Such approaches include QE reranking (Fernandes et al., 2022) or MBR decoding (Freitag et al., 2022) with neural quality metrics. Both methods can be used in tandem with TWA, as one could always decode TWA-trained model with either of these approaches. One could also use the results of such decoding methods to directly finetune model, commonly known as MBR or QE finetuning (Finkelstein & Freitag, 2024). However, given the models powering automated met- rics such as Metric-X are trained on multiple sources of data beyond that of MQM data alone, MBR and QE finetuning are not directly comparable with TWA. Jiao et (2023); Zeng et al. (2023) reformat MQM data as an instruction following task, e.g.. "Translate the following sentence... with major accuracy emrors... and find that supervised fine tuning in this manner can improve translation quality, e.g.. via "Translate the following sentence... with no errors..." However, the instructions incur additional computational
2. MQM data
Each year, the Workshop on Machine Translation (WMT) hosts a shared task competition to assess general machine translation capabilities across different domains and genres. Submitted MT systems scored and evaluated by humans, with top systems annotated via the Multidimensional Qual- ity Metrics (MQM) scheme (Freitag et al., 2021b; Rei et al., 2022). Namely, given the source text and MT output, pro- fessional translators annotate any error spans in the output translation. Each error span is annotated with the category of the error as well as the severity of the error. Each error span is assigned a score of 25 for a non-translation, for major error, 0.1 for minor punctuation error, and for any other minor error. The overall MQM score of an example sequence is the sum of the MQM scores of the annotated error spans in the sequence.
2

---

Learning from others' mistakes: Finctuning machine translation models with span-level error annotations
and memory costs relative direct translation, and TWA differs in its direct use of error signal as negative information rather than an example for the task of mistranslation.
training, the TWA loss for a given sequence is sum of the losses from the error spans and the non-error tokens. Below, we describe and motivate the choices for the constituent losses.
Utilizing fine-grained annotations. There exist other methods which consider fine-grained annotations, but they consider a different setting than TWA. Fine-grained RLHF (FG-RLHF) (Wu et al., 2023) adapts RLHF to reward mod- els which provide finer-grained feedback than a single sequence-level score. Such fine-grained feedback could even come from automated metric trained with MQM data, such as xCOMET as in Ramos et al. (2024). Similar to our work, Wu et al. (2023) achieve better performance using fine-grann tine-grained RLHF with span-level rewards than using RLHF with sequence-level rewards. The difference between FG-RLHF and TWA is that the former is a reinforcement learning method that requires an auxiliary fine-grained re- ward model annotate model generations online, while the latter is finetuning method that can work directly with of- fline annotated data without the need for additional models during training. The performance of FG-RLHF depends on the quality of the fine-grained annotator model, which can be difficult to develop (see Pang et al. (2023) and Ap- pendix D). Moreover, accuracy of the annotations aside, a reinforcement learning approach which only takes into account online data misses out the opportunity to learn from offline examples themselves, not just their annotations.
4.1. Handling error spans
An annotated error span provides information to the model that such a continuation is undesirable given the preceding context (and thus should be unlikely under the model). To decrease the probability of error spans given their context, TWA utilizes the unlikelihood loss, - log(1 - p). The loss is high when the probability p is high and 0 when p zero. In Section 6, consider alternative choices of loss for error tokens and find that the unlikelihood loss outperforms other choices. Moreover, the unlikelihood loss is efficient to compute as it only requires access to the current model being trained.
Applying unlikelihood to each token in an error span may not be desirable, however. Take the output in Figure 1, for example. Imagine the correct translation was "Give example of blessing in adversity", but the submitted trans- lation was "Give me story about a blessing in disguise", as shown in the figure. Moreover, say the sequence was tokenized in the way shown in the figure, with "disguise" being tokenized into "dis" and "guise". First, even though "disguise" is an inaccurate translation of "adversity". "guise" is perhaps the most reasonable continuation of the sequence given the prefix ends with "blessing in dis". Penalizing "guise" given its prefix does not necessarily reflect the inten- tion of the error span; rather, it is probably more appropriate to assign a low probability to "dis" given its prefix while maintaining a high probability for "guise" given a prefix ending in "a blessing in dis". Second, the error annotation around "a story about" does necessarily mean that the article 'a' and the preposition "about" should be assigned low probability given their prefixes. The above examples are just few instances of the broader idea that not all tokens in an error span should be penalized.
an
Next, Targeted Negative Training (TNT) (Zhang et al., 2024) is a method for training on token-level annotations of neg- ative examples, but its motivation is to achieve a targeted update, i.e., reducing unwanted behavior while minimally changing the model otherwise. TWA, on the other hand, is not concerned with making precise updates but rather improving overall quality as much as possible. Finally, FUDGE (Yang & Klein, 2021) is an alternative decod- ing technique which utilizes a token-level auxiliary reward model to sample from the model conditioned on a given at- tribute a; namely, given reward model which approximates p(alyge 2), FUDGE samples from p(yc 1y<c. z, a) using the original model p(yelV<c,,a ar) and reward model p(aly<==,,>) .r)- TWA, on the other hand, is a finetuning-based approach that does not alter the test-time behavior of the model and does not require an auxiliary reward model.
Given these examples and others, one might be able to come up with a series of heuristics to transform the resulting span- level errors into corresponding token-level losses. However, as is common in natural language, manually creating rules can be difficult and error- arror-prone (whether due to low preci- sion or recall). Instead, we choose to let the model learn what to penalize within a span by utilizing a span-level un- likelihood term instead of a token-level one. We additionally take into account the severity of the error by scaling the loss by the absolute value of the severity weight w assigned to the span, equal to the error span's 's negative MQM score: -0.1 for minor punctuation, -1 for all other minor errors, and -5
4. Training with Annotations
Training with annotations (TWA) a finetuning algorithm that takes into account example outputs and their lean-level error annotations. TWA proceeds as follows: first, the ex- ample is tokenized and given weights corresponding to its annotations: tokens which contain any characters within an error span are given& a negative weight, and tokens outside an error span are given a non-negative weight. Then, during
3

---

Learning from others' mistakes: Finctuning machine translation models with span-level error annotations
for major errors.' The loss for error span is the following:
4.3. Overall method
Combining the insights from the above two sections, we have simple finetuning algorithm for TWA as depicted in Figure 1. First, we tokenize the output sequence and its cor- responding annotations. The latter become weights which are negative values for tokens with characters contained in an annotated error span, zero for all tokens following the first error span, and one for all other non-error tokens, Then, we group tokens into spans based on weight (i.e., all contiguous tokens with the same weight are in the same span) and employ either the TWA error span loss or the TWA non-error span loss. The overall TWA loss for a given sequence is the sum of all the span losses.
LTwA (error span) ||w| log( [1
Popan)
= -|| log(1 [1
exp logpt)
EExpan
Rather than forcing the model to push down probability over all tokens in a span given their prefixes, the span-level unlikelihood loss allows the model to learn which tokens to penalize in order decrease the overall probability of the span.
4.2. Handling non-error spans
When the overall quality of the data is high relative to the base model, using supervised finetuning (SFT) maximize the likelihood of the translations in the data can improve the model. On the other hand, when the overall quality of the data is low relative to the model, SFT can hurt perfor- mance, by teaching the model to reproduce errors. Thus, to optimize model quality, most efforts seek to filter out low-quality examples and train just on high-quality ones. However, in reality, there is likely often spectrum of trans- lation quality even within an example itself. Fine-graid -ineing annotations provide extra information about this variation in quality by pinpointing exactly where errors exist. Then, for all other tokens, we can proceed with typical maximum likelihood training via cross entropy loss, without worrying about maximizing the likelihood of errors.
5. Experiments
5.1. Data
Pretraining. We pretrain En->De and Zh-+En models us- ing the parallel WMT' training data (Koemi et al., 2023), which consists of 296 million sentence-level examples. For En->ee, we additionally construct multi-sentence examples from a subset of this data where the overall documents can be recovered and partitioned into longer blocks than those of individual sentences. The multi-sentence examples have a max length of 1024 tokens, with 512 tokens each for the input source and output target.
Finetuning, For both language pairs, we then apply TWA on top of the pretrained model, using MQM data from WMT' 20 (Barrault) and WMT'21 (Akhbardeh) for train- ing. In total, the training dataset contains roughly 2,900 and 3, 100 source texts, with around 28,000 and 31,000 submis-
However, all the subsequent tokens after error out- oupsupp since their prefixes contain an error that should be low or zero probability under the intended new model. We call these subsequent tokens off-trajectory. Generaliza- tion aside, off-trajectory tokens at best are irrelevant to the model distribution and at worst could provide noisy signal. While there is an argument that high-quality off-trajectory tokens could provide signal that generalizes trajectories the model will actually sample, we find empirically that ignoring these tokens in the overall loss can greatly im- prove performance in some settings (see Table 4). TWA on non-error spans thus as follows:
sion outputs for En and Zh-+En, respectively (around ten submissions per source on average).
5.2. Base Model
For both language pairs (En~>De and Zh Zh-EEn,, we use 602-million-paramee
Transformer encoder-decoder archi- tecture implemented in Pax2. The model has 8 encoder and 8 decoder layers (rather than 6). but otherwise is similar to the transformer-big setting in Vaswani et al. (2017), with model dimension of 1024, hidden dimension of 8192, and 16 multi-attention heads. For each language pair, we use a bilingual vocabulary of 32k subword units trained on the WM training dataset (Kocmi et al., 2023). We pretrain with the standard cross entropy loss.
O if span after first error log Pspan otherwise.
CTwA (non-error span)
Note that this is equivalent employing per-token cross entropy loss on non-error tokens before error span, log Pspan Ersspan log,p. pt.
See Table 1 for a comparison of the quality of our base model relative to the average quality of the WMT'20- '21 submissions, and Table 2 for the range of quality across submissions (best and worst systems). On average, the
Under the MQM rating system, some major errors are given score of -25 (namely those categorized non-translations) but we use a weight of -5 for these emorr as well.
Pntops://githin cttpe/occo///oo/ooo 1/pxmm

---

Learning from others' mistakes: Finctuning machine translation models with span-level error annotations
submissions and run SFT on this filtered dataset. We call this baseline Filter SFT. Second, we also run sequence- level analogue to TWA, where we apply sequence-level unlikelihood loss to an output if it contains any error and cross entropy loss otherwise. We call this baseline TWA- seq.
Table 1. Quality of original base model and submissions data, all systems in aggregate.
En + De
Zh- + En
Metric-X
| COMET f Metric-X | COMET
model
2.132
0.406
4.529
0.326
data
1.301
0.525
3,414
0.376
For all the methods, we use a batch size of 8192 (4096 pairs for DPO), learning rate of 2e-6 with constant schedule, and no label smoothing. Greedy decoding is used through- out the experiments.
Table 2. Quality of best and worst system submissions.
En + De
Zh- + En
5.4. Evaluation
Metric-X + COMET
Metric-X + COMET
For evaluation, we use MetricX-23 -23 (Juraska al., 2023), COMET-20 (Rei et al., 2020), and BLEURT (Sellam et al., 2020) as quality metrics. MetricX-23 and BLEURT are both reference-basee bbased metrics which score a translation based on a reference and a hypothesis, without taking into account the source text. COMET-20 takes into account the source text, hypothesis, and reference translation. MetricX-23 has been finetuned MQM WMT 21 data, while COMET-20 and BLEURT have not. Given their differences, considering all three automated quality metrics together in the evalu- ation helps guard against overfitting to the idiosynerasies of any one in particular. Lower is better for MetricX-23, while higher is better for COMET-20 and BLEURT. For checkpoint selection, we average the values of MetricX-23 and the negative COMET-20 the validation set every 500 steps and select the checkpoint with the lowest (i.e., best) score. We then report all three evaluation metrics on the test set, employing BLEURT as a held-out evaluation. Through- out the rest of the paper, we use MetricX and COMET to denote MetricX-23 and COMET-20, respectively.
best
0.194
0.641
2.258
0.517
worst
2.043
0.192
3.573
0.193
submissions are higher quality than our starting base model.
See Appendix A for additional statistics between the base model and submissions data, including error token distri- butions (Figure 5) and histograms per-sequence of quality scores between model generations and data (Figure 6).
5.3. Baselines
We compare TWA with Supervised FineTunnng (SFT) and Direct Preference Optimization (DPO) (Rafailov et al., 2023) as baselines. SFT on the MQM annotated data is anal- ogous to distilling the outputs of other MT systems, without taking into account the annotations. DPO is a preference learning algorithm which operates on pairs of responses to the same input given the knowledge that one response in the pair is preferred to another. Many alternatives to DPO exist which also contrast pairs of responses, e.g., (Meng et al., 2024; Xu et al., 2024; Hong et al., 2024; Xu et al., 2025), and to compare against this pairwise setting, we construct response pairs for DPO from the same MQM data using the sequence-level MQM scores (i.e., the sum of the MQM scores of all the error spans), creating pairs from all com- binations of system translations to the same source input where the MQM score is distinct. We arrived at this setting after testing multiple variations; see Appendix C for details. In other words, DPO utilizes the annotations as additional information, but only at a sequence level.
We use the generalMT2022 test set (Kocmi al., 2022) as our validation set for checkpoint selection, and report all results on the WM (Kocmi al., 2023) test set. The validation set contains roughly 2, 000 and 1,900 source texts (along with their corresponding reference translations) for En-De and Zh-En, while the test set contains 600 and 2, 000 examples for En- and Zh-En, respectively. Note that the 2M En- test set is paragraph-level.
6. Results
6.1. Main Results
When using both submissions and references for finetuning, we treat references as error-free free for TWA and TWA-seq, and treat them as better than all submissions for constructing DPO pairs; the resulting dataset for DPO thus contains all the pairs constructed from submissions only, plus additional (reference, submission) pairs for every submission. We also consider two additional baselines. First, given the quality of the data makes a big difference in the efficacy of SFT, we construct a dataset of only the references and error-free
First, we compare TWA to the baselines described in Sec- tion 5.3. We perform experiments using the submissions data alone, as well as in tandem with the reference translations (one per source). Within each quality metric, we also report performance clusters based on sta- tistically significant performance differences between pairs. For each language pair and data source (i.e. submissions only vs. submissions+ we verify whether the
5

---

Learning from others' mistakes: Finctuning machine translation models with span-level error annotations
Table Results aggregated by language pair and automatic metric. We also indicate the data sources used for each result. Models with statistically significant performance improvements are grouped in quality clusters. We highlight the best ranked models in bold.
En + De
Zh + En
Submissions References Metric-X
COMET
BLEURT f
Metric-X /
COMET t
BLEURT
Base model
4.203
0.429
0.6412
4.938
0.066
0.6079
SFT
3,573 2
0,481 2
0.658 2
4.253 2
0.255 2
0.650 1
DPO
3.792 2
0.455
0.664
4.072
0.113 3
0.615 2
TWA
V
2.944 1
0.507
0.668
4.091
0.277 I
0.651
SFT
V
V
3.159 3
0,491
0.662 2
4.094
0.271 2
0.652 1
DPO
V
3.564 4
0.442 3
0.660 2
4.063 2
0.113 3
0.614 2
Filter SFT
V
2.950 2
0,499 2
0.670
4.004 2
0.289
0.652 1
TWA-seq
V
3,158 3
0,485 2
0.663 2
3,993
0.284
0.652 1
TWA
V
2.882 1
0.513
0.672
3.965
0.290
0.653
measured differences between each system pair is statisti- cally significant via a paired permutation test using 1000 re-sampling runs and significance level of p 0.05. We then group systems with similar performance by following the clustering procedure from (Freitag et al., 2023). Namely, given significance results (p-values) for all pairs of systems, we assign ranks as follows. Starting with the highest-scoring system, we move down the list of systems in descending order by score, and assign rank I to all systems until we encounter the first system that is significantly different from any that have been visited so far in the latter cluster. That system is assigned rank 2, and the process is repeated until all systems have been assigned a rank. This clustering is done independently for each automated metric.
consistent improvement over SFT suggests that even when the data is overall of better quality than the current model being finetuned (i.e., training on all the data still improves performance), it can still be beneficial to treat some spans differently than others. The fact that sequence-level base- lines that take into account negative information (i.e., DPO, TWA-seq) do not necessarily improve performance over SFT highlights the challenge of attribution when utilizing sequence-level information. Namely, both DPO and TWA- seq utilize more information than SFT (i.e., DPO takes into account that one sequence is preferred over another, while TWA-seq knows which sequences have errors and which ones are error-free), but they are not able to effectively utilize this information to gain a systematic improvement over a baseline that ignores this information. These results suggest that even when extra information is available, it is non-trivial to develop method which can effectively take advantage of this information. TWA, on the other hand, is able to take advantage of span-level annotation informa- tion to outperform SFT and Filter+SFT, highlighting the effectiveness of the method. TWA's improvement over Fil- ter + SFT (significant for En En) demonstrates that is able utilize useful signal that is otherwise thrown away with sequence-level filtering. Moreover, TWA is able to improve quality with more fine-grained modifications than other approaches, with a higher sacre-BI.EU of 34.24 and 25.30 on En and Zh-Enn submissions and references vs. 30.71 and 15.66 for DPO, indicating that its use of negative information more precise.
Table 3 summarizes the results. We find that TWA is consis- tently the most performant: it is within the top-performing cluster for all settings, feat shared by no other method (the next most consistently in the top cluster is TWA-seq, its sequence-level counterpart). TWA's differential perfor- mance over baselines is most pronounced for Metric-X, likely due to the metric's sensitivity to the quality captured across MQM data, and least pronounced for BLEURT, likely given its generally more concentrated range of values and the fact that the metric that was fully held out from the checkpoint selection process; even so, TWA still yields the highest BLEURT score in every setting, highlighting the generalizability of the results with respect to quality metrics.
All methods improve quality over the base model, which is in line with the fact that the submissions data are of higher quality overall than the base model's generations. TWA's
TWA yields the best score for all metrics across the board, with the exception of DPO on Zh using submissions data only. While DPO may seem better on Metric-X (though not significantly so), it is substantially and significantly worse on COMET (less than half the COMET score of TWA) and BLEURT, suggesting that DPO has exploited an idiosyneracy of the Metric-X model without truly improv- ing in overall performance. Although other methods will
'Considering each system as distribution the MetrieX or COMET scores for each nource-translation pair, we test how likely a given result between pairs of systems would be if their underlying distribution of scores were the same. In code, we use scipy.stats, stasttttsa... w.t.sttt.rruutt...... permutation.teest (*, stat peatissiicnppmmere mean, permutation.tyee samples')
6

---

Learning from others' mistakes: Finctuning machine translation models with span-level error annotations
occasionally also occupy the rank-1 cluster alongside TWA for a given metric and setting, no baseline does across metrics within setting except TWA-seq Zh-EEn using references and submissions data.
6.3. Negative Losses for TWA
A key component of TWA is how to utilize error spans as negative information. In Table 5, we compare the un- likelihood loss used in TWA with the negative likelihood loss also on the span level, i.e., (span))n) = log Pspan- Table 5 shows that unlikelihood greatly outperforms neg- ative likelihood. This is likely due to the fact that the negative likelihood only grows in its contribution to the loss and corresponding gradient as the probability of an error span goes zero (i.e., limp--o log Pspan -oo and & limp-o By log Pspan = oo) and can thus outweigh likelihood terms as the probability of positive spans moves towards 2 - log Pspan = (i.e., limp-11 - log Pspan = 0 and limp-11 op 1). In contrast, unlikelihood mirrors the loss and gradi- ent of likelihood as the span probability moves towards the desired result (i.e., limp- loog(1 Pspea) 0 and 2- log(1 Psspa) -1
To further test the robustness of TWA, we compare it to baselines on an additional language pair, En-+Zh, following the same experimental setup as described in Section 5 but using WMT- '22 MQM data for finetuning due to the lack of MQM data for the language pair from WMT-'20-21. We find that TWA significantly outperforms baselines in this experiment as well. Strikingly. TWA achieves not just top cluster results but significantly better performance than baselines even on the held-out metric BLEURT, which was not used for checkpoint selection. See Appendix B for details.
6.2. TWA Ablations
Next, we isolate the effect of the individual components of TWA in Table 4. Starting from the base model, we note first that training all the submissions (+ SFT submissions) improves results. Then, given knowledge of span-level errors, the most obvious next step is to treat the tokens with and without errors differently. Absent a method to deal with errors, the most straightforward next step is to include only the non-error tokens in the loss, ignoring the error tokens to prevent the model from maximizing the likelihood of them given their context. We see that this step (+ on non-error tokens only) improves results over training on all error tokens, confirming our hypothesis that training on error tokens negatively contributes to model quality. Then, we incorporate the TWA loss on error spans, whose tokens make up on average 11.0% and 13.6% of the total tokens in given translation (see Figure for additional statisties on the error- vs. non-error makeup of the data). This results in further improvements, demonstrating that it is possible to improve model quality by learning from negative information over ignoring errors entirely. Finally, we ignore off-trajectory tokens, which results in substantial gains in En+De but not in Zn->nn To understand the gains in En En-De, we compare the translations between the TWA variant that ignores off-trajectory tokens and the one that includes them in the loss, and we find that the former results in more fluent translations in contrast the latters more literal and sometimes awkward ones; see Appendix F for further details, including example generations. A possible explanation for the lack of gains in the latter could be that there is less room to differentially improve performance with fine- grained information, as evidenced by the fact that TWA is better than TWA-seq in Zh->En but not significantly so. Regardless, both versions of TWA (ignore off-trajectory vs. not) still outperform baselines.
6.4. Analyzing TWA
Next, we visualize how TWA changes the model distribution. For each submission output in the training data, we obtain its per-token log probabilities. Moreover, for cach token we record its log probability rank under the model relative to all other tokens in the vocabulary. Both can be obtained through a single forward pass. We obtain log probability ranks for both the original base model as well as the TWA- trained model and compute the change in rank for each token from the base model the since the model is decoded via greedy decoding, changes in rank are more indicative of behavior shifts than changes in log probability. We visualize the changes in rank for four different sample training examples in Figure 2. Notably, the configuration of tokens penalized within the error span varies across different samples, demonstrating the flexibility of span-level error loss in enabling the model to learn which tokens to penalize an outcome that would be challenging to encode manually with set of heuristics. Quantitatively, we also find that utilizing span-level error loss substantially outperforms using a token-level loss on each token in a span (3.325/0.495 MeMTTT vs. 3.433/0.470 for token- level on En submissions only).
model. Note that
6.5. TWA with on-policy samples
While the aforementioned experiments all utilize off- policy data generated from MT systems other than the one be- ing finetuned, next we test the efficacy of TWA in an on- policy setting. Concretely, we obtain MQM annotations of the base model's translations and run TWA with this an- notated data. We see substantial improvements in quality, from 4.203/0.4p2 /299 Metric- X/COMET. While these improvements from online data are not as large as those with the off- off-policy data, where the
to 3.710/0.45 Metric-
7

---

Learning from others' mistakes: Finetuning machine translation models with span-level error annotations
Table breakdown of the components of TWA and their isolated effect on model quality. Models with statistically significant performance improvements are grouped in quality clusters, and the best ranked scores are shown in bold.
En De
Zh En
Metric-X 4
COMET 1 Metric-X 4 COMET 1
Base model
4.203
0.429
4.938
0.066
SFT on submissions on non-error tokens only
3.573
0,481
4.253
0.255 3
3.488
0,487
4.120
0.283
span-level loss errors
3.325 2
0,495 2
4.088
0.284 1
ignore off-trajectory tokens
2.944
0.507
4.091
0.277 2
104
10
-10
100
UN
belong
to
the concentrate
special
hard ship area at the
southern
foot g
Da
X
ing
an
ling
-10
100
0
hoped
that
the
Vietne
ary
goverard
aad
rrelevat
functional
agencies of Korea
would provide assistance for the future operation of the association
-101
-101
-102
-102
0
compre Comprehensee hensive
10a
entertainment
-10
10c
They
are
all
young speciali ialist
different sub subject ect
Ster
g0od
at
ating
through
This
the
white
star
sea
POL 860
also
known
as
the
dead
horse sea DO
-10
-104
-10
-102
Figure 2. Change in the rank of each token in the vocabulary from the base model to the "WA-trained model. Dashed red lines indicate annotated errors. Red bars show worsening in rank, while green bars indicate improvement. TWA learns diverse patterns for penalizing specific token conditionals within an error span patne-ptterrns that would be challenging to capture with heuristics.
7. Discussion
Table 5. Comparison of negative losses for use error spans. We compare unlikelihood (UL), the choice TWA, with negative likelihood (NL).
In this work, introduce Training with Annotations (TWA), method for finetuning a language model data with span-level error annotations. While most existing efforts have focused on utilizing sequence-level annota- tions, TWA can take advantage finer-grained information for more effective learning. Our experiments on English- German and Chinese-English machine translation highlight the performance gains TWA offers compared to methods that focus solely sequence-lewe information.
En + De
Zh + En
Loss Metric-X & COMET f Metric-X COMET
UL
2.944
0.507
4.091
0.277
NL
3.477
0.491
4.730
0.108
As model capabilities continue to improve, it will be increas- ingly difficult to rely on the collection or construction of high-quality examples as training signals. In fact, many of the MT system submissions WMT'24 were found to sur- pass the quality of uumannccnstruuted reference translations, highlighting the need to move beyond demonstration data for improving existing models. MQM annotations of model generations offer valuable alternative source of informa- tion for model training, and TWA unlocks the potential to utilize such rich information directly and simply.
submissions data on average better quality than the base model's 's translations (see Table 1), the fact that TWA still significantly improves over the base model this setting speaks the ability of the method to take advantage anno- tation information alone, even when there no difference in translation quality between the model and data. This result provides positive signal in favor of the promise of utilizing TWA to improve upon already very high-quality models, without the need for even higher quality translations (just good error annotations).
8

---

Learning from others' mistakes: Finctuning machine translation models with span-level error annotations
Yet while the experiments focus on MQM data for the task of machine translation, TWA can be used for span-level annotations broadly, paving the way for other applications of fine-grained annotations. While fine- fine-grained informa- tion may be more expensive to collect than sequence-level information for some tasks, Wu et al. (2023) find that for long-form question-ane -anwerrng.....ring........ the time required for hu- mans to annotate span-level errors is comparable to the time required to label the sequence overall. Many other tasks likely fall into this same category; for instance, one needs to locate the hallucination in order to label sequence as "has hallucination", similarly, identifying specific spans of bias or misinformation is necessary before assigning a label such as "biased" or "inaccurate".
T., al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv, 2022.
Barrault, L. e. a. Findings of the 2020 conference on ma- chine translation (WMT20).
Blain, F., Zerva, C., Rei, R., Guerreiro, N. M., Kanojia, D., de Souza, J. G., Silva, B., Vaz, T., Jingxuan, Y., Azadi, F., et al. Findings of the wmt 2023 shared task quality estimation. In Proceedings of the Eighth Conference on Machine Translation, pp. 629 603, 2023.
Brown, T. B.B.C. e. a. Language models are few-shot learners. NeurIPS, 2020.
There exist multiple ways to build upon TWA. One avenue for future work would be to apply TWA in settings beyond machine translation or to language models in general. An- other would be to additionally take into account the fine- grained annotation information in other ways-for instance, given fine-ge -raigga information provides a natural ranking of inputs, one could consider directly providing the model with this relative quality information as well. Other interesting questions to investigate include assessing TWA on online data, analyzing the impact of the quality of the generations and annotations on resulting model performance, and ex- ploring the repeated use of TWA for iterative refinement of a model. Finally, the fact that ignoring off -ffjtrajc tokens was highly beneficial in one language pair but not in the other, provides an opportunity to further refine TWA to better handle off off-trajectory tokens since the latter might contain additional useful information for training.
Christiano, P. F., Leike, J., Brown, T. B., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. ArXiv, (ts//1706.07741,200 2017.
Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., Shum, K., and Zhang, T. Raft: Reward ranked finetuning for generative foundation model alignment. TMLR, 2023.
Ethayarajh, K., Choi, Y., and Swayamdipta, S. Understand- ing dataset difficulty with v-usable information. In ICML, 2022.
Fernandes, P., Farinhas, A., Rei, R., C. de Souza, J. G., Ogayo, P., Neubig, G., and Martins, A. Quality-aware decoding for neural machine translation. 2022.
Finkelstein, M. and Freitag, M. MBR and QE finetuning: Training-time distillation of the best and most expensive decoding methods. In The Twelfth International Confer- ence on Learning Representations, 2024. https: / //openreview. net/ www.mmdddddddxxxxookko
In summary. TWA offers straightforward method to cap- italize on existing span-level annotation data as well as a reason to begin collecting span-level information in appli- cations which currently do not. By taking advantage of previously overlooked sources of supervision, methods such as TWA can help unlock new avenues for pushing the fron- tier of model development.
Freitag, M., Foster, G., Grangier, D.,, Ratnakar, V., Tan, Q., and Macherey, W. Experts, errors, and context: large- scale study of human evaluation for machine translation. TACL. 2021a.
Impact Statement
Freitag, M., Foster, G., Grangier, D., Ratnakar, V., Tan, Q., and Macherey, W. Experts, errors, and context: A large- scale study of human evaluation for machine translation. Transactions of the Association for Computational Lin- guistics, 9, 2021b. URL https: h/tp///ollaantoo..... o org prg/2021-0 waa.ta.cc11111..
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
References
Freitag, M., Grangier, D., Tan, Q., and Liang, B. High quality rather than high model probability: Minimum Bayes risk decoding with neural metrics. Transactions of the Association for Computational Linguistics, 10, 2022. URL https: / /awaaaaaaalog..oooooooooo wwwpog....r///// 2022.
Akhbardeh, F. a. Findings of the 2021 conference machine translation (WMT21). ACL.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan,
tac1-1.47.
9

---

Learning from others' mistakes: Finctuning machine translation models with span-level error annotations
Freitag, M., Mathur, N., Lo, C.-k., Avramidis, E., Rei, R., Thompson, B., Koemi, T.., Blain, F., Deutsch, D., Stewart, C., Zerva, C.,, Castilho, S., Lavie, A., and Foster, G. Re- sults of WMT23 metrics shared task: Metrics might be guilty but references are not innocent. In Koehn, P., Had- dow, B., Koemi, T., and Monz, C. (eds.), Proceedings of the Eighth Conference on Machine Translation, pp. 578- 628, Singapore, 2023, Association for Computational Linguisties.
Mattick, A. Openassistant conversations democratizing large language model alignment. ArXiv, abs/2304.07327, 2023.
Lambert, N., Tunstall, L., Rajani, N., and Thrush, T. Huggingface h4 stack exchange preference dataset, 2023. URI. htttps://haggingeee co co/da oo/dataaeess/tuggigggaeell stack-exchange-preferrees
Gulcehre, C., Paine, T. I.., Srinivasan, S., Konyushkova, K.,
Meng. Y., Xia, M., and Chen, D. Simpo: Simple preference optimization with a reference-free reward. Neurips, 2024.
Weerts, L., Sharma, A., Siddhant, A., Ahern, A., Wang, M., Gu, C., Macherey, W., Doucet, A., Firat, O., and de Freitas, N. Reinforced self-training (rest) for language modeling, arXiv, 2023.
Pang, R. Y., Padmakumar, V., Sellam, T., Parikh, A., and He, H. Reward gaming in conditional text generation. In ACL. ACL, July 2023.
Hong, J., Lee, N., and Thorne, J. Orpo: Monolithic pref- erence optimization without reference model. EMNLP, 2024.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly rrwarr model. In Thirty- seventh Conference on Neural Information Processing Systems, 2023, URL https: http:::///ppeeevviiiee opwnrevierreeeeeeeeeeeeeeemeeeee net/
Ja W., Huang, J.-t., Wang, W., He, Z., Liang, T., Wang. X., Shi, S., and Tu, Z. ParroT: Translating during chat using large language models tuned with human transla- tion and feedback. In Findings of the Association for Computational Linguistics: EMNLP, 2023.
Ramos, M. M., Almeida, T., Vareta, D., Azevedo, F., Agrawal, S., Fernandes, P., and Martins, A. F. T. Fine- grained reward optimization for machine translation using error severity mappings. ArXiv, 2024.
Juraska, J., Finkelstein, M., Deutsch, D., Siddhant, A., Mirzazadeh, M., and Freitag, M. MetricX-23: The Google submission to the WMT 2023 metrics shared task. In Koehn, P., Haddow, B., Koemi, T., and Monz, C. (eds.), Proceedings of the Eighth Conference on Machine Translation, Singapore, December 2023. Association for Computational Linguisties.
Rei, R., Stewart, C.,, Farinha, A. C., and Lavie, A. COMET: A neural framework for MT evaluation. In Webber, B., Cohn, T., He, Y., and Liu, Y. (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), Online, 2020. Association for Computational Linguisties.
Koemi, T., Bawden, R., Bojar, O., Dvorkovich, A., Fed- ermann, C., Fishel, M., Gowda, T., Graham, Y., Grund- kiewicz, R., Haddow, B., Knowles, R., Koehn, P., Monz, C., Morishita, M., Nagata, M., Nakazawa, T., Novák, M., Popel, M., and Popović, M. Findings of the 2022 conference on machine translation (WMT22). In Proceed- ings of the Seventh Conference on Muchine Translation (WMT), pp. 1-45, Abu Dhabi, United Arab Emirates (Hy- brid), December 2022. Association for Computational org/
Rei, R., C. de Souza, J. G., Alves, D., Zerva, C., Farinha, A. C., Glushkova, T., Lavie, A., Coheur, L., and Mar- tins, A. F. T. COMET-22: Unbabel-IST 2022 submis- sion for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pp. 578-585, Abu Dhabi, United Arab Emirates (Hybrid), December 2022. Association for Computational Linguis- tics. URL https:t/p///...... /ttp://aalaathhtc ogy- org/ 2022. wmt 1.52.
Linguistics. URL https: htttt:::/aclaattoooIe 2022. wmt-1.1.
Kocmi, T., Avramidis, E., Bawden, R., Bojar, O., Dvorkovich, A., Federmann, C., Fishel, M., Freitag, M., Gowda, T., Grundkiewicz, R., et al. Findings of the 2023 conference on machine translation (wmn233): Llms are here but not quite there yet. In Proceedings of the Eighth Conference on Machine Translation, pp. 1-42, 2023.
Sellam, T., Das, D., and Parikh, A. P. Bleurt: Learn ing robust metrics for text generation. arXiv preprint vrXiv:2004.04696 2020.
Team, G. Gemini 1.5: Unlocking multimodal understanding
across millions of tokens of context. ArXiv, 2024.
Kopf, A., Kilcher, Y., von Rutte, D., Anagnostidis, S., Tam, Z. R., Stevens, K., Barhoum, A., Duc, N. M., Stanley, O., Nagyfi, R., Shahul, E., Suri, S., Glushkov, D., Dantu- luri, A., Maguire, A., Schuhmann, C., Nguyen, H., and
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, E.,, and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
10

---

Learning from others' mistakes: Finctuning machine translation models with span-level error annotations
Wu, Z.,, Hu, Y., Shi, W., Dziri, N., Suhr, A., Ammanabrolu, P. Smith, N. A., Ostendorf, M., and Hajishirzi, H. Fine- grained human feedback gives better rewards for language model training. NeurlPS, 2023.
Xu, H., Sharaf, A., Chen, Y., Tan, W. Shen, L., Durme, B. V., Murray, K., and Kim, Y. Contrastive preference optimization: Pushing the boundaries of I1m performance in machine translation. ICML, 2024.
Xu, H., Murray, K., Koehn, P., Hoang, D., Eriguchi, A., and Khayrallah, H. X-alma: Plug & play modules and adaptive rejection for quality translation at scale. ICLR, 2025.
Yang, K. and Klein, D. FUDGE: Controlled text generation with future discriminators. In Proceedings of the 2021 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 2021.
Zeng. J., Meng, F., Yin, Y., and Zhou, J. Tim: Teaching large language models to translate with comparison. In AAAI 2023.
Zhang, L. H., Ranganath, R., and Tafvizi, A. Towards minimal targeted updates of language models with tar- geted negative training. Transactions Machine Learn- ing Research, 2024. ISSN 2835-8856. URL https: // //openreview..o htte:/iwwwwneeeeeeeeeeeeeeeeeeee forrforum? 1d=1122yii11sss5

---

Learning from others' mistakes: Finctuning machine translation models with span-level error annotations
A. Additional Dataset & Model Statistics
While Table and Table present average quality scores for the system submissions and base model, here we present additional statistics for both.
Table Average length and percentage of error tokens En-De and Zh-En translation pairs. Standard deviations are shown parentheses.
Average length (tokens)
Percentage of error tokens
En De
38.8 (24.6)
11.0 (19.6)
Zh En
40.4 (27.0)
13.6 (19.2)
EN-DE
ZH-EN
0.7
0.6
0.5
0.3
0.1
0.0
Figure Proportion tokens within emror spans each output sequence.
EN-DE
ZH-EN
10a
10a
10a
109
50
100
150
200
250
50
100
150
200
250
Figure Number of tokens each output sequence.
Figure 5. Histograms of the proportion and number of errors in the training data. Left is En right Zh-En.
12

---

Learning from others' mistakes: Finetuning machine translation models with span-level error annotations
En-De
Zh-En
16
12
Base Model
Base Model
14
Submissions
Submissions
10
0.8
Density
Density
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
-25
-2.0
-15
-1.0
-0.5
0.0
0.5
10
15
-25
-2.0
-1.5
-1.0
-0.5 COMET
0.0
0.5
10
15
COMET
Figure Histogram COMET scores across the submissions and base model generations. Source inputs come from the training data.
Table 7. TWA versus baselines on third language pair, En-2Zh. Results in bold are significantly better than others in the same column.
Submissions References Metric-Xi
COMET1
BLEURT1
TWA
2.342
0.517
0.696
DPO
2.349
0.503
0.693
SFT
2.470
0.504
0.692
TWA
2.324
0.521
0.697
DPO
2.325
0.510
0.694
SFT
V
2,409
0.509
0.694
Filter SFT
V
2.373
0.513
0.693
B. Additional results
We test TWA versus baselines on an additional language pair, En below. We used WMT-'20 and '21 MQM data for finetuning En-+De and Zh-FEn, but these are the only language pairs with MQM data from those years. Instead, we use MQM from WMT- '22 for finetuning En-+Zh, then WMT-'2 for validation, and WM as the test set. As Section 5, we use the parallel WMT'23 training data for pretraining from scratch. All other experimental details match those of Section 5. and find that TWA significantly outperforms baseline methods. Table 7 showcases the results.
C. DPO hyperparameter sweeps
To ensure a fair comparison with baseline methods, we test many settings of DPO, varying the construction of the preference pairs and the method for scoring sequences to determined preferred vs. dispreferred summarizes the results. As the DPO loss seeks increase the probability of the preferred sequence relative to its probability under the original model and decrease the probability of the dispreferred sequence relative its probability under the original model, we first constructed pairs where the reference was always the preferred sequence in pair. As the dispreferred sequence, tested using the best submission (by MQM score), worst submission, all submissions and found that using the worst submission yielded the best results. However, the performance in all these settings paled in comparison to the setting where we constructed as many pairs of distinct score submissions possible, even without access to the reference data. Adding additional pairs using the reference data improved results further, so we chose this setting for constructing pairs. With this setting, we find that using the sum of the span-level MQM scores performs better than the mean MQM score when both references and all submissions are applied; given that sequence-level MQM scores are generally computed using
pair. We set /3 = 0.1. Table
13

---

Learning from others' mistakes: Finctuning machine translation models with span-level error annotations
the sum, we choose it over the mean.
Table 8. DPO En results (test set) for different configuration settings.
Setting
preferred
dispreferred
score
Metric-X
COMET 1
Default
reference
best submission
mean
6.259
0.083
Default
reference
worst submission
mean
5.540
0.174
Default
reference
all submissions
mean
5.575
0.157
Default all submissions
all submissions
mean
3.753
0.455
Default reference, all submissions
all submissions
mean
3.739
0,442
Default all submissions
all submissions
sum
3.792
0,455
Default reference, all submissions
all submissions
sum
3.564
0.442
D. Fine-grained Annotator Model
Here, we consider the endeavor developing model to output fine-grained annotations of sequence, We consider two approaches, direct finetuning and in-context learning (Brown, 2020) with Gemini Pro-1.5 (Team, 2024). For the former, use the WMT 20- 20- MQM datasets. For the latter, as in-context examples for annotating a given output translation for that same source. We utilize the following prompt preceding the ICL examples: "You are an annotator for the quality of machine translation. Your task is to identify errors and assess the quality of the translation".. We test both approaches on the WM test set and find that the latter (ICL) yields better results than the former (direct finetuning). Thus, we use the latter to annotate our base model generated translations.
the MQM submissions data matching given source input
We then ran MQM human evaluation collect the ground-trutt -tuth annotations for these same translations and report the character-level Fl meta-evaluation metric (Blain et al., 2023). In comparison to ground-truth annotations from the human MQM evaluation, ICL with Gemini achieves character-level Fl meta-evaluation metric (Blain et al., 2023) of only 19, 14. These results highlight the loss in annotation accuracy incurred when utilizing model-based annotation of online data (required for reinforcement learning approaches). See 9 for the performance of our fine-grained annotator model on the 20''200'22 21 test sets.
Table 9. Character-level F1, precision, and recall of our fine-grained annotator model when annotating outputs from our base translation model, computed with respect to human MQM annotations collected for the same translations.
Character-Level FI Precision Recall
19.14
14.34
28.76
E. Sample translations
Below, present candidate translations from the Zh->En experiment. Examples were chosen emphasize differences between methods while representing diversity of translation lengths. Concretely, we subsetted examples where the edit distance between any pair of methods was at least 15. Then, we stratified examples into equal equal-sized bins based on the length of the TWA translation and chose one example from each bin. Within each strata, the example was chosen qualitatively based on ease of understanding and diversity in content.
We additionally asked Gemini 2.5 to summarize the differences between all the outputs rather than just the few we pasted into the appendix, and this a summary:
Literalness Fluency: TWA, SFT, and Base tend towards more literal translations, preserving source structure. DPO aims for higher fluency, sometimes sacrificing strict fidelity or introducing enrors. Reliability: TWA and SFT are the most reliable, rarely producing complete nonsense. Base and DPO are significantly less reliable, with instances of hallucination or total failure.
Technical Data Handling: Translating highly specific technical standards is challenging for all models, but TWA and SFT maintain better accuracy and detail compared Base and DPO. TWA demonstrates an edge in
14

---

Learning from others' mistakes: Finctuning machine translation models with span-level error annotations
Table 10. Example comparing TWA with SFT and DPO.
Base I was tired of waiting for the phone to ring, and when saw the order was still in the state of delivery, I immediately contacted the customer service to request that the order be cancelled.
Reference
So I simply waited patiently, when I checked my phone at midday, the order was still in the status and had not been delivered, I immediately contacted the customer service and requested that the order be canceled as quickly as possible, customer service responded that the operation was well.
TWA Well, waited patiently, and when looked at the phone at noon, the order was still in the state of delivery, so I immediately contacted the customer service to request that the order cancelled as soon as possible, and the customer service responded that was OK.
SFT Well, I waited patiently, and when looked at the phone noon, the order was still in the state of delivery, and immediately contacted the customer service to ask for the order to be cancelled quickly, and the customer service responded that it was OK.
DPO
I was nervous about the timing of the delivery, but when saw the order was still in the order and the order had not been shipped, immediately contacted the customer service to request that the order be cancelled.
Table 11. Example comparing TWA with SFT and DPO.
Base
Opioid shortages caused by new coronary pneumonia restraints may lead users to substitute more readily available substances such as alcohol or benzodiazepines mix opioids with synthetic drugs.
Reference
5 Users may substitute more readily available substances, such as alcohol or benzodiazepines, or combine opioids with synthetic narcotics as a result of opioid shortages brought on the COVID-19 regulations.
TWA The shortage of opioids caused by the new coronavirus restrictions may cause users to replace them with more readily available substances such as alcohol or benzodiazepines, or mix opioids with synthetic drugs.
SFT
The shortage of opioids caused by the coronary pneumonia restriction measures may lead users replace them with more readily available substances such as alcohol or benzodiazepines, or mix opioids with synthetic drugs.
DPO Opioid shortages caused by new coronavirus restrictions may lead users to be replaced with more readily available substances such as alcohol or benzodiazepines or to mix opioids with synthetic drugs.
consistently rendering the most intricate technical specifications with higher fidelity compared to SFT, which occasionally displays deviations in complex cases.
F. Comparing TWA on En when ignoring off-trajectory tokens vs. not
To understand the effect of ignoring off trajectory tokens the improved quality of En->e, compare translations from the model trained with TWA (which ignores off-traaectory tokens) with that of an ablation which includes off trajectory tokens in the loss. Tables 15 to 17 for example translations, as well as explanation by Gemini-2.5-PP for which setting is preferred and why.
We additionally asked Gemini-2.5- Pro to compare all the test translations of both variants, by passing the csv of all En->De translations between ignoring and not ignoring and asking you describe the difference. Gemini-2.5-PP states that the translations from ignoring off-trajectory tokens are "slightly more fluent, natural-sounding, and sometimes grammatically refined German translations," compared to translations that are sometimes "literal or slightly more awkward in its phrasing or word choice."
15

---

Learning from others' mistakes: Finetuning machine translation models with span-level error annotations
Table 12. Example comparing TWA with SFT DPO.
Base
The ears are folded into rectangular shape, so that the ears are slightly larger than the head,
Reference
Ear covers come with the earphones, use earbuds and bottom text if uncomfortable.
TWA
The headphones themselves equipped with medium-sized sails, if you feel discomfort to ask the text to blend in small though the outside.
SFT
The headphones themselves are equipped with medium-sized ear sails, they feel unwell to ask the text to merge small though outside the machine.
DPO
The headphones themselves are fitted with medium cedium--izee sails, which can be used if the wearer feels unwell and needs to be externally fitted.
Table 13. Example comparing TWA with SFT and DPO.
Base doesn't 't matter what the smell is.
Reference
makes no difference if the seller's company has already collected the money and does not follow up.
TWA It doesn't 't matter the seller company has also received money anyway and doesn't 't follow up.
SFT It doesn't matter if the seller company has also received money and doesn't 't follow up anymore.
DPO It doesn't 't matter if the seller has taken the money or not followed up.
Table 14. Example comparing TWA with SFT and DPO.
Base
Press about second boot
Reference
To turn on, press and hold for roughly 1 second
TWA Press long about second to start the machine
SFT Start up in about second
DPO Press on for about seconds to boot
Table 15, Example comparing TWA (which ignores off-trajectory tokens) with an ablation which includes off trajectory tokens in the loss.
Source If we did, we'd see these mass gun shootings go down.
Ablation
Wenn wir das täten, würden wir sehen, wie diese MassenschieeBereicn untergehen.
TWA(ignore)
Wenn wir das täten, würden wir sehen, wie diese Massenschießereein abflauen.
LLM preference
The second translation (TWA(ignore)) is better because "abflauen" more accurately conveys the idea of decline or reduction frequency, which aligns with the intended meaning of "go down" in this context. The first translation (Ablation) uses "untergehen,"" which typically means "to perish" "to go under," making it sound like the shootings themselves are disappearing in more absolute even unnatural way rather than decreasing in occurrence.
16

---

Learning from others' mistakes: Finetuning machine translation models with span-level error annotations
Table 16. Example comparing TWA (which ignores off-trajectory tokens) with an ablation which includes off trajectory tokens in the loss.
Source
We are still looking for ways to improve our services and this will be noted as feedback to one of our valued customer.
Ablation
Wir suchen immer noch nach Möglichkeiten, unsere Dienstleistungen zu verbessern, und dies wird als Feedback an einen unserer geschätzten Kunden gemerkt.
TWA(igoore)
Wir suchen immer noch nach Möglichkeiten, unsere Dienstleistungen verbessern, und dies wird als Feedback an einen unserer geschätzten Kunden gewertet.
LLM preference
The second translation (TWAAignore)) better because "gewertet" (evaluated/ccossidered) more naturally conveys the idea that the feedback being taken into account or soknowledged. In contrast, "gemerkt" (noted/remembere)) sounds more like someone is simply taking mental note, which is less formal and may not fully capture the intended meaning of acting on the feedback.
Table 17, Example comparing TWA (which ignores off trajcrrajy tokens) with an ablation which includes off trajectory tokens in the loss.
Source
I'll share couple of steps to perform into your eReader, okay?
Ablation
Ich werde ein paar Schritte in Ihrem eReader ausführen, okay?
TWA(ignore)
Ich gebe Ihnen ein paar Schritte mit, die Sie in Ihrem eReader ausführen können, okay?
LLM preference
The second translation (TWA(ignore)) is better because "Ich gebe Ihnen ein paar Schritte mit, die Sie in Ihrem eReader ausführen können" correctly conveys that the speaker is providing steps for the listener to follow. The first translation (Ablation) "Ich werde ein paar Schritte in Ihrem eReader ausführen" incorrectly implies that the speaker themselves will perform the steps directly on the listener's eReader, which changes the meaning of the sentence.
17


Reviews:
[{"review_id": "8F2ue8hpJS", "reviewer": "Reviewer_gUKj", "summary": "This paper develops a simple finetuning algorithm, called Training with Annotations (TWA), to directly train machine translation models on this annotated data. TWA utilizes targeted span-level error information while also flexibly learning what to penalize within a span. Moreover, TWA considers the overall trajectory of a sequence when deciding which non-error spans to utilize as positive signals. Experiments on English-German and Chinese-English machine translation show that TWA outperforms baselines such as supervised finetuning on sequences filtered for quality and Direct Preference Optimization on pairs constructed from the same data.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review! We appreciate your compliment about the novelty of the idea and the feedback that the paper is well-written. We agree with how you related our work to the broader literature as well.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Language models have advanced to the point where it is often difficult to improve them substantially via supervised finetuning on high-quality human-written examples alone; instead, recent efforts to improve language model or sequence-to-sequence model performance have largely relied on annotations of model generations, from preferences to per-sequence scores. Such data, coupled with techniques to learn from it have yielded impressive results for many top language models.\nMost efforts, however, consider only sequence-level labels, usually in the form of a scalar score assigned to the entire output. In contrast, this work investigates the potential of using fine-grained span-level annotations from offline datasets to enhance language model training. Unlike sequence-level annotations, span-level annotations provide information about specific segments within a sequence, offering more detailed information for model learning. Moreover, in many situations, collecting fine-grained information is similar effort to collecting sequence-level labels, making the former a practical form of data for improving model performance given a method that can take\nadvantage of the information.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "This paper is well-writen and the idea is novel. Specifically,  the development of the Training with Annotations (TWA) algorithm is a key contribution. TWA's ability to utilize targeted span-level error information and flexibly learn what to penalize within a span is innovative. This fine-grained control over the training process has the potential to lead to more accurate models. Additionally, the consideration of the overall sequence trajectory when using non - error spans as positive signals shows a comprehensive understanding of how sequences should be modeled.", "other_comments_or_suggestions": "No", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5yoGf8HIAb", "reviewer": "Reviewer_hzpg", "summary": "This paper explored a new approach to fine-tune machine translation models by utilizing fine-grained span-level annotations for further quality improvement. Previous work mostly focus on sequence-level annotations. while this work takes advantage of more fine-grained span-level annotations. The authors carefully designed the experimental settings and conducted preliminary experiments to explore the best setting in a weighted span-level unlikelihood loss.\n\nThe idea is straight forward. Experimental results on two machine translation tasks are technically sound, highlighting the effectiveness of the propose approach utilizing span-level annotations.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! Please see our response below:\n\n[sacreBLEU ...]:\n\nGood question. To compare the effectiveness of TWA in using targeted negative information we compare the BLEU score of the model after TWA on submissions and references versus the model after DPO on submissions and references. Please see the table below. TWA has a much higher BLEU score, indicating that its use of negative information is more precise.\n\n|                  | En→De | Zh→En |\n|------------------|:-----:|:-----:|\n|                  | BLEU  | BLEU  |\n| TWA_subs_and_refs | 34.24 | 25.30 |\n| DPO_subs_and_refs | 30.71 | 15.66 |\n\n[Ignoring off-trajectory tokens ...]:\n\nWe have pasted a few examples below of the decoded test translations between the model trained to ignore off-trajectory tokens versus the model trained with including off-trajectory tokens. We have also added these to the paper. We additionally asked Gemini Pro 2.5 Pro to compare the two; when we pass the csv of all En->De translations between ignoring and not ignoring, it states that the translations from ignoring off-trajectory tokens are “slightly more fluent, natural-sounding, and sometimes grammatically refined German translations,” compared to translations that are sometimes “literal or slightly more awkward in its phrasing or word choice.”\n\nA)\n\nSource: If we did, we'd see these mass gun shootings go down.\nTWA: Wenn wir das täten, würden wir sehen, wie diese Massenschießereien untergehen.\nTWA(ignore): Wenn wir das täten, würden wir sehen, wie diese Massenschießereien abflauen.\nLLM preference:\nThe second translation (TWA(ignore)) is better because \"abflauen\" more accurately conveys the idea of a decline or reduction in frequency, which aligns with the intended meaning of \"go down\" in this context. The first translation (TWA) uses \"untergehen,\" which typically means \"to perish\" or \"to go under,\" making it sound like the shootings themselves are disappearing in a more absolute or even unnatural way rather than decreasing in occurrence.\n\nB)\n\nSource: We are still looking for ways to improve our services and this will be noted as a feedback to one of our valued customer.\nTWA: Wir suchen immer noch nach Möglichkeiten, unsere Dienstleistungen zu verbessern, und dies wird als Feedback an einen unserer geschätzten Kunden gemerkt.\nTWA(ignore): Wir suchen immer noch nach Möglichkeiten, unsere Dienstleistungen zu verbessern, und dies wird als Feedback an einen unserer geschätzten Kunden gewertet.\nLLM preference: The second translation (TWA(ignore)) is better because \"gewertet\" (evaluated/considered) more naturally conveys the idea that the feedback is being taken into account or acknowledged. In contrast, \"gemerkt\" (noted/remembered) sounds more like someone is simply taking a mental note, which is less formal and may not fully capture the intended meaning of acting on the feedback.\n\nC) \n\nSource:I'll share a couple of steps to perform into your eReader, okay?\nTWA:Ich werde ein paar Schritte in Ihrem eReader ausführen, okay?\nTWA(ignore): Ich gebe Ihnen ein paar Schritte mit, die Sie in Ihrem eReader ausführen können, okay?\nLLM preference: The second translation (TWA(ignore)) is better because \"Ich gebe Ihnen ein paar Schritte mit, die Sie in Ihrem eReader ausführen können\" correctly conveys that the speaker is providing steps for the listener to follow. The first translation (TWA) \"Ich werde ein paar Schritte in Ihrem eReader ausführen\" incorrectly implies that the speaker themselves will perform the steps directly on the listener’s eReader, which changes the meaning of the sentence.\n\n[Regarding D Sample translations ...]:\n\nWe have added the baseline model’s translations to each of the outputs in appendix D.  We additionally asked Gemini Pro 2.5 to summarize the differences between all the outputs rather than just the few we pasted into the appendix, and this is a summary:\n\nLiteralness vs. Fluency: There's a clear spectrum. TWA, SFT, and Base tend towards more literal translations, preserving source structure. DPO aims for higher fluency, sometimes sacrificing strict fidelity or introducing errors.\nReliability: TWA and SFT are the most reliable, rarely producing complete nonsense. Base and DPO are significantly less reliable, with instances of hallucination or total failure.\nTechnical Data Handling: Translating highly specific technical standards is challenging for all models, but TWA and SFT maintain better accuracy and detail compared to Base and DPO. TWA demonstrates an edge in consistently rendering the most intricate technical specifications with higher fidelity compared to SFT, which occasionally displays deviations in complex cases.\n\n[Additional related work]:\n\nThanks for these additional works. We have added them to the related work, presenting them as alternatives to DPO which also contrast pairs of responses.\n\nThank you again for your review. If you find our response satisfactory, would you be willing to consider raising your score?", "claims_and_evidence": "Figure 2 shows how their proposed approach works for the span-level annotations. Results also highlight the effectiveness of the proposed span-level approach against the conventional sequence-level approach (DPO).\n\nThe authors conducted an experiment in two different machine translation tasks for English-to-German and Chinese-to-English directions. For both directions, their approach shows consistent improvement. With results for a few more different directions, the paper could be more experimentally convincing. It would be also interesting to apply this approach in domain adaptation scenario requiring specific terminology selection. This would be another area that the proposed span-level approach might work well. To be clear, I am not asking for additional experiments for this direction.", "methods_and_evaluation_criteria": "The carefully designed experiments and its result are technically sound. The systems are evaluated in different metrics, and the proposed approach shows the best performance in most cases.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Have you ever checked the n-gram match metrics such as sacreBLEU [1]? Since those COMET scores reflect well semantics rather than n-gram match, sacreBLEu score is another suitable metrics to assess effectiveness of your span-level approach. It would capture the span-level correction if it works as expected.\n\nCan you give a few examples when ignoring off-trajectory tokens? When you take a look at those examples, any patterns leading to those semantic evaluation metrics improvement (e.g., fluency)? \n\n[1] Post. \"A Call for Clarity in Reporting {BLEU} Scores\" In Proc of WMT 2018.", "relation_to_broader_scientific_literature": "The proposed approach of the span-level error annotations could be applicable to other text generation tasks, with a good potential impact, considering that most research have been so far well studied at sequence level. The analyses could be improved by reporting any other trends (detected via n-gram match) or highlighting the span-level corrected parts, comparing to the baseline's outputs.", "essential_references_not_discussed": "In addition to DPO, you could also cite some more relevant work:\n\n- Meng et al., \"SimPO: Simple Preference Optimization with a Reference-Free Reward\". In Proc of NeurIPS 2024.\n- Xu et al., \"Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation\" In Proc of ICML 2024.\n- Hong et al., \"ORPO: Monolithic Preference Optimization without Reference Model\" In Proc of EMNLP 2024.\n- Xu et al., \"X-ALMA: Plug & Play Modules and Adaptive Rejection for Quality Translation at Scale\". In Proc of ICLR 2025.", "other_strengths_and_weaknesses": "Please see the comments in the sections above.", "other_comments_or_suggestions": "Please see the comments in the sections above.", "questions_for_authors": "Please see the comments/questions in the sections above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "8GghZj3gwE", "reviewer": "Reviewer_wBPs", "summary": "This paper focuses on improving machine translation models by leveraging span-level error annotations. It proposes a new algorithm called Training with Annotations (TWA). The core idea of the TWA is applying a weighted span-level unlikelihood loss to error spans to encourage the model to learn which tokens to penalize. In experiments, authors pre-trained the base model on WMT’23 data and fine-tuned the base model with MQM data from WMT’20 and WMT’21. Experimental results on English-German and Chinese-English machine translation show that TWA outperforms baseline methods like Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO). Ablation studies verify the effectiveness of each component of TWA, and the unlikelihood loss used in TWA is proven to be better than the negative likelihood loss.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Clarifying experimental details and questions:\n1. Submissions are the machine translations from models entered into the WMT competition.\n2. We pre-trained two base models using a different bilingual dataset for each.\n3. The other MT systems are the specific models entered into the WMT competition (listed in [1]).  The difference between SFT and TWA on these models’ outputs is the difference between pure distillation of offline data and smarter distillation that takes advantage of fine grained annotations. We also consider fine-tuning on annotations of online data in Section 6.5, showing that we can improve model quality this way as well.\n4. We ran a human evaluation, following [2].\n5. We chose to use the baseline models we did to have full control over the entire pipeline, reason about all the data seen by the model, and allow for full transparency (i.e. using standard publicly available WMT data). We additionally compared TWA with baseline methods on a production model and found on internal evaluation sets that it indeed outperformed baselines.\n\nAlso, thanks for the additional references! We've added these into the paper under other works that have considered the use of MQM data.\n\nThank you for your review. Please let us know if you have any additional questions; otherwise, would you be willing to consider raising your score?\n\n[1] https://aclanthology.org/2021.wmt-1.1.pdf\n\n[2] https://arxiv.org/abs/2104.14478", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Some experimental details need to be further clarified：\n\n1) “ .... with around 28,000 and 31,000 submission outputs for En→De and Zh→En, respectively\", What are submissions? \n\n2）“We pretrain En→De and Zh→En models using the parallel WMT’23 training data (Kocmi et al., 2023), which consists of 296 million sentence-level examples.” I am not sure whether the two directions are to use one multilingual dataset (296 million samples）to train one model or two bilingual datasets to train two models.\n\n3) “SFT on the MQM annotated data is analogous to distilling the outputs of other MT systems, without taking into account the annotations.” What are the other MT systems? A fair setting would be to perform data distillation with the base model (self-training).\n\n4) “we obtain MQM annotations of the base model’s translations and run TWA with this annotated data. ” How to obtain the  MQM annotations？", "relation_to_broader_scientific_literature": "As described in Section 7, the proposed TWA can be applied  to large language models in general.", "essential_references_not_discussed": "Utilizing MQM data in Related Work: \n\n1) ParroT: Translating during Chat using Large Language Models tuned with Human Translation and Feedback, Findings of the Association for Computational Linguistics: EMNLP 2023 \n\n2) Teaching Large Language Models to Translate with Comparison, AAAI 2024", "other_strengths_and_weaknesses": "The paper is well-written and easy to follow. The fine-grained annotation method proposed in the paper is worth further discussion. The author's design of the TWA is very novel (e.g., non-error tokens following an error span are ignored as they are off-trajectory.), which is very similar to the process supervision method. These methods may also have some good effects on the logical reasoning of large language models.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "A. Experiment\n\n1) “ .... with around 28,000 and 31,000 submission outputs for En→De and Zh→En, respectively\", What are submissions? \n\n2) “We pretrain En→De and Zh→En models using the parallel WMT’23 training data (Kocmi et al., 2023), which consists of 296 million sentence-level examples.” I am not sure whether the two directions are to use one multilingual dataset (296 million samples）to train one model or two bilingual datasets to train two models.\n\n3) “SFT on the MQM annotated data is analogous to distilling the outputs of other MT systems, without taking into account the annotations.” What are the other MT systems? A fair setting would be to perform data distillation with the base model (self-training).\n\n4) “we obtain MQM annotations of the base model’s translations and run TWA with this annotated data. ” How to obtain the  MQM annotations？\n\n5) Why not use large model-based translation as a baseline system?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TMFjoVszMD", "reviewer": "Reviewer_7Lim", "summary": "This paper proposes training with annotations (TWA), a method for finetuning machine translation models using existing span-level error annotations. Unlike traditional approaches that rely on sequence-level feedback, TWA directly leverages fine-grained annotations for model training. The authors evaluate TWA on two language pairs from the WMT’23 test sets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review. Responding to your concerns below:\n\n[Lack of human evaluation]:\n\nWhile we agree that human evaluation would strengthen the results, we do show improvements over multiple automated quality metrics, including a held out metric that was not used during any part of model selection, and we also provide sample translations in the appendix.\n\n[Evaluation. The MQM data used to train the MT models is also used to train Metric-X. In addition, the gains in BLEURT are small, making it unclear whether the improvements are meaningful. Table 3 shows TWA and other baselines within the same quality cluster in some cases.]:\n\nWe believe it to be important to include Metric-X as a quality metric specifically because it is trained to be sensitive to the information found in MQM data, which is not necessarily the case for other quality metrics. We note that the MQM data has not been used in either COMET or BLEURT. BLEURT gains are likely smaller due to  the fact that it is a completely held out quality metric (i.e., we do not consider it at all during model checkpoint selection), and the fact that TWA remains in the top cluster consistently, oftentimes alone, is especially promising in the context.  \n\n[Baseline models appear weak. Is there any reason to start from these baseline models?]:\n\nWe chose to use the baseline models we did to have full control over the entire pipeline, reason about all the data seen by the model, and allow for full transparency. We additionally compared TWA with baseline methods on a production model and found on internal evaluation sets that it indeed outperformed baselines.\n\n[The authors use an encoder-decoder architecture despite the evidence that decoder-only models perform better. ]:\n\nWe use an encoder-decoder architecture commonly used for MT given its superiority with respect to prediction latency.  Moreover, moderate-sized decoder only models often under perform relative to encoder-decoder MT models [1, 2].\n\n[The authors focus only on two (high-resource) language pairs, even though MQM annotations exist for others.]:\n\nWe specifically run the language pairs we do given MQM annotations already exist for these pairs. In WMT 20-21,  only three language pairs contain both MQM annotations and test sets.  Our current experiments use two of these three pairs, and we have added the results on the third (en->zh) below. Here, TWA once  again significantly out performs other baselines.\n\n\n|                  | Metric-x  | Comet   | Bleurt  |\n|------------------|-----------|---------|---------|\n| TWA              | **2.342** | **0.517**   | **0.696** |\n| DPO              | **2.349**     | 0.503 | 0.693   |\n| SFT              | 2.470     | 0.504   | 0.692   |\n| ---------------- | --------- | ------- | ------- |\n| TWA_with_refs    | **2.324**     | **0.521**   | **0.697**   |\n| DPO_with_refs    | **2.325** | 0.510   | 0.694   |\n| SFT_with_refs    | 2.409     | 0.509   | 0.694   |\n| SFT_filter       | 2.373     | 0.513 | 0.693   |\n\n\n[Are you planning to release the source code?]:\n\nWe are in the process of obtaining approval to release the code for the camera-ready version of the paper.\n\n[Missing related work]:\n\nThanks for the feedback. We have added a section in the related work about work on improving MT through reranking at inference time (e.g., using quality estimation metrics) or finetuning, highlighting the reviewer’s point that the main novelty is using annotations directly without training new quality estimation models.  We have also added the additional references.\n\nThank you for your review. In light of our response and additional results, we hope you'll consider raising your score.\n\n[1] https://arxiv.org/pdf/2401.08417\n\n[2] https://arxiv.org/abs/2202.00528", "claims_and_evidence": "The claims are generally sound, and the authors provide some empirical evidence to support them. However, I have concerns about their design choices and evaluation setup (detailed below). Overall, while the claims are reasonable, the experimental evidence is not entirely convincing and needs to be improved to show that their method really works.", "methods_and_evaluation_criteria": "The authors report results on WMT’23 (En-De and Zh-En), a recent test set commonly used in the MT literature. They could have used WMT’24, but I don’t think this is problematic.\nMy main concern is the lack of human evaluation. The paper relies entirely on automatic evaluation metrics (Metric-X, COMET, and BLEURT). In particular, the authors apply TWA (their method) using MQM data from WMT’20-21. This data is used to train MetricX-23, which they also use for evaluation (this is acknowledged in Section 5.4).", "theoretical_claims": "NA.", "experimental_designs_or_analyses": "I checked the experiments carefully and appreciated the statistical significance tests. I’ll summarize my main concerns below:\n\n(1) Evaluation. See my comment above about Metric-X. In addition, the gains in BLEURT are small, making it unclear whether the improvements are meaningful. Table 3 shows TWA and other baselines within the same quality cluster in some cases.\n\n(2) The baselines appear too weak. Tables 1 and 2 show their performance against the systems submitted to WMT’20-21. The baselines are worse than the average, and the authors do not even consider models submitted to the shared tasks that followed (WMT’22-24). Is there any reason to start from these baseline models?\n\n(3) The authors use an encoder-decoder architecture despite the evidence that decoder-only models perform better. Can you please clarify this decision? Please check Kocmi et al., 2024 (Findings of the WMT24 General Machine Translation Shared Task: The LLM Era Is Here but MT Is Not Solved Yet).\n\n(4) The authors focus only on two (high-resource) language pairs, even though MQM annotations exist for others. Was there a reason to test only two?", "relation_to_broader_scientific_literature": "The paper does a good job covering existing MT literature but lacks a discussion of related work that also uses feedback to improve MT (this idea is not new at all). For instance, the paper does not properly discuss a large body of work on improving MT through reranking at inference time (e.g., using quality estimation metrics) or finetuning. The main novelty here is applying these ideas specifically to MT span annotations (and, most importantly, using these annotations directly without the need for training new “quality estimation” models). Considering the broader impact of this work, it’s unclear how to apply these ideas outside of MT, where human annotations are scarce (but I don’t think this is a problem this paper needs to solve).", "essential_references_not_discussed": "- Welleck et al. 2019 (Neural Text Generation With Unlikelihood Training), which introduces the unlikelihood loss mentioned and used in this paper.\n- Ramos et al., 2024 (Fine-Grained Reward Optimization for Machine Translation using Error Severity Mappings). In the introduction, the authors mention that “While MQM data has previously been used to develop auxiliary reward or metrics models (Juraska et al., 2023; Rei et al., 2022), it has not been directly employed for training machine translation (MT) models.” There’s work using MQM annotations for training MT models, though. For instance, Ramos et al. (2024) use xCOMET (trained on MQM data) to provide fine-grained feedback during training. I believe this should be mentioned in the Related Work section.", "other_strengths_and_weaknesses": "Strengths: The paper provides a simple approach for leveraging span-level annotations to improve machine translation. This is well-motivated in the introduction.\n\nWeaknesses: See my comments above about weak baselines and evaluation.", "other_comments_or_suggestions": "Minor comments:\n- I think Appendix D is not mentioned in the main paper.\n- It would be good to add numbers to the equations.\n\n**Update after the rebuttal**: I updated my score to 3.", "questions_for_authors": "See numbers (1)-(5) above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper explored a new approach for finetuneing language models by utilizing fine-grained span-level annotations, as opposed to focusing on sequence-level annotations. While the approach is relatively straightforward, it is quite effective and shows improvements on two machine translation tasks and consistent improvements. While the reviewers are generally satisfied with the paper, they note several weaknesses, such as evaluating on only two language pairs, concerns with baselines, and limitations in the analysis (e.g., we would like to see the baseline's output before applying TWA, SFT and DPO). Finally, while the motivation of the work applies to language models – as highlighted in the introduction – but the work is only evaluated on MT. 

Given the reviewers’ overall positive sentiment and that no major concerns remain after discussions, I recommend accepting the paper, and I encourage the authors to add a third language pair in the final version of the paper (en->zh as included in the rebuttal).
Decision: Accept (poster)