MAD for Robust Reinforcement Learning in Machine Translation
Authors: Domenic Donato
Lei Yu
Wang Ling
Chris Dyer
Venue: Submitted to ICLR 2023 (2023)
Keywords: Machine Translation
Reinforcement Learning

Abstract: We introduce a new distributed policy gradient algorithm and show that it outperforms existing reward-aware training procedures such as REINFORCE, minimum risk training (MRT) and proximal policy optimization (PPO) in terms of training stability and generalization performance when optimizing machine translation models. Our algorithm, which we call MAD (on account of using the mean absolute deviation in the importance weighting calculation), has distributed data generators sampling multiple candidates per source sentence on worker nodes, while a central learner updates the policy. MAD depends crucially on two variance reduction strategies: (1) a conditional reward normalization method that ensures each source sentence has both positive and negative reward translation examples and (2) a new robust importance weighting scheme that acts as a conditional entropy regularizer. Experiments on a variety of translation tasks show that policies learned using the MAD algorithm perform very well when using both greedy decoding and beam search, and that the learned policies are sensitive to the specific reward used during training.

Under review as a conference paper at ICLR 2023
MAD FOR ROBUST REINFORCEMENT LEARNING IN MACHINE TRANSLATION
Anonymous authors Paper under double-blind review
ABSTRACT
We introduce a new distributed policy gradient algorithm and show that it outper- forms existing reward-aware training procedures such as REINFORCE, minimum risk training (MRT) and proximal policy optimization (PPO) in terms of training stability and generalization performance when optimizing machine translation mod- els. Our algorithm, which call MAD (on account of using the mean absolute deviation in the importance weighting calculation), has distributed data generators sampling multiple candidates per source sentence on worker nodes, while central leamner updates the policy. MAD depends crucially on two variance reduction strategies: (1) conditional reward normalization method that ensures each source sentence has both positive and negative reward translation examples and (2) a new robust importance weighting scheme that acts conditional entropy regularizer. Experiments on a variety of translation tasks show that policies learned using the MAD algorithm perform very well when using both greedy decoding and beam search, and that the learned policies are sensitive to the specific reward used during training.
INTRODUCTION
There is increasing interest in fine-tuning conditional language models on the basis of feedback from lask-specific reward models or similarity functions that compare to human genurnatd reference outputs rather than relying exclusively on supervised learning (Stiennon al., 2020 Ziegler al.) 2019] Wu
et al. 2018 Paulus et al. 2018 Rennie et al. 2017 Ranzato et al. 2016). Maximising sequence
intuitive importance of "getting the full sequence right" in generation problems and the more conventional token-level cross entropy loss. Second, since a policy trained to maximize rewards is supervised with its own outputs-bothh good and bad ones-it mitigates issues arising from "exposure bias," in which a learned policy that has been trained only on correct examples has no experience recovering from errors and therefore performs poorly at test time (Ranzato al. (016). Third, feedback from (learned) rewards can be a cost-effective strategy for incorporating human preferences about how a system should behave (Stiennon et al. 2020 Christiano et al.) 2017).
level rewards has several advantages. First, it avoids the apparent conffict between
Unfortunately, fine-tuning policies for generating in complex output spaces, such as language, on the basis of sparse rewards is challenging. Estimating and debugging reliable auxiliary critic/value functions that are needed by many learning algorithms is challenging (Wu et al. [2018] Bahdanau et al., 2017 Nguyen et al. 2017), ) and commonly used average or batch-level reward baselines (Kreutzer et al. 2017) are poor variance reducers since they are independent of the input, and input difficulty is a strong determinant of reward magnitude.
In this paper, we propose a new distributed policy gradient algorithm ($2) for fine-tuning translation models that addresses these issues. The distributed setup lets us use modest computation to obtain simple and effective empirical reward baselines (Rennie et al. (017)) rather than using inappropriate batch-level statistics or relying on brittle auxiliary value models. Our proposed algorithm has two components designed to make learning from the reward signal more effective: we employ sampling and reward normalization strategy that encourage batches to contain a mix of both positive and negative rewards for each source sentence and, second, importance weighting strategy that encourages the algorithm to pay attention to trajectories that are slightly off the current policy. Thus, our algorithm learns from trajectories that are already relatively likely under the current policy

---

Under review as a conference paper at ICLR 2023
(meaning any updates to the policy will be relatively small), while also encouraging continued exploration throughout training by woighteig trajectories where the model very confident. This enables the algorithm to make large improvements in reward while taking small, conservative steps to change the behaviour and, also, helps slow down the rate of policy collapse letting the model obtain continued performance improvements over many training steps.
The policies learned using our MAD algorithm produce high-quality translations, even when using greedy decoding. In main experiments ($3), we use sentence BLEU the reward and find that the average improvement on held-out test sets over the initial cross entropy trained model is 2.0 BLEU. We also find that the models are less sensitive the beam search hyperparameters beam size and length normalization. This means we do not need to optimize the length normalization for each dataset and observe almost no performance degradation with larger beam sizes. We confirm that the algorithm learns different policies depending on the reward function used during optimization, with the resulting policies showing reward-specific improvements on held-out data. Finally, we carry out careful empirical analysis ($4) to better understand the impact and role the various components of the algorithm have on training dynamics and generalization performance.
ALGORITHM
Our algorithm consists of workers generating training trajectories and rewards, in parallel, from slightly out-of-date copy of the policy, Oou, and a central learner that updates the current policy, 0. At the beginning of training, both and Oug are initialized from a pretrained cross entropy model. The data generation algorithm is shown Alg. [T and the learner in Alg. 2 The learning algorithm has three core components: sampling from a range of temperatures ($2.1). conditional reward normalization on the basis of empirical means and variances ($2.2). and novel robust importance weighting strategy that focuses learning efforts on samples that are slightly off policy ($2.3) We discuss each of these components in turn.
Algorithm Asynchronous data generator
Algorithm Learning algorithm
I:
function GENERATE(N, A, T,ii Toss)
1:
function LEARN(S, OCE)
2: while True do
2:
Ø t OCE
3: Ooid 0 Get current global weights
3:
for E [1 do
4: (ar, Yrot) ~ Drroin
4:
x: y. T, v + DEQUEUE()
5:
for i E [1, N] do b Obtain Ya. and r
5:
p p log x: 0)
6:
A <- (Time // - 1)
6:
" <- exp(p
7:
T' Tmin (i 1) x A
7:
(Giin
2))
y. < SAMPLE(1; (x: Owu, T)
a logp(y u;; dropout (0) ac
9:
Q. + p(y, T.; mala)
9:
0 + 17 86
10:
r. + ( Ueer)
10:
end for
11:
end for
end function
12:
H4
D7 Σ,r
13:
(r:
2
14:
(EDIAN(Q)
15:
0,
MEDIAN( (lq (ry)) Median dev.
16:
for
e [1, do
17:
Ti (r: /4 /r)/orr
18:
U/ - expeelp11 Fg
19:
ENQUEUE( yr, 4:, Ti,
20:
end for
21:
end while
22:
end function
2.1
-ULTI-TEMPERATURE SAMPLING
To obtain suitably diverse candidates to learn from, is conventional to add a temperature hyperpa- rameter used to generate samples (Shen et al. 2016: Papini et al. 2020). We identify two problems with this. First, there is the practical matter of needing to select a temperature order to obtain good performance. Second, it is widely observed that policy gradient algorithms result in increasingly
2

---

Under review as a conference paper at ICLR 2023
peaked distributions training progresses (Kiegeland & Kreutzer 2021 : Choshen et al. 2020 Rennie et al. 2017). J. meaning that the amount of "exploration" being considered by the algorithm decreases over time. While some RL tasks are interested in maximising total accumulated returns over time (meaning that "exploitation" is important), we rather seek to learn a policy that is capable of behaving intelligently as possible in states that not encountered during training phase, and therefore, we seek an exploration-heavy sampling strategy. To avoid the difficulties with drifting entropy. we use simple approach of generating populations of samples at several different temperatures.
Concretely our data generation algorithm (lines 5 of Alg. I begins by sampling N translations for the current policy and source sentence æ. Each sample generated using a different temperature that is determined by finding equally spaced values between the interval [Tmin: m.x] Duplicate translations are removed. The process we use is:
V2- =
0,t); t EE T))
where
Tmin + 6: (1- (1- i (1......,
Timx
Tiin
6,
N
2.2 CONDITIONAL REWARD NORMALIZATION
Reward normalization is well known variance reduction technique for policy gradient methods, with the simplest version being to subtract constant from the reward (Williams) 1992).
(9912) Other methods have been proposed such subtracting model baseline reward (Weaver Tao 2001), the running average of the rewards seen during training (Kreutzer et al. (017). or z-scoring the rewards in a training batch (Stiennon et al. 2020). As demonstrated by Kiegeland & Kreutzer (202T). using the running reward average helps to reduce variance when REINFORCE is applied to translation.
While empirically effective, these baselines explain less reward variation than we might hope, We note that the difficulty of generating good translation is strongly dependent on the intrinsic difficulty of the source sentence, not merely the current policy Yohn-Yehiy et al., (022). Since the usual reward normalization methods do not take this dependency into account, this results in a bias toward giving difficult sentences negative rewards and easier sentences positive rewards, leading to less stable learning (see $4.2 and Appendix A.3)
We therefore use a standardization method that is conditioned on the source sentence. We take the set of translations for source sentence and obtain vector of rewards r, A yee)) Vi [1, 1> where Ay Youf is scalar valued reward indicating how well y communicates the contents of the reference translation Yrer. We then standardize these rewards by removing the mean and dividing by the standard deviation (lines
12 20 of Alg. [..
F (r: pr)/or Vi [1, (Dll), where
1 E(r+ - m,,
(r:-H,))
Hy
Ti o, V [aal
This ensures that every source sentence, irrespective of its translation difficulty, has examples with positive and negative rewards.
In contrast, the standard reward used for the PPO algorithm is the z-scored reward of the training batch, P. (r: - // Vi [1, [1]], where B is the training batch with randomly sampled (æ, y) examples and He and o, are respectively the mean and standard deviation of the rewards in B.
2.3 MAD IMPORTANCE WEIGITS
A key feature of our algorithm and the one that gives it its name is the use of new importance weighting scheme for deciding which sampled trajectories are most valuable for updating the policy
https: / /ttp//mmugg////////.//o orgg wtt.o////taaaeeeeeeeeeeeeeeeeeee wwweeeeeeeeeeeeeeeeeeeeeeeeeeeee
insppcee. html
For most of this work Aly. Yog) refers sentence BLEU, see 34.6 for alternatives and analysis.
3

---

Under review as a conference paper at ICLR 2023
RREIFFORCE: = (r b) log x))
Cpro minfu F, clip (, 1 1,,,1++, .
ply x))
LMRT Σ DEV.
E Ey ply' x))
MADD min(u v, 2) logp(y x))
Figure Sequence level losses used by the algorithms evaluated in this paper. a the source sentence and sampled translation. Here ( Yref) is the reward, which is sentence BLEU; F and r are different ways of normalizing said reward. The policy sequence probability is p(y x) is while and are importance weightings that depend on the this value along with the behaviour sequence probability, g æ). See for full explanation.
and which others should be downweighted. For trajectory i, our full importance sampling correction is
w; min Us 2}.
We explain u, and v; below and note that truncating importance weights is standard variance reduction strategy (Cortes et al. 2010 Schulman et al., 2017).
The first term in Eq. [ u4 is the standard importance weighting ratio (Precup et 2000 Pang
He 202T) to deal with the fact that in a distributed setup, data is generated from stale policy:
d4 exp(p. q:), where
P. = log x; 0), qi = log p(ys x;; Bold).
The second term in [q. 0% encourages continued exploration during training by having the learner pay attention samples that "relatively likely" under the current policy (as approximated by q, since we want the calculation to be carried out on the worker nodes, and p is only available on the central learner). We operationalize the notion of "relatively likely" as something that is near the mediar under of the elements in V2, using the exponentiated negative median absolute deviation (MAD; lines 14 20):
02 = exp( (gyl/ōg)
p. MEDIAN(q)
5, = MEDIAN((Q - 1,
Why do we want to concentrate learning on these relatively likely trajectories, rather than perhaps other trajectories that have even higher (or lower) rewards? First, down-weighting very unlikely samples is a strategy for reducing the gradient variance, since gradients associated with smaller changes the policy will require, advantageous since smaller gradient variance can lead to better generalization error Kreutzer et al. (2017). Second, Choshen et al. 2020) provide evidence that RL algorithms cause the policy to become more peaked during training which means that sampling diversity decreases and, thus, the amount exploration as training progresses reduced ($4.3). Since we seek an exploration-heat heavy RL algorithm, we focus learning effort on instances which are "in reach" of the current policy but not at the current policy's mode.
average, less significant updates to the network. This is
3 EXPERIMENTS
3.1 DATASETS
We run our model along with all the baselines on total of 9 dataset- languae direction translation tasks. See Appendix for the detailed description dataset, preprocessing, and tokenization.
"The median chosen hecause q can have long tail due degenerate translations.
4

---

Under review as a conference paper at ICLR 2023
3.2 TRAINING
For each task, we pretrain sequence-to-seqqenee transformer model (Vaswani et al., 2017). using word level cross entropy loss, until convergence. We refer to this model as the Cross Entropy (CE) model. All treatments for task are initialized using the same CE checkpoint, which is the checkpoint that had the highest development set BLEU. The treatments training/development sets for a total of 200K steps with early stopping if the average development BLEU is not improved for the last 20K steps. In the case of PPO and MAD, every 20 training steps the learning node saves a checkpoint which the workers and evaluators load asynchronously. We use global batch size of 256 training examples per step. See Appendix B for detailed hyperparameter configurations and Appendix D for implementation details.
trained using the same
3.3 COMPARED RL OBJECTIVES
Figure [ gives the primary RL objectives we compare. These are: the REINFORCE algo- rithm (Williams 1992) using a moving average baseline (Weaver & Tao 200T). a proximal policy optimization algorithm (Schulman et al., 2017 PPO), minimium risk training (Shen et al.] 2016 MRT), and our algorithm MAD. Since REINFORCE and MRT are on-policy algorithms, we optimize these objectives on single worker; the distributed infrastructure only used for MAD and PPO.
3.4 EVALUATION AND HYPERPARAMETERS
We use the sacreBLEU script Post 2018 with detokenized model hypothesis and the original references When running on the test set, we select the checkpoint that had the highest development set BLEU under greedy decoding.
The important hyperparameters of our algorithm are N, T mins and T max- During preliminary experi- ments, we swept over N and found that anything in the range of [8, 24] provided similar results. We fixed at N 12 for all the experiments reported in this work. We found that the optimal temperature range was dataset dependent ($ 4.5) so performed sweeps for all the models in order for fair comparison. For MAD, we swept over interval ranges, [Tmm; maxx] E {00.2, 0.6], [0.4, 0.8], [0.6, 1.0], [0.8, 1.2]}. while for the baselines found the single best temperature by sweeping
10 temperatures in the interval (0.2, 1.2] using increments of 0.1. See Table 3 in Appendix B for selected temperatures.
3.5
MAIN RESULTS
Table presents the performance of our algorithm compared to other objectives ($3.3) on 9 machine translation tasks. We compare these models on both greedy decoding and beam search. As can be seen, our reinforcement learning algorithm outperforms REINFORCE, PPO, and MRT by average of 1.0 BLEU when using greedy decoding. With beam search decoding, MAD outperforms the strong MRT baseline by 0.5 BLEU on average.
We observe that models trained using the MAD algorithm display a smaller gap in performance between greedy decoding and beam search decoding. The explanation for this is related obtaining more peaked and possibly smoother output distributions (Dabre & Fujita 2020). The benefit of this is that it reduces the need for per dataset beam search hyperparameter tuning. Specifically, Meister et al.) (2020) show that, when using beam search, it is necessary to tune a length normalization term, for each dataset order to prevent the model from generating sentences that are too short or too long. Since the MAD algorithm results in models that are less sensitive to are able to use uniform on all the datasets without sacrificing substantial performance.
We provide some additional remarks along with more test set results using larger beams and other decoding hyperparameters in Appendix A The test results in Table show that MAD's performance is fairly insensitive to these hyperparameters in marked contrast to CE (Meister et al. 2020
https: / / thub. www.c/oommmmmmm...mm ccom/jjoost/aacces..ll
See Table in Appendix for settings.
5

---

Under review as a conference paper at ICLR 2023
Table 1: Performance of different algorithms translation. We report sacreBLEU the held-out test set between the detokenized model hypothesis and original (not tokenized) references. Note that for IWSLT' 14 and NIST both the hypotheses and references were lower-cased keep comparable with prior works. is the average BLEU across all datasets for each method. MAD outperforms other methods by large margin when greedy decoding is used. The gap shrinks when beam search is used, but MAD still outperforms the next best method, MRT.
Greedy Decoding
NIST
IWSLT' 14
WMT' 14
WMT'20
Model
Zh-En
De-En
En-De
De-En
En-De
Zh-En
En-Zh
Ps-En
En-Ps
P
Cross Entropy
45.5
30.1
26.7
30.5
25.7
25.0
37.3
6.6
6.0
25.9
REINFORCE
45.5
30.2
26.7
29.3
25.7
25.0
37.1
7.2
5.9
25.8
PPO
45.9
31.0
27.7
30.8
25.6
24.8
37.3
7.3
7.5
26.4
MRT
45.2
30.8
26.7
31.3
25.8
26.4
38.8
8.4
7.5
26.8
MAD
47.4
32.1
28.0
31.8
27.1
27.5
39.5
8.8
7.9
27.8
Beam Search
Beams
Length Normalization (a)
1.0
NIST Zh-En
IWSLT' 14
WMT 14
WMT
Model
De-En
En-De
De-En
En-De
Zh-En En
En-Zh
Ps-En
En-Ps
Cross Entropy
47.6
31,4
27.9
31.8
26.4
26.4
37.5
7.0
6.0
26.9
REINFORCE
31.4
28.0
30.6
26.3
37.8
8.1
6.0
26.9
PPO
47.5
31.3
28.2
31.7
26.3
26.2
37.5
7.7
7.6
27.1
MRT
47.0
31.9
27.9
32.0
26.6
27.4
39.3
9.2
7.9
27.7
MAD
48.2
32.4
28.3
32.2
27.3
28.0
39.9
9.4
8.1
28.2
ANALYSIS AND ABLATION EXPERIMENTS
In this section investigate various aspects of the MAD algorithm. We variety of datasets as to not overfit our analysis to the quirks of specific one. We start off with full ablation study ($4.1) followed by closer inspection the effects of conditional reward normalization ($4.2) and the MAD importance weights ($4.3). After this, we move on to looking at training stability ($4.4) and MAD's temperature range hyperparameter sensitivity ($4.5). Finally, we discuss the impact of optimizing different rewards (4.6).
4.1 FULL ABLATION
In this section we run ablation study on the two innovations of the MAD algorithm, the conditional normalized rewards (.2) and the MAD importance weights ($2.3). For each experiment we use the distributed and sampling components of our algorithm. In experiments where conditional reward, Fe is not included, we use PPO's batch normalized reward, F. and in experiments where the MAD weights, Wh not included, baseline setup where both the conditional reward normalization and MAD weights are removed; this the pink line Figure Next we add back each of these components separately and then, finally, add both components back which results in our MAD algorithm. We can see from this ablation that the primary driver of our results the conditional reward normalization and that the MAD weights reduce variance enabling the model to continue learning for more training steps.
the standard importance weights, min f, 2). We start with
4.2 CONDITIONAL REWARD EFFECT
Our ablation study shows that using conditionally normalized rewards is the most important part of our algorithm. We hypothesized that due to the heterogeneous nature source sentence translation difficulty that unconditional normalization schemes would systemically assign negative rewards to difficult source sentence translations. To test this hypothesis, we randomly sampled source sentences from the WMT' 20 En Zh training dataset and ran translation sampling algorithm on them. We then compared the assigned rewards for the translations when using batch normalization conditional normalization. Figure 2 shows that when unconditional normalization is used the sign of the reward is strongly determined by the source sentence rather than the relative quality of the translation and that this issue mitigated by using conditional normalization. We also note that

---

Under review as a conference paper at ICLR 2023
Batch Normalization
Conditional Normalization
Normalized Reward
2-
1-
00
36
-1
-2-
-3
3
0
2
6
8
Sentence
30
Count
15-
0-
-3
-2
-1
3
33 -2
Normalized Reward
Normalized Reward
Some source sentences are intrinsically more difficult to translate than others. This results in their translations receiving systematically lower BLEU scores. When unconditional reward normalization such batch level normalization (left column) is applied, the training examples for these difficult source sentences predominately receive negative rewards. To correct this, we introduce conditional reward normalization ($2.2) (right column) and that it produces both positive and negative rewards for every source sentence. We also find that the batch level reward distribution (bottom row) is more normal when using conditional normalization.
Figure 2:
distribution of rewards at the batch level is more Gaussian when using conditional normalization. This analysis helps to explain why models trained with unconditionally normalized reward exhibit a catastrophic drop in performance during training. These models are consistently reducing the entropy of easy source sentences and increasing the entropy of diffieult ones which eventually leads to divergence. We explore the concept of source sentence difficulty in the Appendix A..3
4.3 MAD WEIGHTS EFFECT
At first it seems counter-intuitive that we would want down-weight both high and low probability translation samples. Lowering the weight of very bad translations makes sense because they are off policy and, many cases, degenerate. However, nown-weightin translations that have a very high probability seems counterproductive. Prior work has shown that translation models fine-tuned with policy gradient methods end up having lower entropy predictions (Kiegeland Kreutzer] 2021 Choshen et al. 2020). We observed that when sampling from pre-trained cross entropy model, as is done at the very beginning of RL fine-tuning, the highest reward sample usually the greedily decoded translation which also usually has the lowest entropy since it comes from the mode of the sampling distribution. This results in the policy's 's mode getting the most positive reinforcement during training and makes the sampling distribution more peaked. We formulated the MAD weights to try and preserve sampling diversity over the course of training and Figure 6 demonstrates that it is effective this regard. We MAD weights focuses learning to occur in the space that is slightly off the current policy. It is conditional in the sense that the level of low entropy penalization the sampling distribution for the given source sentence and policy state. MAD weights are step in the right direction there is more research to be done this area since does not prevent policy collapse, just slows down the rate at which it occurs.
sort of conditional entropy regularizer that
-weighttggh depends on the characteristics of
4.4 TRAINING STABILITY
We look at two aspects of training stability: 1) the amount variance in generalization performance between random seeds and 2) the rate at which over-fitting occurrs during training. Given this

---

Under review as a conference paper at ICLR 2023
NIST Zh-EEn
WMT'20 En-ZZh
34-
50
33-
dev BLEU
32
Temperature Range
Algorithm
[0.2. 0.6]
47-
MAD
[0.4.0.8]
PPO
[0.6, 1.0]
[0.8, 1.2]
20K Training Step
100K
100K
Training Step
Figure 3: Training curves for random seeds per algorithm. We that MAD training is sta- ble and has low variance between random seeds. PPO exhibits catastrophic drops in performance while MRT has high random seed variance.
Figure 4: MAD temperature hyperparameter sweep. In line with the other sampling based sequence level optimization baselines we evalu- ated, MAD is sensitive the temperature range used to generate samples.
definition of training stability, we can see in Figure 3 that MAD provides stable training for many steps. The difference in generalization performance between the best and worst MAD seed on NIST was 0515 BLEU, Also, in terms of over-fitting, PPO and REINFORCE consistently produced learning curves similar to the orange lines in Figure across across all the datasets we evaluated. We did not observe this catastrophic drop in performance when using MRT or MAD. The primary difference between MRT and MAD in terms of training dynamics is that MAD has lower seed variance and obtains its max development dataset BLEU score later in training.
4.5 TEMPERATURE SENSITIVITY
We find that all of the sequence level optimization algorithms evaluated in this work are sensitive to the temperature used during sampling. Figure shows the MAD training curves during a temperature sweep on 20 En + 7h. We can see that getting the correct temperature range is important, but that there is some leeway in terms of the optimal range. Table 3 in the Appendix shows the optimal temperatures found for each of the datasets and algorithms. Generally speaking, there is agreement among the various algorithms as to the optimal temperature range for a given dataset. One of the benefits of using temperature range rather than single temperature is that smaller sweep can be performed to find this optimal hyperparameter. We were able to reduce the number of sweep experiments from 10 to 4 ($3.4). The MAD algorithm works just as well when using single temperature, but we believe the reduction in computational burden during tuning makes using a temperature range worth the effort.
4.6 IMPACT OF REWARD TYPE
In the experiments reported so far, we have used sentence BLEU as a reward function. However, Choshen al. (2020) have shown that uninformative rewards work as well as informative rewards at improving the BLEU score, conjecturing that the reported improvements in these algorithms are due to training artifacts such as reduced entropy of the predictive distribution rather than reward-driven learning. In this section we look at whether training with MAD on different rewards results a policy that improves those rewards on held-out test sets We can see Table 2 that the models able to generalize on their metric and usually outperform models trained to optimize different metrics. We are able train a model optimize multiple metrics and generalize well on all of them.
WWe note that the constant reward used hy Choshen et al. 2020 would result in no learning in our algorithm
on account of the reward standardisation discussed above 2.2
we do not compare this condition.
8

---

Under review as a conference paper at ICLR 2023
Table 2:
MAD used to optimize different rewards the IWSLT De-En dataset. We report test set results for the checkpoint with the max validation performance on the metric being optimized. For TER, lower score is better, so we optimize TER. The last row, ALL, optimized the equally weighted average of the rewards, (1/6)(BEEU) (BLEU- GLEU + ChrF + Token F1 TER + BLEURT).
Reward Optimized
BLEU
GLEU
ChrF
TER
Token F1
BLEURT
ALL
sBLEU (Papineni
2002
32.1
30.0
55.5
50.0
55.9
58.6
30.3
GLEU Mutton el al. 2007)
32.2
30.5
55.7
49.1
31.0
Chrl (Popovic 2015
32.0
30.0
56.5
49.7
56.0
59.0
30.6
-TER (noover et al.
2006
32.2
30.0
55.8
48.8
56.3
59.2
30.8
Token FI
31.9
30.4
55.5
49.5
57.1
59.5
30.8
BLEURT (Sellam et al. 2020
29.5
28.4
54,9
52.0
62.3
29.6
ALL (equal weight)
32.2
30.4
56.0
49.0
56.8
59.9
31.1
5 RELATED WORK
During the era of linear translation models, setting parameters to optimize rewards was standard. Although cross entropy still widely used, Ranzato et al. (2016) inaugurated using RI. techniques for neural translation models, and Edunov et al. (2018) provide a thorough survey of the topic and find that RL fine-tuning usually improves the original model. Although has been shown to optimize biased approximation to the expected reward objective (Choshen et al., 2020), minimum risk training remains extremely effective (Shen et al. 2016 Kiegeland Kreutzer 2021).
More complicated policy gradient methods have been proposed that rely auxiliary networks. MIXER (Ranzato et al., 2016) predicts a per-token reward baseline while actor-critic methods (Bahdanau et al. 2017) learn per-timestep function concurrently with the translation policy. Although not widely used yet in translation, proximal policy optimization (PPO) (Schulman et al. 2017) has been used train summarization models from human preferences (Stiennon et al. 2020 and this algorithm provided inspiration for some of the techniques we used here.
Another important source of inspiration in the development of algorithm fear" online learning algorithm of Chiang (012)), which used margin-based analysis to argue for decision rule that selected positive and negative training samples on the basis of a combination of current model score and reward. Additionally, pairwise ranking optimization (Hopkins & May (011)) reduced the parameter learning problem to classifying pairs of positively and negatively rewarded trajectories, and our objective can be understood as weighted variant of that objective.
the "hope and
Our work differentiates itself through our conditional reward normalization and approach to entropy regularization through the use of MAD weights.
6 CONCLUSION
We introduce a new policy gradient algorithm, MAD, for machine translation and show that it outperforms cxisting reward-aware training procedures such as REINFORCE, minimum risk training (MRT) and proximal policy optimization (PPO). We test these algorithms variety of datasets, language pair directions, and reward functions and demonstrate good test set performance. Our algorithm excels producing models that generate quality translations, when greedy decoding critical
used. Our analysis shows that the conditional reward normalization we introduce component of the model's success while the MAD weights improve sample diversity and training stability. Taken together, we see an average 2.0 BLEU improvement over the initial cross entropy model and a 1.0 BLEU improvement over the best baseline treatment, MRT, Finally, we offer method for reducing the number of sweeps needed to find the optimal sampling temperature hyperparameter making it easy to use on new tasks.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron C. Courville, and Yoshua Bengio. An actor-critic algorithm for sequence prediction. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France,
April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https: //openreview net. nett/oorumiid=sJDaeeee
Albin Cassirer, Gabriel Barth-Maron. Eugene Brevdo, Sabela Ramos, Toby Boyd, Thibault Sottiaux, and Manuel Kroiss. Reverb: A framework for experience replay, 2021.
David Chiang. Hope and fear for discriminative training of statistical translation models. Journal of Machine Learning Research, 13(40):1159-1187) 2012. httt:://:mmmmm/
v13/ c23/chiing222 ht.ml
Leshem Choshen, Lior Fox, Zohar Aizenbud, and Omri Abend. On the weaknesses of reinforcement learning for neural machine translation. In International Conference on Learning Representations. 2020. URL https: http:///pppeeeeeee////me net/ forum? Htw.:/rrrrr...a-aeeeeee-----
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences.
Guyon, U. V. Luxburg, S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Gameet (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. cc/paper/2017/EEE
neurips
Corinna Cortes, Yishay Mansour, and Mehryar Mohri. Learning bounds for importance weighting. In J. Lafferty, C. Williams, J. Shawe-Taylor, R. 7emel, and A. Culotta (eds.), Advances in Neural Information Processing Systems. volume 23. Curran Asso- ciates, Inc., 2010. URL httpp::/pppoccedngg.....t http://awww.....ca////////////// http//pppe//2///LUU// 59c33016884a6211 2552222166667775922222777333 Papeee pdf
Raj Dabre and Atsushi Fujita. Softmax tempering for training neural machine translation models, 2020.
Shachar Don- Yehiya, Leshem Choshen, and Omri Abend. Prequel: Quality estimation of machine translation outputs in advance. arXiv preprint arXiv:2205.0911 09970, 2022.
Sergey Edunov, Myle Ott, Michael Auli, David Grangier, and Marc' Aurelio Ranzato. Classical structured prediction losses for sequence sequence learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics; Human Language Technologies, Volume (Long Papers), pp. 355-364, New Orleans, Louisiana. June 2018. Association for Computational Linguistics. doi: 10.18653/v 1/N18- 1033. URL https //aclanthology. www//o8-n/OO -1033
Mark Hopkins and Jonathan May. Tuning as ranking. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pp. 1352-1362, Edinburgh, Scotland, UK., July 2011. Association for Computational Linguistics. URL httthtt:::/// httaattss/acaaatttlloooo/ooo org/ D11 -1125
Samuel Kiegeland and Julia Kreutzer. Revisiting the weaknesses of reinforcement learning for neural machine translation. In Proceedings of the 2021 Conference of the North American Chapter of
the Association for Computational Linguistics: Human Language Technologies, pp. 1673-1681, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v 1/2021 maacl-main. 133. http::::: //tc//acaaaaa http://aaaaaattoooaaoe org 2001 maacl-main.11
Julia Kreutzer, Artem Sokolov, and Stefan Riezler. Bandit structured prediction for neural sequence-to- sequence learning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 1503-1513, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 11118663/1111111110
org/P17-1138
Taku Kudo. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of ACI., 2018.
10

---

Under review as a conference paper at ICLR 2023
Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of EMNLP, 2018
Clara Meister, Ryan Cotterell, and Tim Vieira. If beam search is the answer, what was the question? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing Online, November 2020. Association for Computational Linguis-
(EMNLP), pp. wwtp:/00m000.mmmai.....mmmm 170. URL
tics. doi: 10. 11531166/22200200..
emr p-ma In. 170
Andrew Mutton, Mark Dras, Stephen Wan, and Robert Dale. GLEU: Automatic evaluation of
sentence-level fluency. In Proceedings of the 45th Annual Meeting of the Association of Compula- fional Linguistics, pp. 344-351, Prague, Czech Republic, June 2007. Association for Computational Linguistics. URL htt.t:::////.. //ac anthan Satholoog www.llyyy...oooooooooooo wtgg/07-114400000
Khanh Nguyen, Hal Daumé III, and Jordan Boyd-Graber. Reinforcement learning for bandit neural machine translation with simulated human feedback. Proceedings of the 2017 Conference Empirical Methods in Natural Language Processing, pp. 1464-1474, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/v URL https::/aclaatthol
org/D17-115;
Richard Yuanzhe Pang and He He. Text generation by learning from demonstrations. In ICLR, 2021.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for auto- matic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 10.3115/1103303.10733 https://acl http://aec.aattoooooooooor/////2
URL
Matteo Papini, Andrea Battistello, and Marcello Restelli. Balancing learning speed and stability in policy gradient via adaptive exploration. In Silvia Chiappa and Roberto Calandra (eds.). The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy,, volume 108 of Proceedings of Machine Learning Research, pp. 1188-1199, PMLR, 2020. URL http:://pooceeiinggs mlr. / papini Sitani20a.htinn
Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. In International Conference on Learning Representations, 2018. URL https
//openreview net wwwp::/wwweeeoeeeeeeeeeea D
Maja Popović. chrF:
character n-gram F-score for automatic MT evaluation. In Proceedings of the
Tenth Workshop on Statistical Machine Translation, pp. 392-395, Lisbon, Portugal, September 2015, Association for Computational Linguistics. doi: 10. 18653// 1/110-3011155500 URL https
//aclanthhology.oor//11----00
Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pp. 186-191, Brussels, Belgium, October 2018.
Association for Computational Linguistics. doi: 10. 10.18663/11/1186663 URL https://
Doina Precup, Richard S. Sutton, and Satinder Singh. Eligibility traces for off-policy policy evaluation. In In Proceedings of the Seventeenth International Conference on Machine Learning (ICML-00), pp. 759-766, 2000.
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican. Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen. Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre. Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenie Donato, Angeliki Lazaridou, Arthur Mensch, Jean- -aptistt Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux,
11

---

Under review as a conference paper at ICLR 2023
Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d' Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, lason Gabriel, William Isaac. Fd Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling language models: Methods, analysis and insights from training gopher, 2022.
Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrett neural networks. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1119-11999 2017. doi: 10.1 1099/CPR.2001.133.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347 2017. URL
org abs/
1707. 06347
Thibault Sellam, Dipanjan Das, and Ankur Parikh. BLEURT: Learning robust metrics for text
generation. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7881-7892 Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020. main. URLhtttpa
www.thhoooo...rg/2000 acl-mallal
704
Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. Minimum risk training for neural machine translation. In Proceedings of the 54th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1683-1692, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10. 118653/1//P1-1111
URL httphhtts:aaaee http///aaaannntlo..ooooo
Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, pp. 223-231. Cambridge, Massachusetts, USA, August 8-12 2006. Association for Machine Translation in the Americas. URL paaetps://aclanhhon ogy ore...rg//000 2006 amta-
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning summarize with human feed- back. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 3008-3021, Curran Asso- ciates, Inc., 2020. URL https::/ptoceeeiinnn wtwp://eeediinnnn.nnueeeeeeeeeee cc wctppppee///////ee/.c/pppp file/
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need.
Guyon,
U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gamett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL htttps://ppocceedinng htwp:eeeeddnnn..eeeniieeee nww.inuur..... httpipss.ccppppee//// ccpppaper////////....e (ateeeeeeeeeee000eeee4443533 Paa.:aacccccaaaeaaaaaa.aer.
pdf
Lex Weaver and Nigel Tao. The optimal reward baseline for gradient-based reinforcement learning. In Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence, UAI'01, pp. 538-545, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc. ISBN 1558608001,
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229 2569925566111 1992.
Lijun Wu, Fei Tian, Tao Qin, Jianhuang Lai, and Tie- Yan Liu. A study of reinforcement learning for neural machine translation. In Proceedings of the 2018 Conference Empirical Methods
12

---

Under review as a conference paper at ICLR 2023
in Natural Language Processing, pp. 3612-3621, Brussels, Belgium, Association for Computational Linguistics. doi: 10.18653/1//18--13977 URL https://
2018.
aclanthology.orgn 01t.-/ooyy.rr//1--133371
Fan Yang, Gabriel Barth-Maron, Piotr Stañczyk,, Matthew Hoffman, Siqi Liu, Manuel Kroiss, Aedan Pope, and Alban Rrustemi. Launchpad: A programming model for distributed machine learning research. arXiv preprint arXiv:2106.044 (6.00606.6,, 2021. URL BRRLttts:://ccc /ttpps////x.....//////aa/./// wttp//////bb////oo
04516
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.0853000 08593, 2019. https:/s:///aae wwwporrgg////ooooo//ot...t..eeet 1909. 08593
13

---

Under review as a conference paper at ICLR 2023
A
SUPPLEMENTARY EXPERIMENTS
A.1
ABLATION STUDY FIGURES
WMT14 En-De
ym
Mean Unique Conditional Sampless
IWSLT'14
WMT'14 En-De
Treatment
BLEU
Cond Rwd MAD
Cond Rwd
Treatment
25
Cond Rwd & MAD
Cond Rwd
Neither
24
0
1K
S0K
100K
Training Step
IWSLT114 De-En
NIST Zh-En
33
30K Training Step
60K
50K
100K
50
Training Step
49
30K
60K
100K
200K
Figure 6: We plot the mean number of unique samples generated by the worker nodes over the course of training for two different datasets. For each source sentence, 12 samples are generated and is thus the upper bound. The green line is the full MAD algorithm while the orange line is ablation where the MAD weights have been removed. We see that including the MAD weights helps slow the loss of diversity during training.
Training Step
Training Step
Figure 5: Ablation study of the components of the MAD algorithm multiple datasets, see for more detailed description. The most important component of algorithm is condi- tional reward normalization, orange lines. Addi- tionally including the MAD weights, green line, improves training stability and offers additional performance improvement.
A.2 ADDITIONAL BEAM SEARCH RESULTS
Table g shows results (on the test set) using larger beams and different decoding hyperparameters. MAD trained with SentenceBLEU continues to outperform other training objectives, and displays the notable behaviour that decoding without length normalization with large beams effective, whereas performs drops significantly with CE and REINFORCE trained models.
We emphasize that the beam search length normalization term is the same for all models and datasets reported in Table Most prior work in machine translation optimizes per dataset and only reports this score. We have confirmed that our cross entropy models match prior work when using the tuned a. In this sense, we believe the BLEU numbers report a lower bound, which might be improved through a tuning.
A.3 SOURCE SENTENCE DIFFICULTY
It's somewhat difficult to quantify how hard source sentence will be translate. However, there is evidence that the uncertainty (as measured by entropy) a powerful Language Model has when modeling the source sentence can used as a proxy for this (Rae et al. (022). We see Figure that there is negative relationship between source sentence entropy and translation performance. This means that unconditional reward normalization methods will incorporate this bias.
A.4 ADDITIONAL REMARKS
We note that the conditional reward normalization could be applied to both REINFORCE and PPO, however, decided against it to keep these baselines line with prior work. We also believe our baselines coincide with the form most likely to be used in practice. We did run some preliminary
14

---

Under review as a conference paper at ICLR 2023
Mean BLEU Samples
WMT En-
Spearnan corr -0.22
0.8
0.4
0.2
Source Sentence Mean Word Entropy
Figure 7: Plot of the relationship between source sentence difficulty and the average score of sampled translations. We use average word entropy from strong 280 billion parameter language model (Rae et al.] 2022) the more uncertainty a language model has about the source sentence, the lower the average BLEU of its sampled translations.
proxy for the uniqueness/diffieulyy of given source sentence. We that
experiments with PPO where we replaced with 7, however, the results were mixed. PPO has few additional hyperparameters that can be tuned we leave this future research.
Table 3: Temperatures used for models in main results. REINFORCE used same temperatures as PPO. Only MAD uses a temperature range, other algorithms single temperature.
Table 4: models.
Base settings for our Transformer
Transformer - Base Settings
Setting
Value
Temperatures
embeddings dim
512
Dataset
Algo
T min
T Tmax
feed forward
2048
PPO
1.0
num layers
6 enc 6 dec
IWSLT' 14 De - En
MRT
0.9
num heads
8
MAD
0.8
1.2
dropout
PPO
1.2
tied embeddings
True
IWSLT' 14 En - De
MRT
0.6
tied softmax
True
MAD
0.8
1.2
optimizer
Adam
PPO
0.2
learning rate (n) 7) warmup steps
le-5
WMT' 14 De + En
MRT
0.6
1000
MAD
0.4
0.8
n scheduling
Constant
PPO
1.2
gradient clipping
[oobl_norm([I)
WMT' En De
MRT
0.4
training steps
200,000
MAD
0.4
0.8
seq length global batch size
128
PPO
1.2
256
WM Ps En
MRT
1.2
MAD
0.8
1.2
PPO
1.2
W0 En + Ps
MRT
1.0
Table 5: Deviations from Transformer base
MAD
0.4
0.8
settings.
PPO
1.0
NIST Zh + En
MRT
0.8
Transformer Deltas
MAD
0.6
1.0
Dataset
Setting
Value
PPO
0.8
seq length
64
WMT'20 Ps En
WM Zh + En
MRT
0.8
dropout
.3
MAD
0.4
0.8
WMT De En
seq length
144
PPO
0.8
feed forward
1024
W0 En + Zh
MRT
0.6
IWSLT' 14 De ++ En
num heads
MAD
0.2
0.6
dropout
.3
15

---

Under review as a conference paper at ICLR 2023
Table 8: Additional test results when using different Beam Search hyperparameters. Even when large beams are used without length normalization, MAD performs well indicating that it is better calibrated.
Beam Search
Beams = Length Normalization (a) = None
NIST
IWSLT'
14
WMT 14
WMT 22
Model
Zh-En
De-En
En-De
De-En
En-De
Zh-En
En-Zh
Ps-En
En-Ps
Cross Entropy
47.0
31.1
27.5
31.0
26.8
25.1
34.7
7.8
5.9
26.3
REINFORCE
47.0
31.1
27.5
29.9
26.7
25.0
34.9
7.6
5.8
26.2
PPO
47.1
31.2
28.0
31.2
26.7
25.0
34.7
8.1
7.5
26.6
MRT
47.4
31.7
27.5
31.8
27.0
27.0
38.8
9.2
7.8
27.6
MAD
48.3
32.4
28.2
32.0
27.4
27.7
39.7
9.3
8.0
28.1
Beam Search Beams 50
Length Normalization (a) = None
NIST
IW
W4
WMT 20
Model
Zh-En
De-En
En-De
De-En
En-De
Zh-En
En-Zh
Ps-En
En-Ps
Cross Entropy
42.8
28.1
26.5
28.5
24.9
22.9
32.3
4.0
1.3
23.5
REINFORCE
43.0
28.0
26.4
28.8
25.4
22.8
32.0
2.5
1.3
23.3
PPO
45.9
31.2
28.0
29.1
24.9
23.1
32.3
6.9
6.3
25.3
MRT
47.1
30.8
26.5
31.2
27.1
26.8
38.4
7.1
27.1
MAD
48.5
32.4
28.1
31.9
27.5
27.4
39.5
7.9
6.4
27.7
Beam Search
Beams 50
Length Normalization (a)
1.0
NIST
IWSLT' 14
WMT 14
WMT 20
Model
7h-En
De-En
En-De
De-En
En-De
Zh-En
En-Zh
Ps-En
En-Ps
Cross Entropy
48.2
31.6
27.9
31.7
25.3
25.9
36.8
5.4
2.9
26.2
REINFORCE
48.3
31.6
28.0
30.5
26.0
25.9
36.9
6.2
2.8
26.2
PPO
47.8
31.3
28.2
31.7
25.3
25.9
36.9
7.1
6.9
26.8
MRT
47.6
32.1
27.9
31.9
26.4
27.5
39.0
9.0
7.2
27.6
MAD
48.3
32.5
28.4
32.1
27.4
28.0
39.8
8.8
7.4
28.1
B HYPERPARAMETERS
We provide the hyperparameters for our Transformer models in Tables 4nd settings are in Table 6 Finally, the setting used when calculation sacreBLEU found in Table
and 5 The algorithm specific
Table 6:
Settings for sequence level algo- rithms. Note that did experiments with D = 12 for MRT and found no improvement in development set BLEU. Given the slow training speed of MRT, we opted to use the con- vention of 5 samples.
Table 7: sacreBLEU settings for each language.
Algorithm Settings
Algorithm
Setting
Value
sacreBLEU Settings
PPO D
0.2
Language
Setting
Value
MRT
5
De
tokenizer
intl
max times sampled
1
En
tokenizer
13a
MAD
min size sample
512
Ps
tokenizer
intl
reverb
max size
4096
Zh
tokenizer
zh
setrings
sampler
Uniform
smooth method
exp
All
remover
Fifo
smooth value
0
16

---

Under review as a conference paper at ICLR 2023
C
DATASETS
We evaluate all the models the following translation tasks: NIST Open MT Chinese English task, IWSLT" 14 English h ++ German translation task, WMT English ++ German news translation task, and the WM Chinese ++ English and Pashto ++ English news translation tasks. The sizes of each dataset is avaliable in Table
Preprocessing We perform text normalization on the datasets before tokenization.
All languages Unicode canonicalization (NKFD from), replacement of common multiple encoding errors present training data, standardization of quotation marks into "directional" variants.
English Replace non-American spelling variants with American spellings using the aspell library Punctuation split from English words using purpose-buil library.
Chinese Convert any traditional Chinese characters into simplified forms and segment into word-like units using the Jíeba segmentation tool
We encode text into sub-word units using the sentencepi ece tool Kudo & Richardson (2018). When generating our own subword segmentation, we used the algorithm from Kudo (2018) with a minimum character coverage of 0.9995.
Tokenization
Table 9: Number of training, development, and test examples in each dataset.
Table 10:
We show the training speed function of the number of 2x2 TPU workers used. Speed is measured in seconds per 1000 training steps.
Dataset
Train
Dev
Test
IWSLT' 14 De En
164K
7,466
6,750
IWSLT' 14 En De
173K
1,474
6,750
Training Speed
W0 Ps + En
516K
5,860
2,719
Algorithm
Workers
See per IK
NIST Zh - En
1,45M
1,664
5,146
REINFORCE
n/a
2,038
WMT' 14 De + En
4.7M
3,000
3,003
MRT
n/a
1,728
WM Zh + En
21.8M
2,000
2,000
2
1,898
PPO MAD
WM En - Zh
21.8M
1,997
1,418
4
946
D IMPLEMENTATION DETAILS
Software We use Launchpad Yang et al. 2021 to create and launch our DAG computing graph. The graph consists of Reverb table (Cassirer et al. 202I) that holds training examples, multiple worker nodes that generate and publish examples, a learner node that pulls and trains on examples, and an evaluator node that loads fresh checkpoints to calculate dev set sacreBL.EU. It is important to not make the Reverb table cache too large avoid the table filling with stale training examples. This can happen if the training node job is temporarily deallocated while the workers are still running.
Compute All of our program nodes for PPO and MAD used 2 data generating workers running for 50K steps, while final results used workers and ran for 200K steps. Table 10 in the body of the paper provides the wall clock training speed for each algorithm.
2x2 TPU configuration. Our hyperparameter sweeps
http:://githbb.coo//////////1/1
17

---

Under review as a conference paper at ICLR 2023
Training time Because of our distributed algorithm, able obtain faster training times by increasing the number of workers that sample from the current policy (this contrast to purely "on-policy" REINFORCE and MRT algorithms). Table I0 shows that get near linear scaling as the number of workers increased.
18


Reviews:
[{"review_id": "ia5lRtwJnHP", "reviewer": "Reviewer_qbvg", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper propose an easy-to-implement fine-tuning for machine translation via reinforcement learning. \nThe major issue for reinforcement fine-tuning is described as variance reducing, both in rewards and updates.\nThe main contributions are listed as the following:\n- A conditional reward normalization to reduce variance;\n- A novel robust importance weighing scheme similar to trust-region update, striking the balance between exploration and exploitation.\nThe translations are first sampled for tuning rewards via a variable temperature. Then, the rewards are standardized to update the central policy by rescaled weights.\nThe methods improves the specific translation metric and certain robustness.", "summary_of_the_review": "The paper presents a interesting framework for fine-tuning MT system. But the analysis and discussion of the method is problematic. It seems to me that the interpretation from the authors are not well-matched with the experiment results.", "strength_and_weaknesses": "# reasons to accept:\n-  This paper is well organized and easy to understand. \n- The MAD importance weight is a sophisticated trust region scheme that achieves robust translation against decoder’s beam-size and length normalization. Experiments show that MAD also achieves stable convergence compared to traditional PPO. The scheme also strikes a balance between exploration and optimization, which is worth noting. \n- The proposed methods greatly boost stability for RL tuning. The improvement is satisfying compared to the baseline. \n- Authors conduct thorough experiments across different languages and different reward types to validate the proposed methods. The results are promising.\n\n# reasons to reject:\n- The proposed reward normalization is a standardization, which can hardly be considered as ‘source conditioned’. The authors do not explain how the conditional reward standardization relates to the difficulty of source sentences for its designed purpose. \n- According to the ablation study, both MRT and PPO fail to converge on the long run. But SOTA RL tuning involves certain implementation to achieve tuning by MRT and PPO, which should be clarified in baseline settings. \n- The paper is somewhat misleading. According to their ablation study, conditional reward normalization is the most important one among the three components. The MAD method, as stated in the title, does not contribute much to the performance gain, and it even fails the training process.\n- Reinforcement learning can be tricky and empirical. The tradeoff to implement such methods for tuning rather than data augmentation is worth discussing. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "M0OGX6G_XsX", "reviewer": "Reviewer_VmFP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a simple, distributed approach to robust RL coined MAD. The main components of MAD are:\n1) conditional mean and variance normalization of the task reward based on a population of conditional samples, sampled from the current (local) policy over a range of temperatures. This promotes balance in terms of ensuring that there is positive and negative feedback for each set of conditional samples, and regularity in the gradients across sets of conditional samples.\n2) An importance weight that depends on the mean absolute deviation (MAD) of the probability of conditional samples from the median probability in a sample population. This is meant to promote more conservative updating of the policy, and mitigate against policy collapse, acting as an entropy regularizer.\n\nMachine translation results on NIST, IWSLT’14, WMT’14, and WMT’20 seem to indicate that the method outperforms several baselines, including MRT (see S&W section for caveats).", "summary_of_the_review": "Based on the current manuscript, it appears that the proposed method is utilizing conditional baselines, while competing methods which support conditional baselines are not, which is not appropriate, and will necessitate a major revision of the paper before it can be considered for acceptance. In addition, further investigations around their importance weighting heuristic and comparisons with techniques that directly estimate their baselines and so do not need a sample population to train (and so are much more efficient), should also really be included in their experimental investigation.", "strength_and_weaknesses": "Strengths:\n\n- Well written in general. Ablations over using conditional reward normalization (fig. 5), MAD IWs (fig. 5,6), and different reward functions (Table 2) are included.\n\n- Performance comparisons with several important baseline methods, including minimum risk training (MRT) (with serious caveats, see limitations below for details).\n\nLimitations:\n\n- Conditional reward normalization (CRN) is identified as the key factor to MAD performance. However, based on the current manuscript (e.g. section 2.2 and discussions elsewhere), it seems that the baseline approaches being compared to such as REINFORCE and MRT, which trivially have conditional variants of their baselines, are NOT conditionally normalized. The importance of CRN is well known, and the comparisons must be \"apples to apples\".\n\n- Techniques such as MRT and MAD rely on sample population statistics for reward normalization, whereas other techniques estimate a conditional baseline and are much more sample-efficient to train, including cited methods in the paper such as in MIXER (learned baseline), self-critical training (MAP sequence estimate defines baseline), and actor-critic methods for sequence prediction (word slot-dependent rewards). Since conditional reward normalization seems to be the dominating factor in performance, it seems essential to also situate the results relative to one or more of these techniques (the former 2 are very easy to try out).\n\n- Importance weighting based on how far from the median probability each sequence is in probability seems excessively conservative for noisy models such as text sequence generators, and will certainly foster the stated goal of \"exploration-heavy\" training, but it will also prevent the model from converging. Since this a heurstic, I feel that some other competing heursitics should be compared to and reported on... For example, perhaps the distance from expected probability is better, it is less \"exploration heavy\", but will allow the policy to converge (for better or for worse).\n\n- Related, it seems that MAD reweighting should be compared with standard regularization baselines, like an entropy term!\n\n- The main components of the approach 1) mean and variance normalization of conditional rewards, and 2) importance weights based on the distance from the median probability can be considered standard and ad hoc, respectively. From this perspective, the paper is lower in novelty for a method-focused ML conference.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "-TvCBV_I6rj", "reviewer": "Reviewer_Gvqo", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper improves the policy gradient algorithm with three methods: (1)multi-temperature sampling, generate multiple translation samples of the same source sentence using different temperatures; (2)conditional reward normalization, standardize the rewards of the same source sentence by removing the mean and dividing by the standard deviation; (3)MAD importance weights, decide which sampled trajectories are most valuable for updating the policy and which others should be downweighted. Experiments on multiple machine translation datasets show the effectiveness of the proposed method.", "summary_of_the_review": "This paper improves the policy gradient algorithm with three methods. The performance looks good, but the major improvements come from conditional reward normalization, which is more of a trick and not novel. ", "strength_and_weaknesses": "The strengths of this paper include:\n1.\tThe proposed method shows good performance, especially on greedy decoding.\n2.\tVery extensive experiments on multiple datasets and multiple settings.\n3.\tThis paper brings new insights for improving RL-Based NMT.\n\nThe weaknesses of this paper include:\n1.\tFigure 5 shows that the major improvements come from conditional reward normalization, which is more of a trick and not novel to me. The improvement of MAD itself seems to be marginal.\n2.\tTable 2 lacks the results of the cross-entropy baseline. It is important to know whether the proposed method only improves BLEU and lower other metrics.\n\nA few questions:\n1.\tWill you release the source code?\n2.\tIn REINFORCE and PPO, did you sample multiple translations and calculate their average reward as the reward baseline?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject