Simple and Effective Input Reformulations for Translation
Authors: Brian Yu
Hansen Lillemark
Kurt Keutzer
Venue: EMNLP 2023 Main (2023)
Keywords: natural language processing
data efficiency
input reformulations
multilingual
translation
foundation language models
finetuning
machine learning

Abstract: Foundation language models learn from their finetuning input context in different ways. In this paper, we reformulate inputs during finetuning for challenging translation tasks, leveraging model strengths from pretraining in novel ways to improve downstream performance. These reformulations are simple data level modifications, require no additional collection of training data or modification of data at inference time. They can be applied either on single language pair translation tasks or massively multilingual translation tasks. Experiments with these techniques demonstrate significant performance improvements up to \textbf{3.5 chrF++ on the Flores200 translation benchmark}. We hope our research accessibly improves finetuning data efficiency, enabling more effective training to scalably improve state-of-the-art performance. Our code is released \href{https://github.com/bri25yu/LanguageModelExperimentation}{here}.

Simple and Effective Input Reformulations for Translation
Brian Yu, Hansen Lillemark, Kurt Keutzer
University of California, Berkeley Berkeley Artificial Intelligence Research (BAIR) (bri25yu, heillllllleelll hlillemark,
/etree Jeut edu
Abstract
target translation pair. POSE simply appends a prefix of the target English output to the Classical Tibetan input. We see qualitative improvements in the variance of the training curves. When evaluated on the same test set with no reformulations, POSE significantly increases overall translation perfor- mance compared to the direct finetuning baseline, up to 10.3% 2.8 BLEU.
Foundation language models learn from their finetuning input context in different ways. In this paper, we reformulate inputs during fine- tuning for challenging translation tasks, lever- aging model strengths from pretraining in novel ways to improve downstream performance. These reformulations ifications, require no additional collection of training data or modification of data at infer- ence time. They can be applied either sin- gle language pair translation tasks or massively multilingual translation tasks. Experiments with these techniques demonstrate significant performance improvements up to 3.5 chrF++ the Flores200 translation benchmark. We hope our research accessibly improves fine- tuning data efficiency, enabling more effective training scalably improve state-of-the-ar per- formance. Our code is released here.
simple data level mod-
The POSE setup had many adjustable hyperper- ameters relating to task difficulty, task curriculum, and substring selection for scaffolding. We find that input reformulation setups should consist of 20% less informative examples, and 80% harder and more informative examples. More ablation details can be found below.
Second, we approach the massively multilingual Flores200 translation benchmark (NLLB-1H -eam et al., 2022). mT5 does not struggle in the initial steps of finetuning on Flores200 the same way it did on tib2eng. Even so, we begin by replicating the tib2eng POSE setup on Flores200 by appending a partial output the target translation to the input translation. As expected, this setup matched but did not improve upon the baseline performance.
1 Introduction
Foundation language models (FLMs) are power- ful and task-agnostic models. They are pretrained on language understanding objectives, enabling strong performance on downstream language tasks (Brown et al., 2020; Shoeybi et al., 2020; Xue et al., 2021; Hoffmann et al., 2022; Chowdhery et al., 2022; Zhang et al., 2022a; Chung et al., 2022; Workshop, 2023; Touvron et al., 2023). FLMs then either prompted or finetuned for downstream use.
The Flores200 benchmark consists of paral- lel examples of the same sentence different languages. In our second approach, we extend the tib2eng POSE reformulation to create the "Parallel Scaffold in English" or ParSE reformula- tion, shown in Figure/ 1. ParSE appends the corre- sponding full parallel English translation (provided by Flores200) the input. Following the tib2eng setup, we use data mix of 20% baseline (less informative) and 80% ParSE (more informative) examples. ParSE significantly improves translation performance, up to 17.2% 3.5 chrF++.
In this paper, we present three different data ef- ficient techniques for improving translation per- formance, applied to the multilingual FLM mT5 during finetuning (Xue et al., 2021). In our first approach, we train mT5 on Classical Tibetan to English (tib2eng) translation task. mT5 strug- gles heavily in the initial training steps. Thus, for the first 20% of finetuning, we apply the "Partial Output English Scaffold" or POSE reformulation, shown in Figure 1. Tib2eng translation examples consist of a Classical Tibetan source and English
We postulate that POSE and ParSE improve translation performance part because they en- able mT5 to attend to an in-distribution pretrain language with strong monolingual performance. our third approach, we explore the efficacy of par-

---

Baseline
German: Das ist gut.
English:
Partial Output English Scaffold (PoIE) (POSE)
German: Das ist gut.
That is good.
English: That is
That is good.
Parallel Scaffold in English (ParSE)
mT5
German: Das 1st gut.
Está bien.
English: That good.
Spanish:
Está bien. That is good.
Mixed-language Parallel Scaffold (MiPS)
German to Spanish: Das ist gut.
Chinese to English: HP/RRF.
Figure 1: Task reformulations. Baseline: direct translation pair. POSE: append a prefix the target translation the input translation. ParSE: append parailel English translation the input translation. MiPS: append different parallel translation to both the input and output.
allel scaffolding that does not require strong mono- lingual performance using the "Mixed-language Parallel Scaffold" MiPS reformulation, shown in Figure MiPS appends different parallel translation to both the input and output for a total of distinct languages per input. Again, we use data mix of 20% baseline and 80% MiPS examples. MiPS also improves translation performance, up to 9.1 % / 1.6 chrF++. Scaffolding with the strongest performing pretraining language (ParSE) outper- forms scaffolding with a mix of other languages (MiPS).
2 Related work
Our work can be viewed as a data efficiency tech- nique for translation. Past works in translation have explored data augmentation (Sennrich et al.. 2016; Fadace et al., 2017), sample re-weighting (Shu al., 2019; Ren et al., 2019; Gu et al., 2018), and curriculum learning (Koemi and Bojar, 2017; Zhang al., 2018; Platanios et al., 2019; Zhang et al., 2019; NLLB-Teame et al., 2022). These ap- proaches vary in effectiveness, are not generaliz- able, and introduce complexity into the training process. Curriculum learning approaches in par- ticular are typically complicated and unsuccessful, because they are designed using intuition how humans treat inputs, which may differ from how models treat inputs. In contrast, our input reformu- lations are simple and can be directly applied any sequence-to-sequeene to-seqeence task.
Finally, we perform analysis on the languages in the translation set. Using balanced dataset like Flores200 allows mT5 to partially overcome pretraining dataset size biases. Naturally, translat- ing into lower resource languages more difficult than translating into higher resource languages, but we find that the ParSE and MiPS reformulations improve translation into all languages across the board, rather than disproportionately improving performance on high resource languages.
Previous work has explored prompting frozen language model using manually curated prompts (Brown et al., 2020; Touvron al., 2023; Petroni et al., 2019). Results are typically sensitive the exact prompt used. This technique cannot be ap- plied to larger corpora because limited by the number of examples that can feasibly fit into a single input context. Other works have explored finetuning with fixed prompt without leveraging the target output as a part of the input (Radford et al., 2018, 2019; Dong et al., 2019; Devlin et al., 2019; Lewis al., 2019; Sun al., 2019; Liu al.,
In summary, we propose input reformulations on translation tasks. These reformulations require no additional data, have few hyperparameters, and are simple to implement. When finetuning gle language pair translation task, if the target out- put language is in the model's pretraining dataset distribution, the POSE reformulation can be ap- plied. When translating between multiple language pairs, the ParSE reformulation can be applied to the strongest performing pretraining language.
sin-
2019; Clark al., 2020; Yang et al., 2020; Raffel
et al., 2020; Gao et al., 2021; Schick and Schütze,
2021; au2 et al., 2021; Xue et al., 2021; He et al.,
2021; Taori et al., 2023).

---

Classical Tibetan to English input reformulations
is challenging for mT3., Additionally, mT5's tok- enizer was not trained on Tibetan. We use mT5's current tokenizer and use the byte-level fallback capabilities of the underlying SentencePiece tok- enizer to encode unknown tokens (Xue et al., 2021). We use the BLEU metric (Papineni et al., 2002) for evaluation.
Baseling mweet sqe Pelli 1
POSE
Gon gp grs RP May we be born on
Target output May we bom on the Copper Coloured Mountain of Glory.
Figure 2: POSE reformulation applied to the tib2eng translation task. Changes are highlighted in red.
The dataset consists of 450k train, 5k validation, and 5k test translation pairs. The tokenized Tibetan inputs are mean 72 and median 51 tokens long; we use maximum sequence length of 256. We train for 10k steps and batch size of 512 translation pairs (about 35k tokens per batch, about 350M to- kens total), equivalent to epochs. We the AdamW (Loshchilov and Hutter, 2019) optimizer with parameters 31 = 0.9, $22 = 0.999, and weight decay We use a constant lcarning rate schedule with no warmup. The models converge success- fully under this data compute budget. We ablate over learning rates in (1e-3, 2e-3, 3e-3) for 600M and IB parameter models (the default finetuning learning rate for mT5 is le-3 (Xue et al., 2021)) and (3e-4, 5e-4, 1e-3) for 3B parameter models, where we found lower learning rates to be empirically better.
Following the success of fixed prompt tech- niques, other works proposed prompt tuning setups (Shin et al., 2020; Schick al., 2020; Li and Liang, 2021; Hambardzumyan et al., 2021; Lester al.,
2021; Zhong et al., 2021b; Wallace al.. 2021; Haviv et al., 2021; Jiang et al., 2020; Chen al., 2022; Qin and Eisner, 2021; Liu et al., 2021; Han et al., 2021; Zhong et al., 2021a; Lu et al., 2022; Ben-David et al., 2022: Wang et al., 2022a; Zhou et al., 2023b). These prompt tuning setups were typically used in the context of compute efficiency: training smaller number of prompt-related param- eters input into larger frozen language model. These setups are an orthogonal improvement to our proposed input reformulations.
Previous approaches also investigated dataset improvements for better downstream task perfor- mance. These approaches gathered additional data for model training augment the model's input context (Chung et al., 2022; Wei et al., 2023; Wang et al., 2023a; Iyer al., 2023; Min al., 2022; Wei et al., 2022: Wang et al., 2022b; Gu et al., 2023; Wang et al., 2023b; Zhang et al., 2022b; Press et al., 2023; Zhou et al., 2023a). They require large, spe- cific, and high quality datasets to be collected. On the other hand, our input reformulations require no additional data.
We perform evaluation on the models and save checkpoints every 200 steps, for a total of 50 evalua- tions, and we use the highest scoring checkpoint for all results. Models were trained on GPU nodes of either NVIDIA A5000 24GB GPUs or 8 NVIDIA A6000 48GB GPUs. The typical train time varied from 8 hours for the smallest models to 80 hours for the largest. We leverage the Deepspeed library https: //wps::/www. dewpspeed. ai/ for training in the half precision 6 as well as for effective multi- GPU training.
In all the following results tables, we report the highest test set BLEU scores and standard deviation (std) values over learning rates.
Overall, our approach differs from previously ex- plored approaches by avoiding prompts and lever- aging the target output as a part of the input refor- mulation. Our input reformulations are a data-level change that can be easily applied to any training setup.
3.2 Motivation
We begin by training baseline mT5 models on the tib2eng dataset. The resulting training curves are shown in Figure 3 with the blue colored curves. Clearly, mT5 struggles in the first 2000 steps 20% of the training steps. With the intuition of reducing task difficulty, we design an easier task reformulation apply only in the first 20% of train- ing. First, we select prefix from the target English translation. The length of this prefix uniformly randomly chosen over the full length of the English
3 Experiments on a difficult single language pair translation task
3.1 Setup
We perform experiments on a Classical Tibetan to English (tib2eng) dataset. Critically, Classical Tibetan is not found in mT5's pretraining dataset, while English is. As a result, the tib2eng dataset

---

Table 1: Task difficulty experiment results on mT5 600M
Table 3: Prefixssuffix experiment results on m15 600M.
Substring
% reform
BLEU
Std
Diffieulty +
% reform
BLEU
Std
Baseline
0%
23.5
1.64
Least difficult
100%
21.1
0.29
Prefix
20 %
24.6
0.26
50%
23.9
0.05
Prefix+suffix
12%
24.8
0.55
20%
24.6
0.26
20%
24.5
0.90
Most difficult
0%
23.5
1.64
40%
24.0
0.12
Table Curriculum experiment results on mT55 600M.
of reformulating only the first 20% of training ex- amples (i.e. all examples in the first 2000 steps), we rigidly add 100% of the output to the input at the beginning of training, and linearly scale down to 0% added the end of training. (Curriculum 2): Instead of reformulating 100% of training ex- amples in the first 2000 steps, we reformulate 80% of the inputs for the first 2000 steps, linearly scale down from 80% reformulated to 40% reformulated for the next 4000 steps, and reformulate no ex- amples for the last 4000 steps. (Curriculum 3): Instead of using uniformly random length prefixes for the first 20% of training examples, we rigidly add 100% of the output the input and linearly scale down 0% at the end of 2000 steps.
Setup
BLEU
Std
Baseline
23.5
1.64
POSE
24.6
0.26
(Curriculum 1)
17.4
0.85
(Curriculum 2)
24.9
0.74
(Curriculum 3)
24.7
2.50
translation. Then, we append this English prefix to the Classical Tibetan translation input. Intuitively, we "scaffold" the Classical Tibetan input with partial English translation. We use a partial pre- fix of the English translation so the model doesn't t degenerate into simply outputting all the English in the input. We name this reformulation "Partial Output Scaffold English" POSE. An example of POSE is found in Figure 2. The next 4 subsections cover ablations over the finetuning reformulation setup. For direct results on the POSE task, which ended up being the most successful, see section 3.7.
Results are found in Table 2. Even though these setups have merit using human intuition, mT5 per- forms markedly worse on all of them in either per- formance, stability, both. The best performing perform better than POSE, but the cost of stability.
3.3 Modulating task difficulty
3.5 Modulating scaffold substring
The POSE reformulation is casier than the baseline task. In order modulate task difficulty, we ablate over different amounts of training examples that use this reformulation: 0% (baseline), 20%, 50%, and 100% (all reformulated).
Rather than using just prefix of the target English output, we experiment with setups that append both a portion of the target English prefix and a portion of the target English suffix ("prefix-sfffix" refor- mulation). The total selected length remains the same for the prefix+sullix experiments. The pre- fix+suffiz input reformulation is still in natural lan- guage, but using different pieces of the target out- put. Additionally, perform more fine- -reigrained sweep over how many initial training examples are reformulated.
Results are found in Table 1. The best condition involves reformulating the first 20% of training ex- amples, achieving 24.6 BLEU, 1.3 BLEU higher than the baseline. We hypothesize that making the task easy c.g. 50% or 100% reformulated makes the task less informative, which hurts down- stream performance. All of the reformulated runs have low variance across the learning rates, sug- gesting that models are better conditioned while training on easier tasks.
Results are found in Table 3. The prefix+suffix reformulation performs better and is less varied than the baseline. but performs worse than the prefix-only reformulation. We hypothesize that the prefix-only reformulation performs the best be- cause it the simplest. Over different amounts of initial training examples reformulated, 12% re- formulated had the best raw performance, closely
3.4 Optimizing the curriculum
We attempt to optimize the curriculum using hu- man intuition setups. (Curriculum 1): Instead

---

Table 4: Matching pretraining experiment results on mT5 600M with masking.
Table 5: Main results on the tib2eng translation task for mT5. Values shown are test BLEU scores. The difference shown is the improvement gained by using the input finetuning reformulations. The NLLB column is the test set BLEU score for the corresponding sized NLLB model.
Setup
BLEU
Std
Baseline
23.5
1.64
POSE
24.6
0.26
(Mask 1)
24.9
0.35
Params
NLLB
Baseline
POSE
Diff
(Mask 2)
23.6
0.20
600M
29.3
23.5
24.6
+1.1
(Mask 3)
23.0
0.15
IB
32.3
27.2
28.3
+1.1
(Mask 4)
23.4
0.04
3B
34.4
27.3
30.1
+2.8
followed by 20%. We chose to stick with the 20% experiment due to the lower variance.
baseline, mT5 POSE, and NLLB on the tib2eng task are shown in Table and Figure 3.
3.6 Matching the pretraining task
The POSE reformulation stabilizes training and improves performance, with the largest mT5 3B model exceeding the performance of NLLB 600M. Additionally, while the baseline runs have con- verged, the mT5 POSE and NLLB models could be trained further for higher performance. NLLB has strong performance on this finetuning task de- spite not being trained on Classical Tibetan. This is because NLLB was trained on modern Tibetan, similar to classical Tibetan, and because NLLB is translation-only model with strong transla- tion inductive prior. Our finetuning paradigm be- gins to bridge the gap between FLMs such as mTS,, and task specifie translation-oniy models such as NLLB.
We hypothesize that matching the pretraining task smooths performance similar to the POSE reformu- lation. We experiment on 4 masking setups: (Mask 1) mask in the first 20% of finetuning steps with p=0.1: (Mask 2) mask in the last 20% of finetun- ing steps with p=0.1; (Mask 3) mask in the last 50% of finetuning steps with p=0.25; and (Mask 4) span-mask in the last 50% of finetuning steps with p=0.25. Results are found in Table 4. Masking setups have less variance compared to the baseline previous best setup, most likely because they are closer to the pretraining task distribution. Setup (Mask 1) performs better than the POSE reformu- lation with slightly higher variance. However, retain the POSE reformulation as the best because it is simpler than setup (Mask 1). The other mask- ing setups (Mask 2), (Mask 3), and (Mask 4) result in lower performance, most likely because the task is less informative to the actual downstream trans- lation task.
Experiments on a massively multilingual translation task
4.1 Setup
The Flores200 dataset consists of around 3,000 par- allel sentences 204 different languages, meaning each sentence is translated into all 204 languages with high fidelity (NLLB-Team et al.. 2022: Goyal et al., 2021; Guzmán et al., 2019). This dataset is challenging for mT5 not only because of the sheer number of languages, but also because mT5 not pretrained on over half of the languages present in the dataset. The Flores200 dataset is purported for evaluation with a separate, partially parallel train set, but the fully parallel nature of the Flo- res200 dataset enables interesting reformulations for finetuning. We take translation pairs from the Flores200 dev set as our training set, and transla- tion pairs from the devtest set validation and test sets.
3.7 Final results and comparison to state-of-the-art
We select the best setup based stability, simplic- ity, and performance. The best reformulation was still the original POSE reformulation. We compare performance of the baseline and POSE mT55 con- ditions with the state-of-the-ee translation model NLLB (NLLB-Team et al., 2022). Because NLLB is a translation-only model, our input reformula- tions cannot be applied to it. NLLBB's s encoded input lengths are mean 26 median 19 tokens. For NLLB, We ablate over learning rates in (3e-4, 5e-4, le-3). For the NLLB tib2eng baseline, we use a lin- ear warmup of 1000 steps, 10% of the total number updates, with constant learning rate afterwards. The final results comparing the finetuning of mT5
Our reformulated Flores200 dataset for training consists of 20M train, 5k validation, and 10k test

---

Comparing
Clessical Tibelan to English
mT5 vs NLLB 600M
mT3 vs NLLB 1B
mT5 vs NLLB 3B
-0
Valication BLEU
10-
baseline mi5 18 POSE
MTB
mI5 600M POSE
mT55 38 POSE
NLLS 600M
NLLS
NLLB
1000
Figure 3: Tib2eng translation task reformulation experiment results. These results compare the mTS baseline (blue), mT5 POSE (orange), and the NLLB (green) experimental configurations. The solid lines and shaded areas are the mean and variance over learning rates, respectively. Left: 600M. Center: 1B. Right: 3B.
Flores200 translation task input reformulations
struggle in the same manner on Flores200. Even so, we begin by replicating the tib2eng POSE setup on Flores200 by appending a partial output of the target translation the input translation. We exper- iment on mT5 300M. The baseline model achieves 16.8 validation set chrF++ and the reformulated model achieves 16.7 validation set chrF++. As expected, this setup matched but did not improve upon the baseline performance.
Baseline
German: Montag haben hergestellt werden
Spanish:
El lunes, los cientificos.. dólar por cada uno.
Parallel Scaffold in English (ParSE)
German: Am Montag haben die hergestellt werden kann. Engliish Monday, scientists one U.S. cent each.
Spanish:
lunes, los cientificos.. dólar por cada uno.
mT5 has strong English performance because it was pretrained on orders of magnitude more En- glish data than other languages. So, we look leverage this strong capability in an input refor- mulation. The Flores200 benchmark consists of parallel examples of the same sentence in different languages. We extend the tib2eng POSE reformula- tion to the "Parallel Scaffold in English" or ParSE reformulation. ParSE appends full parallel En- glish translation to the input translation. For the ParSE setup, we provide the intuition that English is used as a pivot language between the two other languages.
Mixed language Parallel Scaffold (MIPS)
German to Spanish: Am Montag haben die... werden kann.
Chinese to English: [R] Brit
E.
lunes, los cientificos... dólar cada uno. On Monday, scientists... one U.S. cent each.
Figure 4: Examples of the ParSE and MiPS input refor- mulations applied to the Flores200 translation task. The changes the original input are highlighted in red.
translation pairs. Following the tokenization setup for the tib2eng task, mTS's tokenizer yields inputs of mean 52 median 46 tokens and we use a max sequence length of 256. We follow the NLLB team and perform evaluation on the Flores200 task using the chrF++ metric (Popović, 2015) with the xx-yy condition to present the final average score across languages (NLLB-Team al., 2022). We ablate over the learning rates [le-4, 2e-4, 3e-4), where we found lower learning rates to be empirically better. We train for 10k steps with a batch size of 2048 examples (approximately 105,000 tokens).
We explore the efficacy of parallel scaffolding without using English using the "Mixed-languaa Parallel Scaffold" or MiPS reformulation. MiPS appends different parallel translation to both the input and output for total of 4 distinct language translations per input. For simplicity, we use any combination of languages in Flores200, regardless if they 'r in or out of m5's pretraining distribution. Examples of the ParSE and MiPS reformulations are shown in Figures 1 and 4.
For both the ParSE and MiPS reformulations. follow the tib2eng setup and data mix of 20% baseline (less informative) and 80% reformulated (more informative) examples. We use data mix rather than reformulating the last 80% of training examples to further simplify setup and expose the
4.2 Designing task reformulations
For the tib2eng task, we designed POSE to mitigate mT5's struggles early in finetuning. mT5 does not

---

Flores2000 performance baseline, ParSC, figuratitioos
mT5 600M
mT5 18
mTS 3B
15-
Volldation
baselne mTTS 600M PoYSE
mT3 baseline mTs 18 ParSE
mT3 baseline mT5 38 ParSE
miT5 6DOM KIPS
mis MirS
m15 38 MiPS
1000
Figure 5: Flores200 translation task reformulation experiment results. These results compare the mTS baseline (blue), mT5 ParSE (orange), and mT5 MiPS (green) experimental configurations. The solid lines and shaded areas are the mean and variance over learning rates, respectively. Left: 600M. Center: 1B. Right: 3B.
Table 6: Results the Flores200 translation task for mT5. Values shown are test set chrF++ scores. The NLLB column the task performance a correspond- ing size NLLB model. For the NLLB score, use the 200 yx-yy chrF++ scores listed here.
the languages in Flores200 that were not present in its training set.
4.4 Analysis mT5's pretraining dataset and Flores200
Params
NLLB
Bascline
ParSE
MiPS
Flores200 contains 204 languages, while m'T5 was only pretrained on 95 of them. We perform addi- tional analysis on how being pretrained on lan- guage affects the post-finctuning performance Flores200,, well as how the pretraining data size for a specific language affects performance, shown in Figure Translating from a language in the pretraining set into other languages is more diffi- cult than translating from other languages into a language in the pretraining set. This is most likely because decoding into lower-resource languages is more difficult than encoding them.
600M
39.5
17.6
20.7
19.2
IB
41.5
20.3
23.8
21.6
3B
41.8
23.2
25.1
23.6
model to the input reformulations early in train- ing. The input reformulations use up to twice the number of examples per input so we reduce the per-step batch size by a factor of two from 2048 to 1024 in order to hold the data and compute budgets constant across experiments.
When translating from language in the pretrain- ing set into other languages, pretraining data size is slightly correlated with better performance. How- ever, this correlation small considering the large range of dataset sizes. The ParSE and MiPS refor- mulations improve performance across the board, not depending on pretraining data size. Using a balanced finetuning dataset like Flores200 helps mitigate some of the language frequency related pretraining biases of mT5.
4.3 Results
Our results are presented in Figure and Table 6. We observe positive effects on performance similar to the tib2eng results. For the ParSE reformulation, the model learns slightly slower initially, but learns much more over the course of training. For the MiPS reformulation, the model learns faster and better than the baseline. Clearly, our input refor- mulation scheme improves performance, beyond just relying on strong English performance. We hypothesize that both tasks successfully improve performance, in part because they allow for direct attention between the input context in different lan- guages, aligning representations across languages. Interestingly, the ParSE reformulation performs the best, but also has the highest variance over the learning rates. The need for lower learning rates typically indicates poor conditioning, so the input task is likely more ill-conditioned than the bascline. One possible explanation that mT5 is learning
The performance improvement using ParSE when translating from English into other languages is much more pronounced. This can be seen visu- ally in Figure 6 for the rightmost datapoint in each plot in the top row. The corresponding numbers in Table 7 for 3B models shows the increase for from- nom-English is 6.3 chrF++. This makes intuitive sense since the model has seen significantly more English in the input during finetuning.
We break down the performance of different model sizes and reformulation setups in Table 7.

---

Pretrain dataset size chri+++
mT5 600m, translation from
m15 1b, translation from
mT5 3b, translation from
40
basnline ParSt
baseline
Pwriit
30
20
baseline
10:
ParSE
1b, Irarelation inte
translation
ParSE
35
MIPS
25
15
baseline ParSE
ParSE
5: 10r 204 10
NIPS
MIFS
20
104
10
10
104 10'
106
Pretrain dataset examples
Figure 6: Pretraining dataset sizes and Flores200 finetuning performance. The first row represents translation from a language in the pretraining set into other languages, including those not the pretraining set. The second row represents translation from other languages into language present the pretraining set. Each dot represents one language and the value in the graph represents the corresponding chrF4+ test set score for that language and model. Points shown only cover languages present in the mT5 pretraining set. The point corresponding to English is the rightmost point all the graphs. Dataset sizes are calculated using the number of examples of each language present in the mC4 dataset. Dataset sizes range from 100k to IB examples.
Interestingly, the ParSE and MiPS reformulations improve performance involving lower-resoune lan- guages, sometimes a slight cost to performance on higher resource languages. For example, the 3B baseline and ParSE conditions perform about the same when translating from languages in the pretrain dataset to other languages in the pretrain dataset. The ParSE condition performs 1.3 chrF++ worse than the baseline when translating from out- pretrain to in-pretrain languages. However, the ParSE condition performs significantly better than the baseline condition on the in-out and out-out language pairs, with chrF++ improvements of 5.3 and 3.6 respectively. Explanations for this requires further targeted experimental investigations.
conditioned training curves and significantly better performance. These input reformulations are simple to understand and implement, robust over hyperparameters, general to translation tasks, and effective. We hope our technique is used to accessibly improve data efficiency on translation tasks.
Limitations
Our proposed technique has only been applied to two challenging translation tasks, where the input and output are both information rich and sequential in nature. Mechanically, these ideas be applied to other tasks such as sequence classification. Intu- itively, doing so would enable the model attend to multiple inputs in its input context in order better denoise the inputs. This allows the model to learn more effectively. Similar techniques can be applied to other tasks, even explored further in pretraining (Lample and Conneau, 2019).
5 Conclusion
We have explored how FLMs learn from their in- put contexts. We provide two separate techniques that can be applied to any translation use case. For the case of single language pair transla- tion task, we recommend POSE. For the case of a multi- langual pair translation task, we recom- mend ParSE and MiPS. For challenging translation tasks, our scaffolding reformulations produce bet-
The baseline model used here m115, a rel- atively old FLM. As a result, our baseline results are low compared to state-of-the-art NLLB results. Unfortunately, there are no better FLMs in the pa- rameter ranges from 600M to 3B. We believe there

---

is still much to explore here with better FLMs, larger parameter counts, and other creative reformu- lations. We believe that FLMs will eventually out- perform translation-onle models like NLLB, due to the flexibility given by the capability to understand inputs. The input reformulations presented in this paper, which begin to bridge the performance gap between NLLB and mT5, are one example of how FLMs are more flexible in various input contexts.
Eyal Ben-David, Nadav Oved, and Roi Reichart. 2022. Pada: Example-bar based prompt learning for on-the-fiy fly adaptation to unseen domains.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin. Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alee Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
Ethics Statement
As with all work today in deep learning and large models, there are many biases introduced during large data pretraining and finetuning. We did our best choose datasets and models which acknowl- edge and attempt to mitigate these biases as much as they can, and encourage the development of even better datasets and models in the future. Be- cause the techniques introduced in this paper are input reformulations that don't 't introduce new data, we believe they are at least not introducing many additional risks, and are generally safe to intro- duce to other models and techniques. Addition- ally, one surprising outcome of our work is that heavy language-orienter pretraining biases were mitigated by finetuning on a language-balanced dataset. This critical for equity with regards to multilingual applications of language models.
Xiang Chen, Ningyu Zhang, Xin Xie, Shumin Deng, Yunzhi Yao, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. 2022. KnowPrompt: Knowledge- aware prompt-tuning with synergistic optimization for relation extraction. In Proceedings of the ACM Web Conference ACM.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsyyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengeheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev. Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal. Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean. Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways.
We believe the priority of ethics in this line of research is ensure that the future integration of these technologies into society as safe, ethical, and trustworthy. High quality training is critical. Un- derstanding how different inputs affect downstream performance is an important stepping stone. We encourage further research in this direction to im- prove model understanding and control.
Furthermore, aim to increase accessibility of high quality, task-specific, and compute friendly large language models by improving data effi- ciency.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus. Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Al- hert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdh- ery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Ja- cob Devlin, Adam Roberts, Denny Zhou, Quoe Le, and Jason Wei. 2022. Scaling instruction -inntiio---inuneed language models.
Acknowledgements
We would like to thank Prof. Kurt Keutzer for his
wisdom and hardware.
References
Robert L. Logan IV au2, Ivana Balažević, Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel. 2021. Cutting down on prompts and parameters: Simple few-shot learning with language models.
Kevin Clark, Minh-Tharg Luong, Quoe V. Le, and Christopher D. Manning. 2020. Electra: Pre-training text encoders as discriminators rather than genera- tors.

---

Jacob Devlin, Ming-Wa Mei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing.
Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Si- monyan, Erich Elsen, Jack W. Rac, Oriol Vinyals, and Laurent Sifre. 2022. Training compute-optimal large language models.
Dong. Nan Yang. Wenhui Wang. Furu Wei, Xi- aodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao- Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In Advances in Neural Information Pro- cessing Systems, volume 32, Curran Associates, Inc.
Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru. Todor Mihaylov, Daniel Simig. Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, Xian Li, Brian O' O'Horo, Gabriel Pereyra, Jeff Wang. Christo- pher Dewan, Asli Celikyilmaz, Luke Zettlemoyer, and Ves Stoyanov. 2023. Opt-iml: Scaling language model instruction meta learning through the lens of generalization.
Marzich Fadacc, Arianna Bisazza, and Christof Monz. 2017. Data augmentation for low-resource neural machine translation. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics Volume Short Pupers), pages 567-573, Vancouver, Canada. Association for Computational Linguistics.
Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know?
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners.
Tom Koemi and Ondrej Bojar. 2017. Curriculum learn- ing and minihatch bucketing in neural machine trans- lation. In RANLP 2017 Recent Advances in Natural Language Processing Meet Deep Learning. Incoma Ltd. Shoumen, Bulgaria.
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng- Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Kr- ishnan, Mare' Aurelio Ranzato, Francisco Guzman, and Angela Fan. 2021. The 10res-101 evaluation benchmark for low-resource and multilingual ma- chine translation.
Guillaume Lample and Alexis Conneau. 2019. Cross- lingual language model pretraining.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuming.
Jiatao Gu, Yong Wang. Yun Chen, Kyunghyun Cho, and Victor O. K. Li. 2018. Meta-learning for low- resource neural machine translation.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019, Bart: De- noising ggquence-to-eqqenee pre-training for natural language generation, translation, and comprehension.
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2023, Pre-training to learn in context.
Francisco Guzmán, Peng-Jen Chen, Myle Oit, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc' Aurelio Ranzato. 2019. The flores evaluation datasets for low-resource machine translation: Nepali-english and sinhala-english.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation.
Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. 2021. Warp: Word-level adversar- ial reprogramming.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021. What makes good in-context examples for gpt-3?
Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun. 2021. Ptr: Prompt tuning with rules for text classification.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer. and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach.
Adi Haviv, Jonathan Berant, and Amir Globerson. 2021. BERTese: Learning to speak to BERT. In Proceed- ings of the 16th Conference of the European Chap- ter of the Association for Computational Linguistics: Online. Association
Ilya Loshchilov and Frank Hutter. 2019, Decoupled weight decay regularization.
Main Volume, pages for Computational Linguistics.
Yao Lu, Max Bastolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few- shot prompt order sensitivity.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: Decoding-enhant enhanned bert with disentangled attention.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai. Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Han- naneh Hajishirzi. 2022. Metaicl: Learning to learn context.

---

ILLB-Team. Marta R. Costa-jusså, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Hef- fernan, Elahe Kalbassi, Janice Lam, Daniel I. Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Bar- rault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Sehwenk, and Jeff Wang- 2022. No language left behind: Scaling human- centered machine translation.
Timo Schick, Helmut Schmid, and Hinrich Schütze. 2020. Automatically identifyngg words that can serve as labels for few-shot text classification.
Timo Schick and Hinrich Schütze. 2021. Exploiting cloze questions for few shot text classification and natural language inference.
Rico Senarich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 86 86-96, Berlin, Germany, Association for Computational Lin- guistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 31 1-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV au2, Eric Wallace, and Sameer Singh. 2020. Auto- prompt: Eliciting knowledge from language models with automatically generated prompts.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro. 2020. Megatron-Im: Training multi-billion parameter language models using model parallelism.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, An- ton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. 2019. Language models knowl- edge bases?
Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. 2019. Meta-weight- net: Learning an explicit mapping for sample weight- ing.
Emmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig. Barnabas Poczos, and Tom M. Mitchell. 2019. Competence-besd curriculum learning for neural machine translation.
Yu Sun, Shuohuan Wang. Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. Ernic: Enhanced represen- tation through knowledge integration.
Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. Proceedings of the Tenth Workshop on Statisticul Machine Translation, pages 392-395, Lisbon, Portugal. Association for Computational Linguistics.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An cnaructioo-oollowwiin 1laaa model. https: // gi www.ttc..tttteeeeeeeee tat.ttttsssaabbataaaaaa wtwaaaaaatsaodaaaalaaoaooaa
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. 2023. Measuring and narrowing the compositionality gap language models.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez. Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models.
Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying Ims with mixtures of soft prompts.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language under- standing by generative pre-training.
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gard- ner, and Sameer Singh. 2021. Universal adversarial triggers for attacking and analyzing nlp.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Boshi Wang, Xiang Deng, and Huan Sun. 2022a. Itera- tively prompt pre- pre-trained language models for chain of thought. In Proceedings the 2022 Conference Empirical Methods in Natural Language Processing. pages 27 2714-2730, Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023a. Self-consistency improves chain of thought reasoning in language models.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. 2019. Learning to reweight examples for robust deep learning.

---

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023b. Self-instruct: Aligning language models with self-generated instructions.
Ruiqi Zhong, Kristy Lee, Zheng Zhang, and Dan Klein. 2021a. Adapting language models for zero-shot learning by meta-tuning on dataset and prompt col- lections.
Yizhong Wang, Swaroop Mishra, Pegah Alipoor- molabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Is- han Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Maitreya Patel, Kuntal Ku- mar Pal, Mehrad Moradshahi, Mihir Parmar, Mi- rali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Shailaja Keyur Sampat, Savan Doshi, Siddhartha Mishra, Sujan Reddy, Sumanta Patro, Tanay Dixit, Xudong Shen, Chitta Baral, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi, and Daniel Khashabi. 2022b. Super-naturaie naturalinstructtin::: Generalization via declarative instructions on 1600+ nlp tasks.
Zexuan Zhong, Dan Friedman, and Danqi Chen. 2021b. Factual probing is [MASK]; Learning vs. learning to recall. In Proceedings the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 5017-5033, Online. Association for Computational Linguistics.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang. Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoe Le, and Ed Chi. 2023a. Least-to-mos prompting enables complex reasoning in large language models.
Yongchao Zhou, Andrei loan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023b. Large language models are human-level prompt engineers.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du An- drew M. Dai, and Quoe V. Le. 2022. Finetuned language models are zero-shot learners.
A Appendix
A.1 Flores200 in- and out- pretrain results
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Tchter, Fei Xia, Chi, Quoe Le, and Denny Zhou. 2023. Chain-of-thought prompting elic- its reasoning in large language models.
BigScience Workshop. 2023. Bloom: A 176b- parameter open-access multilingual language model.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami A1-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mt5: massively multilingual pre-trained text-to-text transformer.
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Car- bonell, Ruslan Salakhutdinov, and Quoc Le. 2020. Xlnet: Generalized autoregressive pretraining for lan- guage understanding.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mi- haylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022a. Opt: Open pre-trained transformer language models.
Xuan Zhang, Gaurav Kumar, Huda Khayrallah, Kenton Murray, Jeremy Gwinnup, Marianna J Martindale, Paul McNamee, Kevin Duh, and Marine Carpuat. 2018. An empirical exploration of curriculum learn- ing for neural machine translation.
Xuan Zhang, Pamela Shapiro, Gaurav Kumar, Paul Mc- Namee, Marine Carpuat, and Kevin Duh. 2019. Cur- riculum learning for domain adaptation in neural ma- chine translation.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022b. Automatic chain of thought prompt- ing in large language models.

---

Table 7: Breakdown of model and setup performance over different splits of the Flores200 dataset. "In" refers to language that was found the mT5 pretraining dataset and "out" refers a language that was not. "To Eng" and "From Eng" is refererd to xX-eng and eng-xx in some other papers, respectively. Notably, the proposed techniques improve "To Eng" performance up to 4.2 chrF++ and "From Eng" performance up to 9,4 chrF++, in the 600M case. We hypothesize this difference in improvement is due the finetuning task including more English examples in the input, helping with downstream English translations as well as other language translations.
Params
Setup
In-in
Out-in
In-out
Out-out
To Eng
From Eng
Avg
600M
Baseline
20.5
19.2
17.2
16.4
21.2
20.2
17.6
ParSE
24.5
21.1
21.2
18.7
25.4
29.6
20.7
MiPS
22.6
20.5
19.1
17.7
23.9
22.8
19.2
IB
Baseline
28.3
23.6
17.1
15.2
33.8
24.6
20.3
ParSE
30.9
25.2
22.7
19.3
34.6
32.9
23.8
MiPS
27.8
23.6
19.9
17.7
31.3
25.8
21.6
3B
Bascline
33.2
27.3
19.3
16.9
41.0
29.0
23.2
ParSE
33.0
26.0
24.6
20.5
37.9
35.3
25.1
MiPS
30.5
25.5
22.3
19.5
34.8
28.8
23.6


Reviews:
[{"review_id": "xohhDAt897", "reviewer": "Reviewer_wsZB", "summary": "", "questions": "1. How do you get the partial translation (in POSE) or guidance translations (in ParSE and MiPS) during inference? Do you first use the base model to generate these prefixes or guidance translations, then refine the translation by reformatting the inputs?\n2. In cases of ParSE and MiPS, the choice of languages for the guidance translations can be arbitrary. How sensitive the model is to different choices of the guidance language?\n3. It seems POSE can also be applied to multilingual translation tasks, and ParSE/MiPS can also be applied to the single language pair translation tasks like tib2eng. I wonder why you dedicate POSE to the single language pair case, and ParSE/MiPS to multilingual cases?\n4. Are the proposed techniques more helpful in improving xx->En or En->xx translation quality?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to thoughtfully review our paper and provide feedback. We hope our suggested edits included in the responses below demonstrate our appreciation and diligence and further demonstrate the quality of this paper. \n\n\n## Response to “Paper Topic and Main Contributions”\n> “This paper proposes three novel input formats for finetuning translation models. These formats adds \"guidance\" partial or guidance translations to the inputs, which makes it easier for the model to generate real translations. Experimental results on Flores200 and a challenging Tibenten-to-English task shows the effectiveness of the proposed approach.”\n\nGreat summary! To clarify, the guidance translations are applied only during finetuning and during evaluation. Additional clarification is provided in the questions responses below. \n\n\n## Response to “Questions for the Authors”\n> “How do you get the partial translation (in POSE) or guidance translations (in ParSE and MiPS) during inference?”\n\nPOSE, ParSE, and MiPS are only applied during finetuning. There is no partial or guidance translation during inference. The evaluation task consists of just the source and target translations (see Section 3.6). All setups (baseline or reformulated) use identical evaluation tasks. The main result of the paper is that the inclusion of these techniques during finetuning improves translation performance during evaluation, where there is no partial or guidance translation. \n\nDuring revisions, the Abstract will be improved from\n> ... These reformulations are simple data level modifications, require no additional data, and …\n\nto\n\n> ... These reformulations are simple data level modifications, require no additional training data or data at inference time, and …\n\nDuring revisions, the Introduction paragraph 2 will be improved from\n> ... We see qualitative improvements in the variance of the training curves, and significant increases in overall translation performance up to **10.3% / 2.8 BLEU**.\n\nto\n\n> We see qualitative improvements in the variance of the training curves. When evaluated on the same test set with no reformulations, POSE significantly increases overall translation performance compared to the direct finetuning baseline, **up to 10.3% / 2.8 BLEU**.\n\n***\n\n> “Do you first use the base model to generate these prefixes or guidance translations, then refine the translation by reformatting the inputs?”\n\nTranslation examples typically consist of a source and target translation pair. **The prefixes are simply a prefix of the target translation.** There is no prefix or guidance translation generation. \n\nDuring revisions, the Introduction paragraph 2 will be improved from\n> ... POSE appends a prefix of the target English output to the Classical Tibetan input. ...\n\nto \n\n> ... Tib2eng translation examples consist of a Classical Tibetan source and English target translation pair. POSE simply appends a prefix of the target English output to the Classical Tibetan input. ...\n\nDuring revisions, the Introduction paragraph 5 will be improved from\n> ... ParSE appends a full parallel English translation to the input. …\n\nto\n\n> ... ParSE appends the corresponding full parallel English translation (provided by Flores200) to the input. …\n\n***\n\n> “In cases of ParSE and MiPS, the choice of languages for the guidance translations can be arbitrary. How sensitive the model is to different choices of the guidance language?”\n\nParSE and MiPS can be seen as two extremes for choices of guidance languages. MiPS uniformly randomly samples all languages present in the Flores200 dataset, with no single dominating guidance language. MiPS improves translation performance up to 1.6 chrF++, suggesting that many choices of the guidance language would improve performance. ParSE uses English as the only guidance language. ParSE generally performed better than MiPS with improvements up to 2.8 chrF++. We conclude that using guidance languages with a larger pretraining token count correlates to better performance and the **model is somewhat sensitive to different choices of the guidance language**. \n\n***\n\n> “It seems POSE can also be applied to multilingual translation tasks, and ParSE/MiPS can also be applied to the single language pair translation tasks like tib2eng. I wonder why you dedicate POSE to the single language pair case, and ParSE/MiPS to multilingual cases?”\n\nThe motivation for POSE is discussed in Section 3.2. When finetuned on the tib2eng translation task, mT5 struggles in the initial finetuning steps. POSE is specifically designed to mitigate these struggles. In Section 4.2, we investigate POSE applied to the Flores200 task. mT5 300M achieves 16.8 chrF++ on the baseline Flores200 task, with no struggles in the initial finetuning steps, unlike the tib2eng task. Therefore, we didn't think that POSE would be successful at the Flores200 task. **Our results show that applying POSE to the Flores200 task yields 16.7 chrF++, no change to the performance, as expected.**\n\nEven though POSE specifically did not change Flores200 performance, the finetuning input reformulation idea was generalized to produce the ParSE and MiPS reformulations. These reformulations require at least 3 languages in the translation task. **As a result, the ParSE and MiPS reformulations cannot be applied to single language pair translation tasks like tib2eng.**\n\nAs stated in the abstract and the final Introduction paragraph, we provide two separate techniques that can be applied to any translation use case. For the case of a single language pair translation task, we recommend POSE. For the case of a multi-language pair translation task, we recommend ParSE and MiPS. \n\nDuring revisions, the Conclusion will be improved from\n> ... We have explored how FLMs learn from their input contexts. …\n\nto\n> ... We have explored how FLMs learn from their input contexts. We provide two separate techniques that can be applied to any translation use case. For the case of a single language pair translation task, we recommend POSE. For the case of a multi-language pair translation task, we recommend ParSE and MiPS. …\n\n***\n\n> “Are the proposed techniques more helpful in improving xx->En or En->xx translation quality?”\n\nAs reported in Table 7 (denoted as \"from Eng\" and \"to Eng\" in the table), the proposed techniques improve xx -> En performance up to **4.2 chrF++** and En -> xx performance up to **9.4 chrF++**. The proposed techniques are more helpful in improving En -> xx translation quality. \n\n## Response to “Typos Grammar Style And Presentation Improvements”\n> “The paper conflates methodology description and introduction in Sec. 1. It would be great to introduce the motivation and intuition in Sec. 1 first, and dedicate methodological descriptions in a separate section.”\n\nThe Introduction largely consists of motivation and intuition, with select methodological descriptions of the POSE, ParSE, and MiPS reformulations to **emphasize their simplicity**. Additional methodological details are described in separate sections e.g. Sections 3.2 and 4.2. During revisions, we will consider ways to make the intuition behind the reformulations more clear in the introduction. \n\n\n## Response to “Reasons to Reject”\n> “Deeper analysis and insights is preferred.”\n\nSection 3.7 includes analysis on finetuning loss curves and a preliminary comparison of general FLM and translation-only model performance. The POSE reformulation and the SOTA translation-only model setups can be further finetuned for increased performance. \n\nSection 4.4 includes analysis on mT5’s pretraining dataset to explain the improved performance of the ParSE and MiPS reformulations. Critically, we find that these reformulations equitably improve performance on low and high resource pretraining languages. We postulate that using a balanced training dataset like Flores200 can mitigate language frequency related biases in mT5. \n\nSection 4.4 includes analysis on encoding and decoding into different languages. We find that the ParSE and MiPS reformulations improve mT5 performance on the hardest setting of translation into languages that were not present in mT5’s pretraining language spread. \n\n***\n\n> “Some technical details of the proposed approach can be provided.”\n\nSections 3.1 and 4.1 include comprehensive descriptions of the finetuning and evaluation setups. Sections 3.2 and 4.2 include comprehensive descriptions of the input reformulations. \n\n***\n\nThank you again for your thoughtful comments! We hope all of your questions and concerns were addressed here. Looking forward to hearing if there are further points we can clarify or improve.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed approach is novel, simple and appears to be effective in improving MT quality.", "reasons_to_reject": "1. Structure and writing of the paper can be improved.\n2. Deeper analysis and insights is preferred.\n3. Some technical details of the proposed approach can be provided.", "questions_to_authors": "1. How do you get the partial translation (in POSE) or guidance translations (in ParSE and MiPS) during inference? Do you first use the base model to generate these prefixes or guidance translations, then refine the translation by reformatting the inputs?\n2. In cases of ParSE and MiPS, the choice of languages for the guidance translations can be arbitrary. How sensitive the model is to different choices of the guidance language?\n3. It seems POSE can also be applied to multilingual translation tasks, and ParSE/MiPS can also be applied to the single language pair translation tasks like tib2eng. I wonder why you dedicate POSE to the single language pair case, and ParSE/MiPS to multilingual cases?\n4. Are the proposed techniques more helpful in improving xx->En or En->xx translation quality?\n", "paper_topic_and_main_contributions": "This paper proposes three novel input formats for finetuning translation models. These formats adds \"guidance\" partial or guidance translations to the inputs, which makes it easier for the model to generate real translations. Experimental results on Flores200 and a challenging Tibenten-to-English task shows the effectiveness of the proposed approach.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The paper conflates methodology description and introduction in Sec. 1. It would be great to introduce the motivation and intuition in Sec. 1 first, and dedicate methodological descriptions in a separate section."}}, {"review_id": "cl3D1ae5Pk", "reviewer": "Reviewer_V5ez", "summary": "", "questions": "Question A: Are these methods more suitable for low-resource languages? Does it still work for high-resource languages?\nQuestion B: POSE works only for tibeten2eng, while ParSE works for Flores200. Which one is more generalized or in what condition to use which one?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to thoughtfully review our paper and provide feedback. We hope our suggested edits included in the responses below demonstrate our appreciation and diligence and further demonstrate the quality of this paper. \n\n## Response to “Paper Topic And Main Contributions” and “Reasons To Accept”\nThese sections exhibit perfect understanding of the paper’s contributions. Great summary!\n\n## Response to “Questions For The Authors”\n> “Question A: Are these methods more suitable for low-resource languages? Does it still work for high-resource languages?”\n\nFrom Section 4.4, we find that the balanced nature of the Flores200 dataset helps to mitigate some of the pretraining language-size bias of mT5, more equitably improving performance over low and high resource languages. Furthermore, in Section 4.4 and Figure 6, we find that the ParSE and MiPS reformulations significantly improve performance translating from higher resource in-pretrain languages across the board, up to 5.3 chrF++. These reformulations result in slight degradations translating into the same higher resource languages, up to 1.3 chrF++ on ParSE. As reported in Table 7, the proposed techniques improve high-resource English performance up to 4.2 chrF++ when translating into English and up to 9.4 chrF++ when translating from English. \n\n***\n\n> “Question B: POSE works only for tibeten2eng, while ParSE works for Flores200. Which one is more generalized or in what condition to use which one?”\n\nGreat question! As stated in the abstract and the final Introduction paragraph, we provide two separate techniques that can be applied to any translation use case. For the case of a single language pair translation task, we recommend POSE. For the case of a multi-language pair translation task, we recommend ParSE and MiPS. \n\nWhen this paper is accepted, the Conclusion will be improved from\n> ... We have explored how FLMs learn from their input contexts. …\n\nto\n\n> ... We have explored how FLMs learn from their input contexts. We provide two separate techniques that can be applied to any translation use case. For the case of a single language pair translation task, we recommend POSE. For the case of a multi-language pair translation task, we recommend ParSE and MiPS. …\n\n## Response to “Reasons to Reject”\n> “However, there is no comparison to directly finetune the LLMs.”\n\nTo clarify, the “baseline” condition referenced in the paper is the direct finetune condition, as described in Sections 3.2 and 4.2. Compared to this baseline direct finetune condition, our input reformulations improve performance up to 2.8 BLEU on the Classical Tibetan to English task and up to 3.5 chrF++ on the Flores200 task. \n\n***\n\n> “And the best results in this paper have a big gap to the normal NMT systems.”\n\nThis concern is addressed in the Limitations section, paragraph 2. If there were a stronger multilingual FLM to use in the parameter ranges from 600M to 3B parameters, the authors would have used them. This parameter range was selected due to hardware, compute, and accessibility constraints. Additionally, the SOTA translation model NLLB is a translation-only (non-FLM) model that our input reformulations cannot be applied to. Critically, the authors believe that the gap between translation FLMs like mT5 can reach SOTA translation-only model performance through training techniques like ParSE and MiPS. As stated in the Introduction, these techniques are specifically designed for FLMs due to their popularity and ubiquity. \n\n***\n\n> “The paper proposed three data techniques. It is not very clear in what condition to use each one.”\n\nPlease see the response to the question “Question B: POSE works only for tibeten2eng, while ParSE works for Flores200. Which one is more generalized or in what condition to use which one?”\n\n## Response to “Typos Grammar Style And Presentation Improvements”\n> “Avoid using complex vocabulary in writing and focus on precise and accurate expression in English.”\n\nCan you please provide some concrete examples?\n\n***\n\nThank you again for your thoughtful comments. We hope all of your concerns were addressed here. Looking forward to hearing if there are further points we can clarify or improve.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper introduces three data-efficient techniques for improving translation performance during fine-tuning based on foundation language models. The paper highlights the importance of understanding how inputs affect downstream performance, which will improve model understanding and control. \nThe paper aims to increase accessibility to high-quality, task-specific, and compute-friendly large language models by improving data efficiency.", "reasons_to_reject": "- The effect of the proposed methods is not very persuasive. The proposed data strategies improved the translation results by fine-tuning on foundation language models. However, there is no comparison to directly finetune the LLMs. And the best results in this paper have a big gap to the normal NMT systems. \n\n- The paper proposed three data techniques. It is not very clear in what condition to use each one.", "questions_to_authors": "Question A: Are these methods more suitable for low-resource languages? Does it still work for high-resource languages?\nQuestion B: POSE works only for tibeten2eng, while ParSE works for Flores200. Which one is more generalized or in what condition to use which one?", "paper_topic_and_main_contributions": "This paper is about the reformulation of inputs during fine-tuning for challenging translation tasks using pre-trained language models .  The authors propose three different data-efficient techniques to improve translation performance on multilingual FLMs, here is mT5, during finetuning. These techniques are the \"Partial Output English Scaffold\" (POSE), the \"Parallel Scaffold in English\" (ParSE), and the \"Mixed-language Parallel Scaffold\" (MiPS).\n\"Partial Output English Scaffold\" (POSE) reformulation appends a prefix of the target English output to the input, and the \"Parallel Scaffold in English\" (ParSE) reformulation appends a full parallel English translation to the input. The \"Mixed-language Parallel Scaffold\" (MiPS) reformulation appends a different parallel translation to both the input and output for a total of 4 distinct languages per input. \nThe experiments are performed on the Flores200 translation benchmark and a Tibetan to English", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "-", "Typos_Grammar_Style_and_Presentation_Improvements": "- Avoid using complex vocabulary in writing and focus on precise and accurate expression in English."}, "extra_text": {}}, {"review_id": "1jNjcz1ERn", "reviewer": "Reviewer_Mb7f", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper. We would love to respond to any specific feedback, but there was not any provided. If you could give some examples of exactly how you misunderstood the paper, we are more than happy to have a discussion about that.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is an organized paper.", "reasons_to_reject": "- Section 2 lists numerous related works but the idea of this work is still not clear.\n- Sections 3 and 4 are \"experiments on a ... task\", which make the entire work an experimental report than a scientific paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "It is difficult to say. This paper lists many related works and reports various experiments.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces three data input formulation approaches aimed at enhancing translation performance during the fine-tuning process.

**Pros**:
- The proposed techniques are straightforward and consistently demonstrate effectiveness for both high-resource and low-resource languages.
- The authors conducted exhaustive experiments to understand why and when the proposed data reformulation techniques work.

**Cons**: The only drawback of this work is that when fine-tuning large foundation models, it still falls short of the performance achieved by dedicated multilingual translation models such as NLLB. Additionally, the proposed technique is not compatible for models like NLLB. Nevertheless, I believe this work retains its significance given the prevalent practice of fine-tuning language models with instructional input. The authors have adeptly addressed other queries and concerns raised by the reviewers.
Decision: Accept-Main