Scaling Laws for Downstream Task Performance in Machine Translation
Authors: Berivan Isik
Natalia Ponomareva
Hussein Hazimeh
Dimitris Paparas
Sergei Vassilvitskii
Sanmi Koyejo
Venue: ICLR 2025 Poster (2025)
Keywords: scaling laws
transfer learning
machine translation
large language models
data valuation
Primary area: foundation or frontier models, including LLMs
TL;DR: We study the scaling behavior of the downstream translation metrics as the pretraining data grows and propose scaling laws for COMET and BLEU scores, and downstream cross-entropy.

Abstract: Scaling laws provide important insights that can guide the design of large language models (LLMs). Existing work has primarily focused on studying scaling laws for pretraining (upstream) loss. However, in transfer learning settings, in which LLMs are pretrained on an unsupervised dataset and then finetuned on a downstream task, we often also care about the downstream performance. In this work, we study the scaling behavior in a transfer learning setting, where LLMs are finetuned for machine translation tasks. Specifically, we investigate how the choice of the pretraining data and its size affect downstream performance (translation quality) as judged by: downstream cross-entropy and translation quality metrics such as BLEU and COMET scores. Our experiments indicate that the size of the finetuning dataset and the distribution alignment between the pretraining and downstream data significantly influence the scaling behavior. With sufficient alignment, both downstream cross-entropy and translation quality scores improve monotonically with more pretraining data. In such cases, we show that it is possible to predict the downstream translation quality metrics with good accuracy using a log-law. However, there are cases where moderate misalignment causes the downstream translation scores to fluctuate or get worse with more pretraining, whereas downstream cross-entropy monotonically improves. By analyzing these, we provide new practical insights for choosing appropriate pretraining data.

Published as a conference paper at ICLR 2025
SCALING LAWS FOR DOWNSTREAM TASK PERFORMANCE IN MACHINE TRANSLATION
Berivan Isik Natalia Ponomareva** * Hussein Hazimeh 9 Dimitris Paparas*
Sergei Vassilvitskii Sanmi Koyejo
Google Research, OpenAI, SStanford University
serivan@gooogee.com
ABSTRACT
Scaling laws provide important insights that can guide the design of large language models (LLMs). Existing work has primarily focused on studying scaling laws for pretraining (upstream) loss. However, in transfer learning settings, in which LLMs are pretrained on an unsupervised dataset and then finetuned on a downstream task, often also care about the downstream performance. In this work, we study the scaling behavior in a transfer learning setting, where LLMs are finetuned for machine translation tasks. Specifically, investigate how the choice of the pretraining data and its size affect downstream performance (translation quality) judged by: downstream cross-entropy and translation quality metries such as BLEU and COMET scores. Our experiments indicate that the size of the finetuning dataset and the distribution alignment between the pretraining and downstream data significantly influence the scaling behavior. With sufficient alignment, both downstream cross-entropy and translation quality scores improve monotonically with more pretraining data. In such cases, we show that it is possible to predict the downstream translation quality metrics with good accuracy using a log-law. However, there are cases where moderate misalignment causes the downstream translation scores to fluctuate or get worse with more pretraining, whereas downstream cross-entropy monotonically improves. By analyzing these, we provide new practical insights for choosing appropriate pretraining data.
INTRODUCTION
Scaling laws quantify the relationship between model's performance and key design factors such as the size of the training data or the model's architecture. In the context of LLMs, these laws offer valuable guidance for model development, resource allocation, and selection of appropriate training data. Extensive research has focused on scaling laws for upstream perplexity or tross-entrope loss (i.e., evaluated on pretraining data), demonstrating that these quantities can be well predicted using power laws (Kaplan et al. 2020; Hoffmann et al. 2022] Gordon et al.) 2021 Hernandez et al. 2022
Fernandes et al.] [203] Henighan et al. [2020] Johnson al. 2018). However, in practice, LLMS often undergo transfer lcarning-ther they are first pretrained on unsupervised data and then finetuned for specific downstream" tasks such as coding or translation. The question of whether scaling laws can be used to predict downstream task performance is critical ()penAA|2024) yet remains largely unanswered (Hernandez al.] 20221: Tay et al., (02T). Here, the term task performance refers to metrics that measure task-related quantities such as accuracy and translation scores like BLEU, ROUGE, or COMET, which are different from next-token prediction metrics such as cross-entropy.
In this work, we study scaling laws for transfer learning and focus on machine translation tasks. Specifically, we look into the relation between the pretraining dataset size and the downstream task performance after finetuning the task. We find that, in addition to the finetuning data size and the
Work done when We use the term downstream to refer to the finetuning task or metrics computed on it, and the term upstream refer the metries computed the pretraining dataset.
authors were at Google.

---

Published as a conference paper at ICLR 2025
choice of the performance metric, this relation fundamentally depends on the alignment between the pretraining data and the downstream task (based on the translation alignment score provided in Section [.). While similar observations have been made in different contexts in the transfer learning literature TTmmkin et al., 2020, Agostinelli et al., 2022), our work provides new insights and concrete scaling laws for the downstream performance in machine translation.
We carry out systematic experiments in which we pretrain LLMs on multilingual unsupervised datasets and then finetune them on several machine translation tasks. Across the experiments, we vary the type of pretraining data (to control the degree of distribution a alignment with the downstream task) and the finetuning data size. We study the following metries: downstream BLEU score (Papineni et al. 2002) downstream ROUGE score (Lin) (204). downstream COMET score (Rei et al.) 2020; Stewarr et al. 2020, Rei et al. 2 and downstream cross-entropy. We find that in settings where the distributions are well-aligned, both the translation scores and downstream cross-entropy improve monotonically with more pretraining (see Figure I. orange curves). In these settings, we demonstrate that the translation scores (e.g., BLEU, ROUGE, and COMET) can be well predicted using the following log-law: f(pp) = (log(A (A- D2))², where Dr denotes the size of the pretraining data, and A, a, B are the coefficients to be fit. We further propose a power-law (Dp)) = F + A for the downstream De
cross-entropy as the pretraining data scales - echoing similar laws developed for the upstream cross- entropy a function of the pretraining dataset size (Kaplan et al., 2020; Hoffmann et al., 2022) and downstream cross-entropy as a function of the finetuning dataset size (Hernandez et al. 2021).
However, when distributions are not sufficiently aligned and the finetuning data size is relatively small, we find that there are cases where the translation scores exhibit an unclear, non-monotonic behavior, whereas the downstream cross-entropy still improves monotonically following power-law This observation suggests that using cross- coossoy as a proxy for task-related metrics like BLEU, ROUGE, or COMET scores may lead to critical misjudgments in practice if used make decisions about the "relevance" of the pretraining data for the downstream task or the required size of the pretraining data for the target downstream performance. Finally, our empirical studies suggest that pretraining brings little to no improvement on the translation quality when the finetuning (translation) dataset is already large enough, complementing the findings of Hernandez et al. 2021
Our contributions and main findings can be summarized as:
We carry systematic experiments on 770-million and 3-billion encoder-decoder TS (Raffel et al.] 2020) models to study how downstream performance, measured by downstream cross-entropy and translation scores, scales with the pretraining dataset size. For pretraining, we experiment with different subsets of the Multilingual C4 (MC4) dataset (Raffel et al. 2020), including English (en), German (de), French (fr), and Romanian (ro). For finetuning, we study the following translation tasks: WMT-17 en-de (Bojar et al.) 2017). WMT-15 en-fr (Bojar et al. 2014), and 1M en-ro (Bojar et al. 2016).
We observe that, when the distributions of the pretraining and downstream tasks are well- aligned, both the translation scores and downstream cross-entropy improve monotonically with more pretraining (Figure [ orange curves). For BLEU, ROUGE, and COMET scores, we propose new log scaling law and show that it has good predictive accuracy.
When the distributions are not sufficiently aligned and the finetuning data size is relatively small, translation scores fluctuate or even get worse with more pretraining-losie the monotonic scaling behavior (Figure | red curves). In these same settings, we find that the downstream cross-entropy still scales monotonically according to a power-law.
We argue that the value of pretraining data for translation tasks should be evaluated using downstream translation scores like BLEU, ROUGE, and COMET score and propose a prac- tical guide for such an assessment by leveraging the proposed scaling law for these scores.
We show that the proposed log scaling law generalizes to tasks beyond translation, with experiments on SuperGLUE (Wang et al., (019)) tasks, which covers question answering, reasoning, reading comprehension, and textual entailment.
II the rest of the paper, we will drop "downstreamm when we refer to the downstream translation scores such BLEU, ROUGE, and COMET.
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORK
Scaling laws for transformers. Scaling laws for LLMs have attracted significant attention as they can inform the decisions about key design choices such model size and the type and size of the
pretraining data (Kaplan et al. 2020; [Hffmmann et al. 2022: Hernandez et al. (021). Most of the
pioneering work has focused how upstream cross-entropy loss perplexity scales with more
pretraining data, larger models, or longer training (Kaplan et al. 2020 Hoffmann et al. 20022). Follow-
up works have analyzed scaling behavior of translation models (Ghorbani al. | Zhuocheng
et al. [0233] Gordon et al. 2021 Fernandes et al.) [023] Bansal et al. 2022: Zhang et al. (2022).
studied theoretical foundation behind scaling laws (Sharma & Kaplan 2020 Hutter] [202]] Bahri
et al. 2021), or extended the laws to the vision models (Zhai et al. 2022 Jain et al. (023). Closest
to our work, Hernandez et al. (202T) have analyzed transfer learning but with a focus on how the
cross-entropy loss behaves as the finetuning data scales. Unlike our work, their scaling law describes the relation between the size of a (finctuning) dataset and the cross-entropy loss on the same dataset making this closer to the standard scaling laws in the literature since the finetuning loss and the finetuning dataset are computed over samples from the same distribution. On the other hand, we propose scaling laws for the downstream metrics the finetuning dataset as the pretraining data scales switching the focus to an cout-of-distribuiinn
analysis. The only work we are aware of that proposed scaling laws for the downsfream task performance function of pretraining dataset size (Sun et al. 2017) has focused on classification tasks in the vision domain and used smaller models.
Transferability metrics and value of pretraining. While it may be commonly suggested that pretraining data improves both upstream and downstream performance, this rule has been challenged in the vision domain. Zoph et al. (2020); He et al. (2019): Shen et al. (2019); Ghiasi et al. (2018); Mikami et al. (2022) have demonstrated that pretraining can sometimes have no effect on the downstream task performance and sometimes it can even hurt the performance. We make similar observations in the language domain with extensive experiments on machine translation tasks and identify cases where (a) adding more pretraining data hurts the downstream task performance when pretraining data is not aligned enough with the task and (b) pretraining does not improve the downstream task performance noticeably when the finetuning dataset is large enough. Our observations about the importance of "aligned"" pretraining data are also supported by recent work on machine translation (Alves et al. 2024] Xu et al. 2024) trying to keep the pretraining data as multilingual as possible instead of being heavily English-centric (Stap et al. [022 Li et al. (024).
Another related line of work is on transferability metrics (Tamkin et al. [020 Chiang & Lee 2022
Ibrahim et al. [2022] Agostinelli et al. (2022; Nguyen et al., 2020 You et al.) 2021 Dai et al. 2019
Huang et al.] 2022] Ibrahim et al.] 2022; Tran et al. [019 [ao et al. 2019 Van Asch & Daelemans,
2010 Plank& & Van Noord 2011), ,, which are efficient heuristics used to select the most appropriate
source models or pretraining data for a given target task. We note that transfcrability metrics are designed to solve ranking problems, different from scaling laws. For example, these metrics answer questions such as given a pool of source models (or pretraining datasets), which source model (or pretraining dataset) is the best to finetune on for given target task. These metrics are not designed to predict the performance of the model when key quantities (e.g., pretraining data size) are scaled.
SCALING LAWS FOR TRANSFER LEARNING
In this section, we present our proposed scaling laws for translation quality metrics (e.g., BLEU, ROUGE, and COMET scores) and downstream cross-entropy. We also introduce an alignment score for translation tasks, discuss when the proposed scaling laws apply, and provide practical guidance for assessing the value of a pretraining dataset for a given target downstream translation task. The experimental results will be later discussed in Section 5
3.1 A SCALING LAW FOR TRANSLATION QUALITY METRICS
Different from cross-entropy and perplexity, which follow a power-law scaling behavior (Kaplan) jtt al. 2020 Hoffmann et al. 2022). we find out that translation scores, such as BLEU and COMET, scale closer a log-law, as evident from Figures 2 and 4 Therefore, we propose the following
3

---

Published as a conference paper at ICLR 2025
scaling law for translation scores as a function of the pretraining dataset size Dp:
f(pp) (log(A A D2))),
where A, a, and 3 are coefficients to be fit. We notice that these coefficients depend on how aligned the pretraining dataset with the target downstream task (translation from language to language 2) and how large the finetuning (translation) dataset is. With extensive experiments across several translation tasks and multilingual pretrained models, we demonstrate that the law in (T) indeed well describes translation quality scaling, with a small prediction error which quantify in Appendix C.3
3.2 TRANSLATION ALIGNMENT SCORE
It is nontrivial to define a general alignment score that could be used for any pair of pretraining data and downstream task since it an open research question what makes a pretraining data more aligned with (or relevant to) particular task. Therefore, we focus alignment score for translation tasks that captures the language overlap between the pretraining data and the translation task. We note that there might be alternative definitions of translation alignment. We propose one that measures what percentage of the languages in the translation task is present in the pretraining data in balanced way.
more controlled setting and define an
Definition 1 Translation Alignment Score), We use the following score to measure alignment between a multilingual pretraining data D and translation task T ( (Isouve: Ldea):
A(D, T[ [edest)) PLomae PLan + 0.7 Plemee + 0.8 P PL
where D is the pretraining data mixture, T( Ldea) is the translation task faskfrom Lyouree to Ldees Plenwee is percentage of Liswnre in D, and Pi.ze is percentage of Lieet in D.
For instance, for en-to-fr fr translation task, a pretraining data mixture with 50% en and 50% fr data would yield an alignment score of A(D, T(en, fr)) = 0.5 - 0.5 + 0.7 0.5 + 0.8 0.5 = 1. Likewise, pretraining data mixture with 100% would have an alignment score of A(D,T(en, fr)) = 1- 0 0.7 1 0.8 0 = 0.7.
3.3 IS CROSS-ENTROPE ENTROPY LOss ALWAYS A GOOD METRIC?
We compare the downstream cross-entropy loss and the translation scores empirically as prior work made the assumption that upstream or downsiream cross-entropy loss is good indicator for a model's downstream task performance. Following the well-understood scaling behavior of the upstream cross-entropy loss as a function of the pretraining dataset size (Kaplan et al. 2020: Hoftmann et al.)
2022
we show that the same scaling law can also describe the downstream cross-entropy loss as
A D3
L(Dp) F
where E, A, and are the coefficients be optimized. In Section 5 we report BLEU score and cross-entropy together for direct comparison and discover several cases where the two metrics do not correlate well. We provide similar results for COMET score Appendix C. These results support some of the findings Ghorbani et al. (20TT) suggesting inconsistency between the translation quality scores and the cross-ente cross-entropy. but also shows that the exponential relationship between BLEU score and cross-entropy advocated by Gordon et al. (2021) does not always hold. More specifically, our empirical results show that while cross-entropy loss always monotonically decreases (with appropriate learning rate) as the pretraining dataset size increases, translation score may show non-monotonic trend when the pretraining data is not sufficiently aligned with the task. For instance, in Figure [ we show the scaling behavior of translation scores like BLEU, ROUGE, and COMET and cross entropy as the size of a more aligned (AA = 1) and a less aligned (AA = 0.7) pretraining data increases. The first three plots show that increasing the less aligned data's size sometimes hurts the translation scores (more detailed results with full description of datasets and tasks will be in Sections [and (5). Even though they may initially follow the law in 4 for smaller pretraining dataset sizes, the scaling law breaks for larger data for the "less aligned" pretraining data. However, if we were to look at only
show that the same law also applies to other tasks, including question answering. reasoning, reading comprehension, and textual entailment.
IIn Appendix B
4

---

Published as a conference paper at ICLR 2025
the cross-entropy loss in Figure (right), we would conclude that both the more aligned and less aligned data bring noticeable improvements to the model and they both are worth being added into the pretraining mixture- which would be poor decision.
remotely related study on the mismatch between the task-related metrics and the cross coss-yntropy (McKenzie al.) (023) looked how the downstream task performance changes the model grows and suggested that LLMs may show worse task performance with increased model size but, similar our findings, this is not captured by the monotonically decreasing cross-entropy loss.
ROUGE score
COMET score
CE val. loss
BLFI
more aligned
less aligned
37 O-=8 tokens (pretraining)
an
saft Dy=H tokens (pretraining)
1st
Dy=H tokens (pretraining)
Dy= H tokens (pretraining)
Figure I: Scaling behavior of BLEU, ROUGE, COMET, and cross-entropy when the pretraining and downstream data are aligned with A (orange) and A = 0.7 (red). Task: en-to-fr translation.
3.4 WHEN Do SCALING LAWS FALL SHORT IN TRANSFER LEARNING?
While the cross-entropy loss always follows monotonically decreasing trend which can be captured by the scaling law in (3), we do not always see a monotonic increase in the translation scores when increasing the pretraining dataset size (See Figure I
(red curves) for an example on English-to- to-French translation task.). We observe that this only happens when the pretraining dataset is not sufficiently aligned with the translation task which results in low translation scores overall compared models that were pretrained in other datasets. For the pretrained models that lead to high translation scores after finetuning, we consistently see that the translation scores increase monotonically and can be well described with the scaling law in (Ι). Therefore, whether the scaling law could fit the empirical translation scores or not could be a good first-check in assessing the value of pretraining data for the downstream translation task. We elaborate more on this in the next section.
3.5 A GUIDE FOR PRETRAINING DATA VALUATION
Finally, combining our findings on the scaling behavior of translation scores, we propose the following guide for assessing the value of pretraining dataset for target downstream task:
1. Given pretraining dataset, pretrain long as possible under the given computational and time constraints [4] Periodically choose pretraining checkpoints, finetune on them, and record the downstream performance metric (we recommend BLEU, ROUGE, or COMET scores over cross-entropy due to the discussion in Section 3.4).
2. Since the law in T has three coefficients to be fit, once we have pairs of (number of pretraining tokens seen, translation score), we fry find the optimal coefficients and move onto one of the following steps:
(a) the translation scores have a mon-monotonic behavior (e.g., red curves in Figure D.. cannot fit the scaling law. Since the non-monotonic behavior could be an indication of misalignment (following the discussion in Section (3.4), we expect worse performance with more pretraining data. Therefore, recommend checking the score of the best available finetuned checkpoint and comparing it to the performance of the non-pretrained model trained on the downstream task directly. This would indicate how much value we can get from this pretraining dataset.
(b) If the scaling law fits well (e.g., orange curves in Figure []] then make the initial prediction for the translation score as we increase the pretraining dataset size. If we are not satisfied with the predicted score, then we conclude that it is not worth pretraining on this dataset. the predicted score is high enough, then we keep pretraining until
"We avoid repeating sequences repetitions may complicate the scaling behavior (Hernandez et al. 2022 Muennighoff et al. 2023 | Tirumala et al. 2023). This means pretraining goes effectively pretrain each checkpoint "larger dataset".
5

---

Published as a conference paper at ICLR 2025
reach the target score. If the scaling law breaks at any point, conclude that the pretraining dataset is not sufficiently aligned with the downstream task and pretraining further may not be beneficial.
4 EXPERIMENTAL SETUP
In the experiments, we first pretrain a model without doing more than one pass over any of the examples. Then, we finetune selected checkpoints of the pretrained model. Naturally, there is one-to-one mapping between the checkpoint number and the number of pretraining tokens seen. This way, we collect pairs of (number of pretraining tokens, translation score) and (number of pretraining tokens, downstream cross-entropy loss) to analyze them with the proposed scaling laws in (1) and (3). All the plots are on a log-log scale. We present BLEU results in this section and provide COMET results in Appendix C.I The observations and conclusions are similar in both scores.
Model. We use the 3-billion encoder-decoder T5 model with 24 encoder layers, 24 decoder layers, embedding dimension 1024, and 32 heads with dimension 128. We note that this is the same model as T5-3B in Abnar et al. 2022)) In Appendix [ we also provide results with the 770-million encoder- decoder T5 model. This model corresponds to T5-Larrge in Raffel et al. 2020). We share more details about the architectures in Appendix A For encoding the text WordPiece tokens (Sennrich et al.) 2016: Kudo 2018), we use SentencePiece (Kudo & Richardson, (0018) trained with a vocabulary of size 250, 112 that covers all the languages the MC4 dataset (Raffel al. 2020).
Datasets. We use the English (en). German (de), French (fr), and Romanian (ro) portions of the MC4 dataset. We experiment with both pretraining on these languages individually as well as mixing pairs of languages. of 50% en-MC4 + 50% MC-MC4 (center) mixture of 50% en-MC4 + 50% fr-MC4, and (right) a mixture of 50% en-MC4 + 50% ro-MC4 - meaning that 50% of one pretraining batch is sampled from en-MC4 and the other 50% is sampled from the other language. Notice that all the pretraining data-translation task pairs in Figure 2 has an alignment score of 1. In Figure 3 we show results for the models pretrained only en-MC4, corresponding to alignment score of A 0.7. In Figure 4 in addition to these, we also present results for the models pretrained on a mixture of 30% en-MC4 70%-fr and mixture of 70% en-MC4 + 30%-fr as well as models pretrained only on de-MC4, only on fr-MC4, and only on ro-MC4. We finetune the pretrained models on 1M en-de with 3B tokens (Bojar al. 2017). WMT- 15 en-fr with 21B tokens (Bojar et al. 2014), and WMT-16 en-ro with 312M tokens (Bojar et al. 2016), separately. To understand the effect of the finetuning data size on scaling, we sometimes use a smaller randomly sampled portion from these translation datasets and indicate the number of tokens used in the plots.
Figure 2 we present results for the models pretrained on (left) a mixture
In Appendix B we provide additional experimental results to demonstrate that the proposed scaling law is applicable to tasks beyond translation as well. For this, we analyze models pretrained on en-MC4 and finetuned on SuperGLUE (Wang et al.) 2019), which includes several classes of tasks such as question answering (BoolQ, MultiRC), reasoning (COPA), reading comprehension (ReCoRD), and textual entailment (RTE).
Hyperparameters. During pretraining, we use a batch size of 256 and a sequence length of 512 for 1, .000 000 steps except for the ro-MC4 pretraining. For ro-MC4. pretrain for 510, 000 steps since otherwise, we would need to do repetitions over the sequences. Following Raffel et al. (2020), we use an "inverse square root" learning rate schedule,
where is the current pretraining step "max(n,k))
and k = 10 We do grid search for the base learning rate from (0.05, 0.1, 0.5, 1.0, 2.0. 5.0) and pick the best one for each pretrained model based on upstream cross entropy. We perform full-weight finetuning, During finetuning, again following Raffel et al. (2020), we use a batch size of 128 and a sequence length of 512 for 300 steps. We use a constant learning rate by selecting the best from {0.001, 0.005, 0.01, 0.05, 0.1). In both stages, use AdaFactor optimizer (Shazeer & Stern 2018).
Optimizing the scaling law coefficients. To fit the coefficients in the scaling laws in and (3). similar to Hoffmann et al.) (2022), we use the Huber loss (Huber) 1992) and the L-BFGS algorithm (Nocedal 1980) estimate the scaling law robustly in the presence of outliers. For the

---

Published as a conference paper at ICLR 2025
501
Kcrr WMTT 10 (711110
BUBU UNT.17
101
BL2 score on 10700-
30F
10
pretraink
polhantr
pretrating!
valdation ande
2004 Dw
10
Thw
60
Tee MNX on
valtation
MCC pretriink
Oged
50%
Figure 2: (top) BLEU score pretraining dataset size: f(Dp) (log(A (D)))) D2)) (left) WM en-to-de translation task. Pretraining dataset has 50% en-MC4 50% de-MC4. Dotted, dashed, and solid blue curves correspond to the fitted scaling laws for different finetuning dataset sizes, D, = 6.M, D, 31M, D, 3B tokens, respectively. (center) WMT-15 en-to-fr translation task. Pretraining dataset has 50% en-MC4 and 50% fr-C4. Dotted, dashed, and solid orange curves correspond to the fitted scaling laws for different finetuning dataset sizes, Dy = 42M, D, = 210M, D, 21B tokens, respectively. (right) WMT-16 en-to-ro translation task. Pretraining dataset has 50% en-MC4 + 50% ro-MC4. Dotted, dashed, and solid green curves correspond to the fitted scaling laws for different finetuning dataset sizes, Dj = 625K, Dj 3M, Dj 312M tokens, respectively. (bottom) Cross-entropy (CE) validation loss pretraining dataset size: L(D,) E D
Same models as the top row. For all the plots, the markers are the actual experimental results and the black horizontal curves correspond to the nonppretrained model directly trained the task dataset. The finetuning dataset size increases in the order of dotted-dashed curves including the black horizontal lines. Note that all the plots have alignment score of A 1.
for all the
Huber loss, we use = 0.1 for the translation scores and 6 = le- for the downstream cross-entropy loss. We select the best fit among grid of initializations and report the prediction error computed via the Huber loss in Appendix C.3 To optimize the coefficients, we use the first four data points that require the smallest amount of pretraining data and leave the remaining data points held-out data to evaluate the accuracy of the laws. We note that, ideally, three points should be enough since both laws have three coefficients be optimized for. However, adding more points improves the fit by making the optimization more robust to outliers. We provide more details about how to optimize the scaling law coefficients Appendix A.2 We refer the reader to Appendix C.3 for the list of optimized coefficients and the prediction errors for each law present in the next section.
RESULTS AND ANALYSIS
In Figure we analyze the models that are pretrained on different portions of (left) a mixture of 50% MC- + 50% de-MC4 (A = 1), (center) mixture of 50% en-MC4 + 50% fr-MC4 (A4 = 1). and (right) a mixture 50% en-MC4 50% ro-MC4 (A4 = 1). These models are then finetuned on different portions of (left) en-de, (center) en-fr, and (right) en-ro translation datasets. In the top row, we report the BLEU score and, in the bottom row, we report the downstream cross-entropy loss. The dotted, dashed, and solid lines correspond to the scaling laws in (1) and (3) for different finetuning dataset sizes Dy. The black lines correspond to "non-pu "non-pretrand poet--iraa models (randomly initialized) that directly trained different portions of the finetuning dataset. In all cases, the scaling laws fit well to the empirical results (the markers) with prediction error at most 0.061 for the BLEU score (8 = 0.1) and 5.95e 12 for the downstream cross-entropy (ō le 3) (see Appendix C.3 for more details).
7

---

Published as a conference paper at ICLR 2025
As expected, as the finetuning dataset size increases (e.g.. going in the order of wotted-dashed-eee lines), the BLEU score increases and the cross-entrops loss decreases smoothly and monotonically. Similarly, as the pretraining dataset size D, increases (along the x-axis), metrics. Notice that the improvements by an increase in the pretraining dataset size is more effective for smaller finetuning datasets. When the finetuning dataset is large enough (e.g., solid lines), BLEU score is more or less constant regardless of the pretraining dataset size. In fact, we see little to no improvement of pretraining compared the non-pretrained models (black lines) when the finetuning dataset is large. This implies that, for these tasks, there is no need to pretrain the models when the finetuning dataset is large enough (We note that typically supervised finetuning data is not as widely available as unsupervised data due to its cost hence pretraining unsupervised data important practice.). Luckily, we can correctly predict whether this going to be the case (i.e., whether the available finetuning data is enough to eliminate pretraining altogether) with the use of scaling laws.
improvements in both
In Figure we change the pretraining dataset to 100% en-MC4 in all plots, giving an alignment score of A 0.7. Intuitively, we expect this dataset to be less aligned with the translation tasks than the multilingual pairs in Figure 2 since it does not include one of the languages in the translation tasks. Indeed, we see smaller BLEU score and higher cross-entrops loss in general for the same finetuning dataset size. Most of the conclusions from carur over the results in Figure 3 For instance, the pretraining data matters less when the finetuning dataset is large enough. One noticeable difference is in the BLEU scores for the en-fr translation task (center). We see that, for D, 42M and D, 210M, the scaling law for BLEU score actually breaks once the pretraining dataset size passes a threshold while the cross-entropy loss scales as expected. This is counter-intuitive because the BLEU score sometimes decreases for larger pretraining dataset. Notice that this break in scaling law does not happen in en-de or en-ro translation tasks as the scaling law fits well the pretraining data with prediction error at most 0.025 for these tasks (5 0.1). To better investigate this, in Figure 4 we take a closer look at some less aligned pretraining datasets due to the choice of language.
DUEU scrre WMT127
to1
WON 22
soons on
10
BLLU
200
10a
12"
10
10
10"
10
0p- tolene (100% en M9C4 pretraining
D tokens (100% en MC4 pretrainingi
0p-P takens 1100% en MC4 pretraiiing
loss on
---.
112
1024
105
UPa
1) Sokama (100% an M000 protivening)
1m lokana 1100% an MCSA inntraining
12 tokana 1100% M0-- gewttentingg
Figure 3: Same as Figure 2 except that the pretraining dataset is 100% en-MC4 for all plots, i.c., the alignment score is A
0.7.
In Figure 4 (left), we provide the scaling laws for en-de translation task where the pretraining datasets are 100% en-MC4 (A 0.7, same as Figure (left). 50% en-MC4 and 50% de-MC4 (A = 1, same as Figure ( (left). 100% de-MC4 (AA = 0.8), 100% fr-MC4 (AA = 0, less aligned), and 100% ro-MC4 (A = 0, less aligned). Notice that the last two pretraining datasets are expected to be the least aligned with the translation task since the translation pair does not include these languages. We see that, despite this, the scaling laws consistently fit well for both the BLEU score and the cross-entropy loss. However, this is not always the for the en-fr translation task. In (right), we provide the scaling laws for the en-fi fr translation task where the pretraining datasets are different mixtures of en-MC4 and MC-MC4 datasets. We also include the "less aligned" pretraining datasets such as
8

---

Published as a conference paper at ICLR 2025
BLEU score on WWTT-17 an-48
B.E.J. score on WMT--15 endi
10
500% MCA. 1-007
Min cealce
10
129H
10
300% rac.
Dp- & to tokens kens (pretraining)
Dp-At takens (pretraining)
2000
Now noc
CE validatinn loss on WAT an
Mice sanui HDA 50% MVD 50% HDA.A wa
120% de INCL
vallicazion loxs on WMT-11
10
10-1
100
10
10
121
10th
10
Dz-9t tokens (pretraining))
takens (pretraining)
Figure 4: Comparison of scaling behavior for different pretraining datasets. (top) BLEU score vs pretraining dataset size: f(Dp) ) = (log(A D (left) WMT-17 en-de translation task. (right) WMT-15 en-fr translation task. (bottom) Cross-entropy (CE) validation loss vs pretraining dataset size: L(Dp) = E + Dp
Same as the top row but for CE loss instead of BLEU score.
For all the plots, the markers are the actual experimental results and the black horizontal curves correspond to the non-pretrained model directly trained on the task dataset.
100% Mc-MC4 (A4 = 0) and 100% ro-MC4 -A= = 0). Surprisingly, we see that the scaling law for the BLEU score breaks after some point for the only-English (100% en-MC4, A = 0.7), only-German (100% de-MC4, A = 0), and only-Romanian (100% ro-MC4, AA = 0) pretraining datasets while the cross-entropy loss always follows the scaling law in 33. Interestingly, we do not observe such break in the BLEU score scaling for the only-French (100% fr-MC4, A = 0.8) pretraining dataset hinting that not including French data pretraining leads to poor scaling the en-fr translation task but not including English does not have such an effect. We also notice that the BLEU score is the lowest for these three pretraining datasets where scaling breaks. This suggests that the scaling law in (1) works well for the BLEU score as long as the pretraining dataset has the promise to give rise to a good performance. However, when the scaling law does not fit well, we may suspect the BLEU score to be low overall. Therefore, whether we can fit the scaling law for the BLEU score seems to give good indication about the degree of alignment between the pretraining data and the particular translation task.
Remark 1. We observe another interesting phenomenon in Figure For both en-de and en-fr tasks, 100% MC- leads to significantly worse BLEU score and downstream cross-entropy than the more aligned 50% en-MC4 + 50% de/fr-MC4 balanced datasets, respectively. However, de-MC4 and fr-MC4 perform almost as well as the balanced datasets in en-de and en-fr tasks. We believe this is because, in these translation tasks, the model generates text in German/Frende (not English), and de/fr-MC4 pretraining is more helpful than en-MC4. We leave further investigation future work.
We also highlight that we cannot make any strong conclusion about the degree of alignment of the pretraining dataset with the task by only looking at the downstream cross-entropy loss because of the inconsistency with the BLEU score, task-related metric, observed in the en-fr plots in Figures
3 and This counter-example for the claim by Gordon et al. (021)) that the two metrics have exponential relation. To better demonstrate this, in Figure 5] we provide BLEU score vs. downstream cross-entropy log-log plot for en-de and en-fr translation tasks, respectively. While the two metrics indeed seem correlated in Figure (left) the en-de task, observe

---

Published as a conference paper at ICLR 2025
BLEU score on WAT--17
BLE. WMMT-15
101
100
C-131 CE velicaaion loss on WM eIHT
CE validation loss on WMT-17 en-de
Figure 5: BLEU score vs. downstream cross-entropy loss. (left) For en-de translation task, we see a consistent correlation between the two metries for all the pretraining datasets. This supports the findings of Gordon et al. (2021). (right) For en-fr translation task, the two metrics usually show an arbitrary relation. Sometimes, the BLEU score increases while the cross-entropy also increases. Unlike the en-de results (left), the exponential relation in (Gordon et al. 2021 is not observed here.
somewhat arbitrary relation for the en-fr task in Figure (right) in some cases - which clearly cannot be explained with exponential relation. This suggests that downstream cross-entropy is not always good indicator for BLEU score and raises the question whether the existing scaling laws for cross-entropy are actually useful predictors for models' downstream behavior.
All the observations on BLEU score presented in this section carry over to COMET score as well (see Figure Tand Appendix C.T.
Remark 2. To better understand the root cause of the non-monotonic behavior of the BLEU score when the alignment is not sufficient (i.e., when the BLEU score fluctuates with more pretraining data), we revisit its definition. Recall that the common form of BLEU score is as follows
1/4
BLEU brevity-penalty
II precision,
where precision, refers to the precision of n-grams, and the second term is the geometric mean of the precision when varied from to 4. In all the experiments, we observe brevity-penalty 1, i.c. the non-monotonic behavior comes from the precision term, not the brevity penalty.
5.1 OTHER TASKS
we show that the proposed log scaling law is not only applicable to the translation scores/tasks but also to metrics on question answering, reasoning. reading comprehension, and textual entailment tasks within SuperGLUE (Wang et al.) (019). Our results demonstrate that the same scaling law captures the scaling of these metrics as the pretraining data grows.
In Appendix B
DISCUSSION AND CONCLUSION
We study the scaling behavior of the downstream performance in machine translation as the pretraining data grows and propose scaling laws for both downstream cross-entropy and translation quality metrics. We demonstrate through extensive experiments that the scaling behavior is significantly influenced by (1) the degree of alignment between the pretraining and the downstream data and (2) the finetuning dataset size. In favorable cases where the distributions are sufficiently aligned, we show that downstream translation quality, measured by translation scores, can be accurately predicted using a log scaling law. However, with less alignment, there are cases where translation scores fluctuate unpredictably whereas downstream cross cross-entropy improves monotonically. We also observe that when the finetuning dataset size is sufficiently large, pretraining has little to value. Our findings highlight the importance of studying downstream performance metrics and not making decisions solely based cross-entropy (whether upstream downstream).
10

---

Published as a conference paper at ICLR 2025
Limitations. Our work goes beyond cross-entropy loss to understand and predict the downstream model performance at scale. While the proposed laws fit the empirical data well and predict the translation scores scale successfully when there is sufficient alignment, there are cases where these scores do not scale monotonically. Our work identifies many such cases; however, as mentioned in Remark [ more linguistic approach into alignment in translation could provide better understanding.
Reproducibility Statement. We used publicly available datasets and models, and specified their @ and Appendix A We provided details on the training
versions with proper citations in Section procedure and hyperparameters for both pretraining and finetuning stages.
REFERENCES
Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hamie Sedghi. Exploring the limits of large scale pre-trainin In International Conference on Learning Representations, 2022. URL
fttps://opperest 3ttp:r/ewwweeeeeeeerumummmmm forum?id=v338pp
net.
Andrea Agostinelli, Jasper Uijlings. Thomas Mensink. and Vittorio Ferrari. Transferability metrics for selecting source model ensembles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pp. 7936-7946, 2022.
Duarte Miguel Alves, José Pombal, Nuno M Guerreiro, Pedro Henrique Martins, João Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, Pierre Colombo, José G. C. de Souza, and Andre Martins. Tower: An open multilingual large language model for translation-related tasks. In First Conference on Language Modeling, 2024. URL https //openreview
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jachoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. arXiv preprint arXiv:2202.067700 2021.
Yamini Bansal, Behrooz Ghorbani, Ankush Garg, Biao Zhang, Colin Cherry, Behnam Neyshabur, and Orhan Firat. Data scaling laws in nmt: The effeet of noise and architecture. In International Conference on Machine Learning, pp. 1482. PMLR, 2022.
Yajie Bao, Yang Li, Shao-Lun Huang. Lin Zhang. Lizhong Zheng, Amir Zamir, and Leonidas Guibas. An intommation-thorettii approach transferability in task transfer learning. In 2019 IEEE international conference on image processing (ICIP), pp. 2309-2313. IEEE, 2019.
Roy Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Beraaro Magnini, and Idan Szpektor. The second PASCAL recognising textual entailment challenge. 2006.
Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. The fifth PASCAL recognizing textual entailment challenge. 2009.
Ond rej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. Findings of the 2016 conference on machine translation. In Proceedings of the First Conference on Machine Translation, pp. 131-198, Berlin, Germany, August 2016. Association for Computational Linguistics. URL http:/ www. ac web. orroor/g/aao antho ogy/o////////////ooo /w/ 6/W1 6-201
Ond rej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. Findings of the 2017 conference on machine translation (wmt17). Proceedings of the Second Conference on Machine Trunslation, Volume 2: Shared Task Papers, pp. 169-2144 Copenhagen, Denmark, September 2017. Association for Computational Linguistics. URL TRLLttt::/ www. aclweb. org/ anthology//017-660
Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve workshop on statistical machine translation. In Proceedings of the ninth workshop on statistical machine translation, pp. 12-58, 2014.
et al. Findings of the 2014
11

---

Published as a conference paper at ICLR 2025
Cheng-Han Chiang and Hung-yi Lee. study from artificial datasets. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp.
On the transferability of pre pre-trained language models: A
2022.
Christopher Clark, Kenton Lee, Ming- Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of NAACL-HHLT 2019, 2019.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment, pp. 177-190. Springer, 2006.
Xiang Dai, Sarvnaz Karimi, Ben Hachey, and Cecile Paris. Using similarity measures to select pretraining data for ner. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 1460-1470, 2019.
Caarie-Cathrrine De Marneffe, Mandy Simons, and Judith Tonhauser. The CommitmentBank: Investigating projection in naturally occurring discourse. 2019. To appear in proceedings of Sinn und Bedeutung 23. Data can be found at
Patrick Fernandes, Behrooz Ghorbani, Xavier Garcia, Markus Freitag, and Orhan Firat. Scaling laws for multilingual neural machine translation. arXiv preprint urXiv:2302.0965: 2023.
Golnaz Ghiasi, Tsung-Y Yi Lin, and Quoe Le. Dropblock: regularization method for convolutional networks. Advances in neural information processing systems, 31, 2018.
Behrooz Ghorbani, Orhan Firat, Markus Freitag. Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. Scaling laws for neural machine translation. In International Conference on Learning Representations, 2021.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pp. 1-9. Association for Computational Linguistics, 2007.
Mitchell A Gordon, Kevin Duh, and Jared Kaplan. Data and parameter scaling laws for neural machine translation. In Marie-Francine Moens, Xuanjing Huang. Lucia Specia, and Scott Wen- tau Yih (eds.), Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pp. 5915-5922, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguisties. doi: 10.18653/v1/2021 www.:p-mmmlppaa...... URL htthttps! www..mmamlpppmm.m.mm
T/aclantholooy.orgggggooo
Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4918 0108-4927220 2019.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Hcewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.1 /4701, 2020.
Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint arXiv: 20t.::20022/229000 2021.
Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage. Zac Hatfield-Dose Doddss Tom Henighan, Tristan Hume, et al. Scaling laws and interpretability of learning from repeated data. arXiv preprint 2r0iv: 10487, 2022.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Training compute-optima large language models. arXiv preprint arXiv:2203. 15556, 2022.
Long- Kai Huang, Junzhou Huang, Yu Rong, Qiang Yang, and Ying Wei. Frustratingly easy trans- ferability estimation. In International Conference on Machine Learning, pp. 9201-9225. PMLR, 2022.
12

---

Published as a conference paper at ICLR 2025
Peter J Huber. Robust estimation of location parameter. In Breakthroughs in statistics: Methodology and distribution, pp. 492-518. Springer, 1992.
Marcus Hutter. Learning curve theory. arXiv preprint 1xXiv:2102.04074, 2021.
Shibal Ibrahim, Natalia Ponomareva, and Rahul Mazumder. Newer not always better: Rethinking transferability metries, their peculiarities, stability and performance. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 693-709. Springer, 2022.
Achin Jain, Gurumurthy Swaminathan, Paolo Favaro, Hao Yang, Avinash Ravichandran, Hrayr Harutyunyan, Alessandro Achille, Onkar Dabeer, Bernt Schiele, Ashwin Swaminathan, et al. A meta-learning approach predicting performance and data requirements. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3623-3632, 2023.
Mark Johnson, Peter Anderson, Mark Dras, and Mark Steedman. Predicting accuracy on large datasets from smaller pilot data. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 450-455, 2018.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08.60 2020.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume (Long Papers), pp. 252-262, 2018
Taku Kudo. Subword regularization: Improving neural network translation models with multiple sub- word candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Pupers), pp. 75,75, 2018.
Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. EMNLP 2018, pp. 66, 2018
Jiahuan Li, Hao Zhou, Shujian Huang, Shanbo Cheng. and Jiajun Chen. Eliciting the translation ability of large language models via multilingual finetuning with translation instructions. Transactions of the Association for Computational Linguistics, 12:576-592. 2024.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summarics. Text summarization branches out, pp. 74-81, 2004.
Ian McKenzie, Alexander Lyzhov, Michael Pieler, Alicia Parrish, Aaron Mueller, Ameya Prabhu, Euan McLean, Aaron Kirtland, Alexis Ross, Alisa Liu, al. Inverse scaling: When bigger isn' t better. arXiv preprint arXiv:2306.09479 2023.
Hiroaki Mikami, Kenji Fukumizu, Shogo Murai, Shuji Suzuki, Yuta Kikuchi, Taiji Suzuki, Shin-ichi Maeda, and Kohei Hayashi. A scaling law for syn2real transfer: How much is your pre-training effective? In Joint European Conference Machine Learning and Knowledge Discovery in Databases, pp. 477-492. Springer, 2022.
Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models. In Thirty-seventh Conference Neural Information Processing Systems, 2023. URL
Cuong Nguyen, Tal Hassner, Matthias Seeger, and Cedric Archambeau. Leep: A new measure to evaluate transferability of learned representations. In International Conference on Machine Learning, pp. 7294-7305. PMLR, 2020.
Jorge Nocedal. Updating quasi-newton matrices with limited storage. Mathematics of computation, 35(151):773-782, 1980.
13

---

Published as a conference paper at ICLR 2025
OpenAL.
Learning to reason with llms. 2024.
In bttps:/
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Mohammad Taher Pilehvar and Jose evaluating context-sensitive meaning representations. In Proceedings of NAACL-HLT, 2019.
WIC: The word-in-context dataset for
Barbara Plank and Gertjan Van Noord. Effective measures of domain similarity for parsing. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 1566-1576, 2011.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu, Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-5511 2020.
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. Comet: A neural framework for mt evaluation. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 2685-2702, 2020.
Ricardo Rei, José GC De Souza, Duarte Alves, Chrysoula Zerva, Ana Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André FT Martins. Comet-22: Unbabel-ist 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pp. 578-585, 2022.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning, 2011 AAAI Spring Symposium Series, 2011.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1715-1725, 2016.
Utkarsh Sharma and Jared Kaplan. neural scaling law from the dimension of the data manifold. arXiv preprint XXiv:2004.10802 2020.
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pp. 4596 4596-4604. PMLR, 2018
Zhiqiang Shen, Zhuang Liu, Jianguo Li, Yu-Gang Jiang, Yurong Chen, and Xiangyang Xue. Object detection from scratch with deep supervision. IEEE transactions on pattern analysis and machine intelligence, 42(2):398 412, 2019.
David Stap. Eva Hasler, Bill Byrne, Christof Monz, and Ke Tran. The fine-tuning paradox: Boost- ing translation quality without sacrificing LLM abilities. In Lun-We Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 6189-620 Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10.28 acl-oo..33............3.333 URL
long. 336
Craig Stewart, Ricardo Rei, Catarina Farinha, and Alon Lavie. Comet-deploying new otate-of-the-art itt evaluation metric in production. In AMTA (2), pp. 78-109, 2020.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pp. 843-852, 2017.
Alex Tamkin, Trisha Singh, Davide Giovanardi, and Noah Goodman. Investigating transferability in pretrained language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1393-1401, 2020.
14

---

Published as a conference paper at ICLR 2025
Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler. Scale efficiently: Insights from pretraining and finetuning transformers. In International Conference on Learning Representations, 2021.
Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari S Morcos. D4: Improving Ilm pretraining via document de-duplication and diversification. Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
Anh T Tran, Cuong V Nguyen, and Tal Hassner. Transferability and hardness of supervised classifi- cation tasks. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 1395-1405, 2019.
Vincent Van Asch and Walter Daelemans. Using domain similarity for performance estimation. In Proceedings of the 2010 Workshop on Domain Adaptation Natural Language Processing, pp. 31-36, 2010.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy. and Samuel R. Bowman. SuperGLUE: A stickier benchmark for general-purpose language understanding systems, arXiv preprint 1905.00537, 2019.
Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. A paradigm shift in machine translation: Boosting translation performance of large language models. In The Twelfth Interna- tional Conference on Learning Representations, 2024. URL
net/
Forumpid=far16XWns
Kaichao You, Yong Liu, Jianmin Wang, and Mingsheng Long. Logme:
Practical assessment of pre-trained models for transfer learning. In International Conference on Machine Learning, pp. 12133- PMLR, 2021.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
1210411213, 2022.
Biao Zhang, Behrooz Ghorhani, Ankur Bapna, Yong Cheng, Xavier Garcia, Jonathan Shen, and Orhan Firat. Examining scaling and transfer of language model architectures for machine translation. In International Conference on Machine Learning, pp. 26176-26192. PMLR, 2022.
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. ReCoRD: Bridging the gap between human and machine commonsense reading comprehension. 2018.
arXiv preprint
Zhang Zhuocheng, Shuhao Gu, Min Zhang, and Yang Feng. Scaling law for document neural machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 8290-8303, 2023.
Barret Zoph, Golnaz Ghiasi, Tsung-Y Yi Lin, Yin Cui, Hanxiao Liu, Ekin Dogus Cubuk, and Quoe Le. Rethinking pre- ve-trrining and self seff-training. Advances in neural information processing systems, 33: 3833-3845, 2020.
15

---

Published as a conference paper at ICLR 2025
A ADDITIONAL EXPERIMENTAL DETAILS
For the T5-3B experiments, pretraining for 1M steps takes 15-20 hours and finetuning takes 5-7 hours on an 8x8 TPU. For the sake of anonymity, we are unable to provide further information on compute specifications at this time, but we will add details upon acceptance.
A.1 MODEL ARCHITECTURES
We provide the architecture details of the T5-3B and T5-770M models in Tables models were initially introduced by Raffel et al. (2020).
and
These
Table 1: T5-3B RRafell et al. 2020) architecture details.
Embedding Dimension Number of Heads
1024
32
Number of Encoder Layers
24
Number of Decoder Layers
24
Head Dimension
128
MLP Dimension
16384
Table 2: T5-770M
RRfffel et al.
2020
architecture details.
Embedding Dimension
1024
Number of Heads
16
Number of Encoder Layers
24
Number of Decoder Layers
24
Head Dimension
64
MLP Dimension
2816
A.2 OPTIMIZING THE SCALING LAW CORFFICIENTS
In this section, provide more details how optimize the coefficients of the scaling laws. Following Hoffmann et al. (0022). we use the Huber loss (Huber 1992) to minimize overfitting to the outliers. Huber loss is particularly useful to suppress the effect the outlier data points in the optimization problem. More specifically, if the data point with value r is predicted by the law as f, the loss for that data point would be
(r - 7)2 6 (1r - fl 18) otherwise.
for Ir F| 6,
(a(r, f)
(5)
Due to the numerical range difference between the COMET/BLEU score (between 0 and 100) and the downstream cross-entropy typically taking much smaller values, we use = 0.1 for BLEU score law in ( and 8 1e 3 for the downstream cross-entropy law in (3)
For optimization, we use the L-BFGS algorithm (Nocedal 1980 Specifically, for COMET/BLEU score law in (T) we solve
min E.A,o.B
Σ
Ea(log fi. log f (Dp:))
(6)
Data point
where Dps is the pretraining dataset size and ft is the COMET/BLEU score for the data point i, and f(-) the approximation for the optimal law f().. Similarly, for the downstream cross-entropy loss law in (3), solve
16

---

Published as a conference paper at ICLR 2025
min E,A.a
Σ
(log log pPii))
Data point é
where Dps is the pretraining dataset size and Lg is the downstream cross-entropy loss for the data point i, and L() is the approximation for the optimal law L(.).
B SUPERGLUE EXPERIMENTS
Figure 6 demonstrates how SuperGLUE (Wang et al. 2019) task metrics such as Boolean Ques-
tions (BoolQ) (Clark et al., 2019). CommitmentBank (CB) (De Marneffe et al.) 2019). Choice of Plausible Alternaties (COPA) (Roemmele et al.
20.1). Multi-Sentence Reading Comprehension
(MultiRC) (Khashabi al. 2018), Recognizing Textual Entailment (RTE) (Dagan et al. 2006
Bar Haim et al.j 2006 Giampiccolo et al. 2007 [entivogli et al., 2009), Reading Comprehen- sion with Commonsense Reasoning Dataset (ReCoRD)) (Zhang et al.) 2018), and Word-in-Conteex
(WiC) (Pilehvar & Camacho-Collados 2019) scale as the pretraining data grows. For these exper- iments, we use T5-3B model pretrained on en-MC4 data (same as Section [5). For finetuning on SuperGLUE, we use batch size of 128 and a sequence length of 512 for 300 steps. We use constant learning rate by selecting the best from {0.001, 0.005, 0.01, 0.05, 0.5..
The results indicate that the same scaling law, f(Dp) (log(A A -D))) that was demonstrated to fit well to translation scores in Section S also captures the scaling of question answering (BoolQ, MultiRC), reasoning (COPA), reading comprehension (ReCoRD), and textual entailment (RTE) tasks, as well.
C ADDITIONAL EXPERIMENTAL RESULTS
In this section, we provide additional experimental results that we had to skip in the main body due to the page limit.
C.I RESULTS WITH COMET SCORES
We extend our experimental evaluation to COMET score, which we had to skip in the main body
due the page limit. In Figure 7 we provide the COMET scores for the models previously used
in Figures 2 and 3 for BLEU score and cross-entropy. Similar to BLEU score, the law given in (T)
well describes the scaling behavior of COMET score, when there is sufficient alignment between the pretraining and dowsntream data (Figure (top)). When the alignment is not sufficient (Figure 7 (bottom)), again similar to the BLEU score, COMET score fluctuates and sometimes gets worse with more pretraining.
C.2 RESULTS ON T5-770M
In Figures and 9 we present results similar to Figures and 3 in Section 5 but for T5-770M instead of T5-3B. In general, observe similar trend. The proposed scaling laws describe the
downstream behavior well when the pretraining and downstream data are aligned. Similar to the results in T5-3B in the main body of the paper, Figure (top, right), we observe break in the scaling law when the pretraining dataset is 100% en-MC4 and the task is en-fr translation suggesting the same misalignment for this pretraining data and task that was also observed in Section on the larger T5-3B model.
C.3 OPTIMIZED COEFFICIENTS AND PREDICTION ERRORS OF THE SCALING LAWS
In Tables and 6 we provide the optimized coefficients for the scaling laws plotted Figures and together with the prediction error.
17

---

Published as a conference paper at ICLR 2025
1:2
20th
102
1014
107
10
10a
10
Py5P trlens fer MAC pretroining)
0,20 tolems (en MCA pg)
0 trkens (en MC4 peetraning)
maltrc exact matching
100
mulbrc F2
20
matching
10
10
10"
10a
10
10"
D==h tokens (en MC4 pretraining!
Dy= + tokens (en MC4 pretraining)
1:30
f2
0::37
wwraot matchig
10
10
Dych tokens (en MC4 pretraining!
Dy= P tokens (en MC4 pretraining)
10
10tt
10
10"
10
10"
Dy=h tokens (en MC4 pretraining!
Dy= tokens (en MC4 pretraining)
Figure 6: SuperGLUE scores vs pretraining dataset size: f(Dp) = (log(A D Pretraining dataset is en-MC4 and finetuning dataset is SuperGI.UE. For all the plots, the markers are the actual experimental results and the black horizontal curves correspond to the non-pretrained model directly trained on the SuperGLUE dataset.
18

---

Published as a conference paper at ICLR 2025
COMEET score on WMF-17 en-de
COMET WMT--15
6X44
Dv 42M
31M
Dy 210M
10
209
2010
201l
0== tokens (50% en-MC4 50% de-MC4 pretraining
Dp= tokens (50% en MC4
50% MC4 pretralning)
COMET COMET score on WMT-11 en-de
COMET WMTT-15
6.39
68
42M
218
10a
1016
1021
100
1010
2011
pretraining:
D.=
pretraining)
Figure 7: (top) COMET score results for the 50%-50% balanced experiments in Figure 2
f(Dp) = (log(A DD)) (left) WMT-17 en-to-de translation task. Pretraining dataset has 50%
en-MC4 + 50% de-MC4. Dotted, dashed, and solid blue curves correspond to the fitted scaling laws for different finetuning dataset sizes, D, 6.,, D, 31M, D, 3B tokens. respectively. (right) WMT-15 en-to-fr translation task. Pretraining dataset has 50% en-MC4 and 50% fr-MC4. Dotted, dashed, and solid orange curves correspond the fitted scaling laws for different finetuning dataset sizes, D 42M, Dj- 210M, D = B tokens, respectively. (bottom) COMET score results for the 100% en-MC4 pretraining experiments in Figure 3 Same as the top row, except that the pretraining dataset is 100% en-MC4. For all the plots, the markers the actual experimental results and the black horizontal curves correspond to non- con-pretrained model directly trained on the task dataset. The finetuning dataset size increases in the order of the curves including the black horizontal lines.
for all
Table 3: The coefficients for the BLEU score law f(Dp) = (log(A-D))) for the results in Figure 2 (top). For the BLEU score laws, use 8 0.1 for the Huber Loss. We report log A instead of A since A typically takes very small and very large values.
Pretraining Dataset
Finetuning Dataset
Finetuning Dataset Size
log A
R
Prediction Error
50% en + 50% Me-MC4
WMT-17 en-de
6M
-180.75
9.00
0.75
0.034
50%
dC-MC4
10
50% 50% Mc-MC4
WMT-17 ea-de
3B
1.64 10a
9.91 x 108
0.19
0.048
50% en 50% frMC4
WMT-15 en-fr
42M
- 1.82 10*
8.98 x 102
0.12
0.061
50% cn 50% fr-C4
WMT-15 en-ft
210M
-2.33 10*
1.21 x 10
0.40
0.013
50% en 50% fr-M44
WMT-15 en-fr
5.08
4.61 x 10
0.16
0.005
50% 50% 1o-MC4
WMT-
625K
-36.02
1.28
50% 50% ro-MC4
WM en-ro
3M
-0.115.03
5.69
0.59
0.015
50% en + 50% ro-MC4
WM en-mo
312M
1.82 x
9.04 x 10P
0.40
0.015
19

---

Published as a conference paper at ICLR 2025
BLEU score on WNTF-15 en-fr
ELEU on WMT-17
104
10P
10
10
Opmdl tokans en-MG4 50% de-MC4 pretraining)
Dymll tokens en 50% fr MC4 pretraining)
42M
CE dationiion loss on WWT 15 en
10
10-11
validation
10P
10:4
1.0
101
10
2014
Opmdl takans (50% en-MC4 50% de-MC4 pretraining)
Dymal tolkens 150% en MC4 50% fr MC4 pretraining)
Figure (top) BLEU score vs pretraining dataset size: f(Dp) (log(A D (left) WMT-17
en-to-de translation task. Pretraining dataset has 50% en-MC4 + 50% de-MC4. Dotted and dashed blue curves correspond to the fitted scaling laws for different finetuning dataset sizes, D, = 6M and D2 31 M tokens, respectively. (right) WMT-15 en-to-fr translation task. Pretraining dataset has 50% en-MC4 and 50% fr-MC4. Dotted and dashed orange curves correspond to the fitted scaling laws for different finetuning dataset sizes, D, 42M and D, = 210M tokens, respectively. (bottom) Cross-entropy (CE) validation loss vs pretraining dataset size: L(Dp) E + D
Same models the top For all the plots, the markers the actual experimental results and the black horizontal curves correspond to the non non-pretr model directly trained on the task dataset. The finetuning dataset size increases in the order of dotted-dasher for all the curves including the black horizontal lines.
Table The coefficients for the downstream cross-entropy law L(D,) E +
for the results in
Do
Figure (bottom). For the downstream cross-entropy laws, we use = for the Huber Loss.
Pretraining Dataset
Finetuning Dataset
Finetuming Dataset Size
a
Prediction Error
50% dc-MC4
WMT-17
3.21
10
1.36
10 12
50% en 50% de MC4
WMT-17 en-de
31M
3.28
10
4.70 x 10a
0.78
3.17
10
WMT-17 en-de
2.24
10
10-2
14
50% 50% fr-CC4
WMT-15 en-fr
42M
2.72 10-3
2,01 x 10'
1.18
7.52
10
50% en 50%
frMC4
WMT-15 en-fr
210M
2.57
10
1.75 x 107
1.30
2.24
18
50% 50%
MC4
WM
21B
1.11 10
3.41
10
1.82 10 2
5.20
10
50% en 50% to MC
WMT-16 16TT-16aoo en-ro
625K
2.45
10
0.49
0.41
3.61
10
en-ro
2.22
10
2.19
10
50% on 50% mo-C4
WMT- 16 en-ro
312M
2.08
10-3
3.94
0.53
5.95
10-12
20

---

Published as a conference paper at ICLR 2025
BLEU on WMT-17
BLEU WWTT-1
10t
Cyc:
PMM
Dye
10"
10"
10
10
Dy=ff tokens (100% en-MC4 pretraining!
D==#f tokens (100% en MC4 pretreining)
CE validation loss on 35 en
Sy
on WPT-117
10
10m1
10"
10
10
10'
10
10
0==#
1100% pretraining!
Dy=#f
pretraining)
Figure 9: (top) BLEU score vs pretraining dataset size: f(D,) = (log(A D (left) WMT- 17 en-to-de translation task. Dotted and dashed red curves correspond to the fitted scaling laws for different finetuning dataset sizes, D, 6M and D, = 31 M tokens, respectively. (right) WMT-15 en-to-fr translation task. Dotted and dashed red curves correspond to the fitted scaling laws for different finetuning dataset sizes, D, 42M and D, 210M tokens, respectively. (bottom) Cross-entropy (CE) validation loss pretraining dataset size: L(Dp) E + Dę
A
Same models the top row. For all the plots, the markers are the actual experimental results and the black horizontal curves correspond to the non non-pretrained model directly trained on the task dataset. The finetuning dataset size increases the order dotted-dasher for all the curves including the black horizontal lines.
Table 5: The coefficients for the BLEU score law f(D,) = (oog(AA ()) for the results in Figure 3 (top). For the BLEU score laws, use 8 = 0.1 for the Huber Loss. We report log A instead of A since A typically takes very small and very large values.
Pretraining Dataset
Finetuning Dataset
Finetuning Datasct Size
log
3
Prediction Error
100% en-MC4
WMT-17 en-de
6M
0.014
100% en-MC4
WMT-17 en-de
31M
-1.81 x 10
896. 196.12
0.28
0.006
100% en-MC4
WMT-17 en-de
3B
1.02 10-7
104.92
0.42
0.015
100% en-MC4
42M
1.00
2.57 10
100% en-MC4
WMT-15 en-fr
210M
-6,38 x 107
3.43
102
0.20
0.034
100% MC-MC4
WMT- 15 en-fr
21B
204.81
3.80
10
9.97 10-3
0.004
100%
16
625K
-10.54
1.12
0.008
100% MC-
I6 en-ro
3M
-40.41
2.11
0.79
0.025
100% eB-MC4
WMT-16 en-ro
312M
3.61
x 10
0.19
0.018
21

---

Published as a conference paper at ICLR 2025
Table 6: The coefficients for the downstream cross-entropy law I(Dp) = F +
for the results in
D
Figure 3 (bottom). For the downstream cross-entropy laws, we use = 10-5 for the Huber Loss.
Pretraining Dataset
Finctuning Dataset
Finctuning Dataset Size
E
A
a
Prediction Error
100% en-MC4
WMT-17 en-de
x 10-13 13
3.18
10-3
0.15
5.79 10 12
100% en-MC4
WM en-de
31M
3.24 10-5
5.20
10-3
0.20
9.25 10-18
100% en-MC4
WMT-17 en-de
3B
2.24 x 10
2.56 x 10
0.36
5.70 x 10-14
100% en-MC4
WM1-15 en-fi
42M
3.49 10-3
1.03 10
0.25
3.63 10
100% MC4
WM
210M
4.24 10-5
19.39
0.66
5.40 10
100% en-MC4
WMI--1 ea-fr
21B
1.26 1077
2.59 10
4.81 10
3.63 10
100% en-MC4
WMT-16 en-ro
625K
5.79 x 10
1.03 10-3
7.76
5.56 10
100% en-MC4
WM en-mro
3M
1.78
9.98 10
8:33
8.23
100% en-MC4
WMT-16 cn-ro
312M
5.85 x 10-3
1.37 x 102
0.88
3.05 x 10-13
22


Reviews:
[{"review_id": "IuPd6pYxjL", "reviewer": "Reviewer_2Keq", "summary": "The paper proposes scaling laws for pre-training data and downstream data in the context of machine translation. The authors demonstrate that when the pre-training data is sufficiently aligned with the downstream task, both the downstream cross-entropy (CE) loss and translation quality adhere to the proposed scaling laws. However, if the pre-training data is not well-aligned, the downstream CE may still align with the scaling laws, but other metrics of translation quality may exhibit misalignment.", "questions": "N/A", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The paper conducts extensive experiments and present some interesting findings.", "weaknesses": "1. \"Inaccurate\" statement: The field of machine translation is increasingly adopting decoder-only architectures, whereas this paper focuses exclusively on encoder-decoder models. This focus leads to findings that conflict with those from SoTA decoder-only translation models. For example, in lines 355-359, the authors claim that \"there is no need to pretrain the models when the fine-tuning data is large enough.\" This assertion totally contradicts the findings in [1], which states that \"you only need a large amount of data for pre-training and a small number of data for continued fine-tuning.\" The key difference is that [1] employs a decoder-only model but it achieves top performance among other SoTA translation models. I am inclined to agree with the latter explanation because your training and test data are from the same domain (WMT). Training on WMT data usually yields better results on the WMT test set (especially BLEU), which might give the illusion that using only fine-tuning data is sufficient. However, this approach may lead to poor generalization abilities. A large training dataset may have less impact on in-domain test data, but it's essential to ensure that the model generalizes well to out-of-domain data. Therefore, the authors should also investigate the model's performance on other out-of-domain test datasets to provide a more comprehensive evaluation before make this statement.\n\n2. Out-of-date metric: Most of the findings in this paper rely on BLEU scores. However, BLEU is increasingly regarded as an inaccurate metric that does not align well with human judgments, especially when compared to other metrics like COMET. The inclusion of ROUGE also seems weird, as ROUGE is primarily used for summarization evaluation rather than translation. While it might be applicable in certain cases, it is not a mainstream choice for assessing translation quality. The authors should consider using more suitable metrics, such as BLEURT, to provide a more accurate evaluation of their models.\n\n3. Unclear writing: While this is a minor issue, the paper's writing feels disjointed and contains redundant information. There are several instances where the same points are reiterated, such as \"when the distribution is well-aligned, both metrics and CE align with the scaling laws.\" Moreover, the frequent references to figures that are located several pages away disrupt the reading flow and make it difficult to follow the arguments being presented.\n\n4. Lack of definition: I struggled to understand what the authors mean by the \"distribution of pre-training and downstream task.\" It wasn't until midway through the paper that I realized this likely refers to the amount of monolingual target language data included in the pre-training dataset—where more target language data equates to a more \"aligned distribution.\" One question here, if the authors were to conduct experiments on another low-resource language like Icelandic, where the fine-tuning data consists of a small amount of news content but the available pre-training monolingual data is primarily from games, the \"distribution/alignment\" becomes ambiguous. Alternatively, one might employ cross-lingual learning using a high-resource language like German, for which abundant news data is available. In this scenario, which dataset constitutes a \"more aligned distribution\"—the Icelandic data from a different domain or the German data from the same domain? I believe that the term \"distribution\" is too abstract and makes the paper challenging to understand. Providing a more precise definition or elaborating on this concept would greatly enhance the clarity of the paper.\n\n5. A narrow scope in high-resource languages: The focus on high-resource languages is not particularly compelling, as SoTA models already perform exceptionally well in these languages, regardless of scaling laws. People can simply amass large amounts of data for training to achieve excellent results. The paper's emphasis on utilizing billions of tokens for pre-training and millions of parallel sentences for fine-tuning is not practical for many mid- or low-resource languages, where those languages do not have the resource.  A more interesting and valuable direction would be to explore scaling laws that examine how high-resource languages can be used to support low-resource languages. Investigating how data from other languages can enhance translation quality in languages with limited resources would address a critical need in the field of MT and contribute to more inclusive and effective translation models.\n\nUpdate: I just found that I forgot adding the references:\n\nReferences:\n\n[1] A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YFeFKRxxMN", "reviewer": "Reviewer_tpup", "summary": "Past work on scaling laws has mostly focused on pretraining loss rather than downstream transfer learning task performance. This paper attempts to establish scaling laws for transfer learning for machine translation. The paper specifically investigates how pretraining data size affects downstream translation performance, measured in terms of downstream cross-entropy loss as well as in terms of downstream machine translation metrics like BLEU. Results show that that size of finetuning data, as well as the similarity between the pretraining and finetuning datasets, have a large impact on downstream results and scaling behavior. Most interestingly, the results indicate that when pretrain and finetune are well enough aligned, there is a clear log scaling law for pretraining size on downstream machine translation metric performance. However, when pretrain and finetune are dissimilar, more pretraining data can sometimes even hurt performance on downstream machine translation metrics. This stands in contrast with downstream cross-entropy, which improves monotonically with pretrain size regardless of pretrain/finetune mismatch.", "questions": "see above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper seeks empirical answers to important questions -- scaling laws (even rough ones) for downstream transfer can provide useful guidance: When should one seek out different pretraining data vs more pretraining data? How much finetuning data is sufficient for good downstream performance?\n\nThe paper focuses specifically on machine translation -- an important NLP task that has not been the main focus of prior transfer learning scaling law work. This is a strength. The application focus lets the paper go into more depth in empirical analysis and leads to clearer takeaways. \n\nThe paper concretely shows that downstream cross-entropy can be misleading. The paper finds that downstream cross-entropy monotonically improves with pretrain size, even when other machine translation metrics like BLEU do not. This is a useful result for future researchers.", "weaknesses": "Aspects of the presentation could be improved. Specifically, as a reader, I found it confusing that the paper essentially starts with a results / takeaways section before discussing experimental details. Then an experimental details section comes next, followed by a results / takeaways section. I think a more traditional ordering could be more effective here -- the intro could be used to discuss high-level takeaways up front, leaving the rest of the detailed results to be described after experimental setup is clear. Without having a clear understanding of experimental setup, I was left wondering in many cases how to interpret the early results discussion. \n\nThe one experimental weakness that stood out to me: It's not entirely clear how \"alignment\" between pretrain and finetune is being defined. For the most part, it seems to mean in the context of this paper whether and to what extend that datasets share languages. This could be further clarified and formalized -- but, further, more nuanced measures of alignment could easily be reported. These might help clarify experimental takeaways having to do with alignment, which represent some of the more interesting results in this paper. \n\nMinor:\n\n-Discussion of results on other tasks is odd given that paper is focused on MT.\n\n-I couldn't follow remark 2 -- what was the main takeaway / hypothesis? Consider rephrasing this section for improved clarity. \n\n-\"3.4 A Guide for ...\" section is kind of confusing -- especially part 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GaHN3fogq4", "reviewer": "Reviewer_f7Dz", "summary": "The paper analyses to what extent scaling laws apply if the pre-training and downstream data are well or less well aligned. They find that iff the data is less aligned, task-specific metrics can fluctuate when increasing the pre-training data, while increasing more monotonically if the data is well aligned. Their findings are mostly based on machine translation experiments but also generalize to more general tasks such as the tasks in the SuperGlue benchmark set.", "questions": "To what extent do you expect your findings to carry over to decoder-only models more commonly found in current LLMs?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- well written paper\n- addresses a problem that is relevant for a large community\n- thourough empirical analysis", "weaknesses": "Although the concept of alignement is central to this work and many times the authors refer to \"degree of\" or \"not sufficient\" alignment, no definition of alignment is given. I understand that this is not trivial but I would appreciate at least an attempt to provide a---ideally formal---definition that could also be used to quantify alignment. Currently, it seems the only way is to determine the degree in hindsight (see lines 227--228) by failing to fit the scaling law.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z4mBjTbkid", "reviewer": "Reviewer_nK1N", "summary": "The paper investigates the scaling behavior of downstream performance metrics, including translation scores and cross-entropy loss, as a function of the size of the pretraining dataset in machine translation tasks. The authors conduct experiments using T5 models trained on different subsets of the Multilingual C4 dataset and fine-tuned on various translation tasks. They find that when the pretraining and downstream tasks have well-aligned distributions, both translation scores and cross-entropy loss improve monotonically with more pretraining data. However, when the distributions are not well-aligned, translation scores can exhibit non-monotonic behavior, while cross-entropy loss still improves monotonically. The authors propose a practical guide for evaluating the value of pretraining data for translation tasks based on downstream translation scores and demonstrate the effectiveness of their approach on multiple natural language processing tasks.", "questions": "The alignment between the pre-training and downstream task is crucial to understanding the scaling law. I am not entirely clear on what “align” means in this context. Could you please explain in detail the meaning of “align” in this work?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper's main contribution is its in-depth investigation of the impact of pre-training data on machine translation tasks and the proposal of a new logarithmic law to describe the change in downstream performance as the size of the pre-training data increases. This research is helpful to understand the performance of large-scale language models in specific downstream tasks and provides a new method for evaluating the value of pre-training data. From the perspectives of innovation and importance, the paper is highly valuable. Additionally, the quality and clarity of the paper are good, with detailed descriptions of experimental design and result analysis, and abundant charts and empirical evidence to support conclusions.", "weaknesses": "One potential weakness of the paper is it is focusing solely on downstream performance changes in machine translation tasks. This limits the comprehensive evaluation of the applicability of the logarithmic law to a wider range of natural language processing tasks. \nFor NLP tasks that use precision and recall as metrics, can these be effectively aligned with the pre-training procedure? Additionally, can cross-entropy be generalized to observe the scaling law?\n\nIt is recommended to include a reference to the inference scaling law from OpenAI’s o1 model and to discuss the scaling law within a broader framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VXqBX0ccTt", "reviewer": "Reviewer_9RdU", "summary": "This paper investigates scaling laws for downstream task performance in machine translation, specifically examining how pretraining dataset size affects translation quality after finetuning. The authors propose a novel log-law to describe the scaling behavior of translation quality metrics (BLEU, ROUGE, COMET) and demonstrate that the relationship between pretraining data size and downstream performance heavily depends on distribution alignment and finetuning dataset size. A key finding is that while cross-entropy consistently follows power-law scaling, translation quality metrics may deviate from expected scaling patterns when pretraining and downstream distributions are not well-aligned.", "questions": "- This is in not a critique but I'm curious about your thoughts. You study pre-training with up to 2 languages. How do you think your findings translate to the more realistic setting of pre-training on a large number of languages, with severe misalignments in data size between the languages?\n\n- Similarly, do you have any findings related to language similarity?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Exceptionally well-grounded in literature, with relevant prior work effectively woven throughout the paper's narrative.\n- Strong experimental methodology with fully controlled experiments that isolate different variables' effects. In contrast to a lot of recent LLM works, the experiments are fully controlled which I appreciate a lot. The paper demonstrates comprehensive validation across multiple metrics and scenarios, with clear empirical support for the proposed log-law.\n- The writing is clear and the structure effective. Clear and systematic presentation of results with thorough analysis.\n- The proposed log-law for translation quality metrics is well-motivated and comprehensively validated.\n- Makes important observations challenging common assumptions about cross-entropy and translation quality metrics.", "weaknesses": "- Limited to pairwise language combinations in pretraining - would benefit from discussion about generalization to realistic multilingual pretraining scenarios.\n\n- It would be a great addition to have a short discussion that connects these findings to recent successes in LLM-based MT. For instance, some works like https://aclanthology.org/2024.tacl-1.32/ and https://aclanthology.org/2024.acl-long.336/ fine-tuned directly on parallel data, but these LLMs are heavily English-centric and therefore there is a low distribution alignment. More recent work like ALMA https://openreview.net/forum?id=farT6XXntP and TowerLLM https://openreview.net/forum?id=EHPns3hVkj#discussion circumvent this issue by continuing pre-training on multilingual data before fine-tuning on parallel data.\n\n- Minor  issues\n    - Lines 142-143: \"Tran et al., 2019\" is repeated 3 times\n    - Lines 264-265: it'd be nice to include fine-tuning dataset sizes here", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)