Scaling Laws for Downstream Task Performance in Machine Translation

## Abstract

Abstract Scaling laws provide important insights that can guide the design of large language models (LLMs). Existing work has primarily focused on studying scaling laws for pretraining (upstream) loss. However, in transfer learning settings, in which LLMs are pretrained on an unsupervised dataset and then finetuned on a downstream task, we often also care about the downstream performance. In this work, we study the scaling behavior in a transfer learning setting, where LLMs are finetuned for machine translation tasks. Specifically, we investigate how the choice of the pretraining data and its size affect downstream performance (translation quality) as judged by two metrics: downstream cross-entropy and BLEU score. Our experiments indicate that the size of the finetuning dataset and the distribution alignment between the pretraining and downstream data significantly influence the scaling behavior. With sufficient alignment, both downstream cross-entropy and BLEU score improve monotonically with more pretraining data. In such cases, we show that it is possible to predict the downstream BLEU score with good accuracy using a log-law. However, there are also cases where moderate misalignment causes the BLEU score to fluctuate or get worse with more pretraining, whereas downstream cross-entropy monotonically improves. By analyzing these observations, we provide new practical insights for choosing appropriate pretraining data.

## 1 Introduction

Scaling laws quantify the relationship between a model’s performance and key design factors such as the size of the training data or the model’s architecture. In the context of LLMs, these laws offer valuable guidance for model development, resource allocation, and selection of appropriate training data. Extensive research has focused on scaling laws for upstream perplexity or cross-entropy loss (i.e., evaluated on pretraining data), demonstrating that these quantities can be well predicted using power laws (Kaplan et al.,, ; Hoffmann et al.,, ; Gordon et al.,, ; Hernandez et al.,, ; Fernandes et al.,, ; Bansal et al.,, ; Henighan et al.,, ; Johnson et al.,, ). However, in practical applications, LLMs often undergo transfer learning–they are first pretrained on unsupervised data and then finetuned for specific downstream(^2^22We use the term downstream to refer to the finetuning task or metrics computed on it, and the term upstream to refer to the metrics computed on the pretraining dataset.) tasks such as coding or translation. The question of whether scaling laws can be used to predict downstream task performance is critical, yet remains largely unanswered (Hernandez et al.,, ; Tay et al.,, ). Here, the term *task performance* refers to metrics that measure task-related quantities such as accuracy and BLEU score, which are different from next-token prediction metrics such as cross-entropy.

In this work, we study scaling laws for transfer learning and focus on machine translation tasks. Specifically, we look into the relation between the pretraining dataset size and the *downstream task performance* after finetuning on the task.
We find that, in addition to the finetuning data size and the choice of the performance metric, this relation fundamentally depends on the alignment between the pretraining data and the downstream task.
While similar observations have been made in different contexts in the transfer learning literature (Tamkin et al.,, ; Agostinelli et al.,, ), our work provides new insights and concrete scaling laws for the downstream performance of LLMs.

We carry out systematic experiments in which we pretrain LLMs on multilingual unsupervised datasets and then finetune them on several machine translation tasks. Across the experiments, we vary the type of pretraining data (to control the degree of distribution alignment with the downstream task) and the finetuning data size. We study two metrics: *downstream* BLEU (Papineni et al.,, ) score(^3^33In the rest of the paper, we will drop “downstream” when we refer to the downstream BLEU score.) and *downstream* cross-entropy. We find that in settings where the distributions are well-aligned, both BLEU and *downstream* cross-entropy improve monotonically with more pretraining. In these settings, we demonstrate that the BLEU score can be well predicted using the following log-law: $f(D_{p})=(\log(A\cdot D_{p}^{\alpha}))^{\beta}$, where $D_{p}$ denotes the size of the pretraining data, and $A$, $\alpha$, $\beta$ are the coefficients to be fit. We further propose a power-law $L(D_{p})=E+\frac{A}{D_{p}^{\alpha}}$ for the *downstream* cross-entropy as the pretraining data scales – echoing similar laws developed for the *upstream* cross-entropy as a function of the pretraining dataset size (Kaplan et al.,, ; Hoffmann et al.,, ) and *downstream* cross-entropy as a function of the finetuning dataset size (Hernandez et al.,, ).

However, when distributions are not sufficiently aligned and the finetuning data size is relatively small, we find that there are cases where the BLEU score exhibits an unclear, non-monotonic behavior, whereas the *downstream* cross-entropy still improves monotonically following a power-law. This observation suggests that using cross-entropy as a proxy for task-related metrics like BLEU score may lead to critical misjudgments in practice if used to make decisions about the “relevance” of the pretraining data for the downstream task or the required size of the pretraining data for the target downstream performance.

Finally, our empirical studies suggest that pretraining brings little to no improvement on the BLEU score when the finetuning (translation) dataset is already large enough, complementing the findings of Hernandez et al., ().

Our contributions and main findings can be summarized as:

- •
We carry out systematic experiments on 770-million and 3-billion encoder-decoder T5  (Raffel et al.,, ) models to study how downstream performance, measured by *downstream* cross-entropy and BLEU score, scales with the pretraining dataset size. For pretraining, we experiment with different subsets of the Multilingual C4 (MC4) dataset (Raffel et al.,, ), including English (en), German (de), French (fr), and Romanian (ro). For finetuning, we study the following translation tasks: WMT-17 en-de (Bojar et al.,, ), WMT-15 en-fr (Bojar et al.,, ), and WMT-16 en-ro (Bojar et al.,, ).
- •
We observe that, when the distributions of the pretraining and downstream tasks are well-aligned, the BLEU score and *downstream* cross-entropy improve monotonically with more pretraining. For BLEU score, we propose a new log scaling law and show that it has good predictive accuracy.
- •
When the distributions are not sufficiently aligned and the finetuning data size is relatively small, the BLEU score fluctuates or even gets worse with more pretraining–losing the monotonic scaling behavior. In these same settings, we find that the *downstream* cross-entropy still scales monotonically according to a power-law.
- •
We argue that the value of pretraining data should be evaluated using downstream task-related metrics like BLEU score and propose a practical guide for such an assessment by leveraging the proposed scaling law for BLEU score.

## 2 Related Work

#### Scaling laws for transformers.

Scaling laws for LLMs have attracted significant attention as they can inform the decisions about key design choices such as model size and the type and size of the pretraining data (Kaplan et al.,, ; Hoffmann et al.,, ; Hernandez et al.,, ). Most of the pioneering work has focused on how *upstream* cross-entropy loss or perplexity scales with more pretraining data, larger models, or longer training (Kaplan et al.,, ; Hoffmann et al.,, ). Follow-up works have analyzed scaling behavior of translation models (Ghorbani et al.,, ; Zhuocheng et al.,, ; Gordon et al.,, ; Fernandes et al.,, ; Bansal et al.,, ; Zhang et al.,, ), studied theoretical foundation behind scaling laws (Sharma and Kaplan,, ; Hutter,, ; Bahri et al.,, ), or extended the laws to the vision models (Zhai et al.,, ; Jain et al.,, ). Closest to our work, Hernandez et al., () have analyzed transfer learning but with a focus on how the cross-entropy loss behaves as the *finetuning* data scales. Unlike our work, their scaling law describes the relation between the size of a (finetuning) dataset and the cross-entropy loss on the *same* dataset – making this closer to the standard scaling laws in the literature since the finetuning loss and the finetuning dataset are computed over samples from the same distribution. On the other hand, we propose scaling laws for the *downstream* metrics on the *finetuning* dataset as the *pretraining* data scales – switching the focus to an “out-of-distribution” analysis. The only work we are aware of that has proposed scaling laws for the *downstream task performance* as a function of pretraining dataset size is by Sun et al., () who have focused on classification tasks in the vision domain and used small models relative to LLMs.

#### Transferability metrics and value of pretraining.

While it may be commonly suggested that pretraining data improves both *upstream* and *downstream* performance, this rule has been challenged in the vision domain. Zoph et al., (); He et al., (); Shen et al., (); Ghiasi et al., (); Mikami et al., () have demonstrated that pretraining can sometimes have no effect on the *downstream* task performance and sometimes it can even hurt the performance. We make similar observations in the language domain with extensive experiments on LLMs and identify cases where (a) adding more pretraining data hurts the *downstream task performance* when pretraining data is not aligned enough with the task and (b) pretraining does not improve the *downstream task performance* noticeably when the finetuning dataset is large enough. Another related line of work is on transferability metrics (Tamkin et al.,, ; Chiang and Lee,, ; Ibrahim et al.,, ; Tran et al.,, ; Agostinelli et al.,, ; Tran et al.,, ; Nguyen et al.,, ; You et al.,, ; Dai et al.,, ; Huang et al.,, ; Ibrahim et al.,, ; Tran et al.,, ; Bao et al.,, ; Van Asch and Daelemans,, ; Plank and Van Noord,, ), which are efficient heuristics used to select the most appropriate source models or pretraining data for a given target task. We note that transferability metrics are designed to solve ranking problems, different from scaling laws. For example, these metrics answer questions such as given a pool of source models (or pretraining datasets), which source model (or pretraining dataset) is the best to finetune on for a given target task. These metrics are not designed to predict the performance of the model when key quantities (e.g., pretraining data size) are scaled.

## 3 Scaling Laws for Transfer Learning

In this section, we present our proposed scaling laws for BLEU score and *downstream* cross-entropy. We also discuss when these laws apply and provide practical guidance for assessing the value of a pretraining dataset for a given target downstream task. The details of the experimental results will be later discussed in Section [5](#S5).

### 3.1 A Scaling Law for the BLEU Score

Different from cross-entropy and perplexity, which follow a power-law scaling behavior Kaplan et al., (); Hoffmann et al., (), we find out that BLEU score scales closer to a log-law, as evident from Figures [1](#S4.F1), [2](#S5.F2), and [3](#S5.F3). Therefore, we propose the following scaling law for BLEU score as a function of the pretraining dataset size $D_{p}$:

$$ $\displaystyle f(D_{p})=(\log(A\cdot D_{p}^{\alpha}))^{\beta},$ (1) $$

where $A$, $\alpha$, and $\beta$ are coefficients to be fit. We notice that these coefficients depend on how aligned the pretraining dataset with the target downstream task (translation from language 1 to language 2) and how large the finetuning (translation) dataset is. With extensive experiments across several translation tasks and multilingual pretrained models, we demonstrate that the law in ([1](#S3.E1)) indeed well describes BLEU score scaling, with a small prediction error which we quantify in Appendix [B.2](#A2.SS2).

### 3.2 Is Cross-Entropy Loss Always a Good Metric?

We also compare the *downstream* cross-entropy loss and the BLEU score empirically as prior work has made the assumption that *upstream* or *downstream* cross-entropy loss is a good indicator for a model’s *downstream task performance*. Following the well-understood scaling behavior of the *upstream* cross-entropy loss as a function of the pretraining dataset size Kaplan et al., (); Hoffmann et al., (), we demonstrate that the same scaling law can also describe the *downstream* cross-entropy loss as

$$ $\displaystyle L(D_{p})=E+\frac{A}{D_{p}^{\alpha}},$ (2) $$

where $E$, $A$, and $\alpha$ are the coefficients to be optimized. Throughout the paper, we report BLEU score and cross-entropy together for a direct comparison and discover several cases where the two metrics do not correlate well. This supports some of the findings of Ghorbani et al., () suggesting inconsistency between the BLEU score and the cross-entropy, but also shows that the exponential relationship (between the two metrics) advocated by Gordon et al., () does not always hold. More specifically, our empirical results show that while cross-entropy loss always monotonically decreases (with appropriate learning rate) as the pretraining dataset size increases, BLEU score may show a non-monotonic trend when the pretraining data is not sufficiently aligned with the task. For instance, in Figure [3](#S5.F3)-(top, right), increasing the en-MC4, de-MC4, or ro-MC4 pretraining datasets’ size sometimes decreases the BLEU score on WMT-15 English-to-French (en-fr) translation task. Even though they may initially follow the law in ([1](#S3.E1)) for smaller pretraining dataset sizes, the scaling law breaks for larger data for these datasets and task. Overall, the BLEU score never reaches a good value compared to other pretraining datasets that include some amount of French – indicating that pretraining datasets that do not include French are not aligned enough with this particular translation task. However, if we were to look at only the cross-entropy loss in Figure [3](#S5.F3)-(bottom, right), we would conclude that all the pretraining datasets bring noticeable improvements to the model and they all are worth adding into the pretraining data – which would be a poor decision.

A remotely related observation on the mismatch between the task-related metrics and the cross-entropy by McKenzie et al., (), who looked at how the *downstream task performance* changes as the model grows, suggests that LLMs may show worse task performance with increased model size but, similar to our findings, this is not captured by the monotonically decreasing cross-entropy loss.

### 3.3 When Do Scaling Laws Fall Short in Transfer Learning?

While the cross-entropy loss always follows a monotonically decreasing trend which can be captured by the scaling law in ([2](#S3.E2)), we do not always see a monotonic increase in the BLEU score when increasing the pretraining dataset size (see Figure [2](#S5.F2)-(top, center) and Figure [3](#S5.F3)-(top, right)). We observe that this only happens when the pretraining dataset is not sufficiently aligned with the translation task – which results in low BLEU scores overall compared to models that were pretrained in other datasets. For the pretrained models that lead to high BLEU scores after finetuning, we consistently see that the BLEU score increases monotonically and can be well described with the scaling law in ([1](#S3.E1)). Therefore, whether the scaling law could fit the empirical BLEU scores or not could be a good first-check in assessing the value of pretraining data for the downstream (translation) task. We elaborate more on this in the next section.

### 3.4 A Guide for Pretraining Data Valuation

Finally, combining our findings on the scaling behavior of BLEU score, we propose the following guide for assessing the value of pretraining dataset for a target downstream task:

- 1.
Given a pretraining dataset, pretrain as long as possible under the given computational and time constraints(^4^44We avoid repeating sequences as repetitions may complicate the scaling behavior (Hernandez et al.,, ; Muennighoff et al.,, ; Tirumala et al.,, ). This means as pretraining goes on, we effectively pretrain each checkpoint on a “larger dataset”.). Periodically choose pretraining checkpoints, finetune on them, and record the downstream performance metric (we recommend the BLEU score over cross-entropy due to the discussion in Section [3.3](#S3.SS3)).
- 2.
Since the law in ([1](#S3.E1)) has three coefficients to be fit, once we have 3 pairs of (number of pretraining tokens seen, BLEU score), we *try* to find the optimal coefficients. If the BLEU scores have a non-monotonic behavior, we cannot fit the scaling law. Since the non-monotonic behavior could be an indication of misalignment (following the discussion in Section [3.3](#S3.SS3)), we recommend checking the BLEU score of the best available finetuned checkpoint and comparing it to the performance of the non-pretrained model trained on the downstream task directly. If the scaling law fits well, then we make the initial prediction for the BLEU score as we increase the pretraining dataset size (or pretrain for more steps). If we are not satisfied with the predicted BLEU score, then we conclude that it is not worth pretraining on this dataset. If the predicted BLEU score is high enough, then we keep pretraining until we reach the target BLEU score. If the scaling law breaks at any point, we conclude that the pretraining dataset is not sufficiently aligned with the downstream task and pretraining further may not be beneficial.

## 4 Experimental Setup

In the experiments, we first pretrain a model without doing more than one pass over any of the examples. Then, we finetune selected checkpoints of the pretrained model. Naturally, there is a one-to-one mapping between the checkpoint number and the number of pretraining tokens seen. This way, we collect pairs of (number of pretraining tokens, BLEU score) and (number of pretraining tokens, *downstream* cross-entropy loss) to analyze them with the proposed scaling laws in ([1](#S3.E1)) and ([2](#S3.E2)). All the plots are on a $\log$-$\log$ scale.

#### Model.

We use the 3-billion encoder-decoder T5 model with $24$ encoder layers, $24$ decoder layers, embedding dimension $1024$, and $32$ heads with dimension $128$. We note that this is the same model as the T5-3B model in Abnar et al., (). In Appendix [B](#A2), we also provide results with a smaller 770-million encoder-decoder T5 model. This model corresponds to T5-Large in Raffel et al., (). We share more details about the architectures in Appendix [A](#A1). For encoding the text as WordPiece tokens (Sennrich et al.,, ; Kudo,, ), we use SentencePiece (Kudo and Richardson,, ) trained with a vocabulary of size $250,112$ that covers all the languages in the MC4 dataset (Raffel et al.,, ).

#### Datasets.

We use the English (en), German (de), French (fr), and Romanian (ro) portions of the MC4 dataset. We experiment with both pretraining on these languages individually as well as mixing pairs of languages. In Figure [1](#S4.F1), we present results for the models pretrained on (left) a mixture of $50\%$ en-MC4 + $50\%$ de-MC4, (center) a mixture of $50\%$ en-MC4 + $50\%$ fr-MC4, and (right) a mixture of $50\%$ en-MC4 + $50\%$ ro-MC4 – meaning that $50\%$ of one pretraining batch is sampled from en-MC4 and the other $50\%$ is sampled from the other language. In Figure [2](#S5.F2), we show results for the models pretrained only on en-MC4. In Figure [3](#S5.F3), in addition to these, we also present results for the models pretrained on a mixture of $30\%$ en-MC4 + $70\%$-fr and a mixture of $70\%$ en-MC4 + $30\%$-fr as well as models pretrained only on de-MC4, only on fr-MC4, and only on ro-MC4. We finetune the pretrained models on WMT-17 en-de (Bojar et al.,, ), WMT-15 en-fr (Bojar et al.,, ), and WMT-16 en-ro (Bojar et al.,, ), separately. To understand the effect of the finetuning dataset size on the scaling laws, we sometimes use a smaller randomly sampled portion from these translation datasets and indicate the number of tokens used.

#### Hyperparameters.

During pretraining, we use a batch size of 256 and a sequence length of 512 for $1,000,000$ steps except for the ro-MC4 pretraining. For ro-MC4, we pretrain for $510,000$ steps since otherwise, we would need to do repetitions over the sequences. Following Raffel et al., (), we use an “inverse square root” learning rate schedule, $\frac{1}{\sqrt{\max(n,k)}}$, where $n$ is the current pretraining step and $k$ is set to $10^{4}$. We do a grid search for the base learning rate from $\{0.05,0.1,0.5,1.0,2.0,5.0\}$ and pick the best one for each pretrained model based on *upstream* cross entropy. During finetuning, again following Raffel et al., (), we use a batch size of $128$ and a sequence length of $512$ for $300$ steps. We use a constant learning rate by selecting the best from $\{0.001,0.005,0.01,0.05,0.1\}$. In both stages, we use the AdaFactor optimizer (Shazeer and Stern,, ).

Figure: Figure 1: (top) BLEU score vs pretraining dataset size: $\mathbf{f(D_{p})=(\log(A\cdot D_{p}^{\alpha}))^{\beta}.}$ (left) WMT-17 en-to-de translation task. Pretraining dataset has $50\%$ en-MC4 + $50\%$ de-MC4. Dotted, dashed, and solid blue curves correspond to the fitted scaling laws for different finetuning dataset sizes, $D_{f}=6M$, $D_{f}=31M$, $D_{f}=3B$ tokens, respectively. (center) WMT-15 en-to-fr translation task. Pretraining dataset has $50\%$ en-MC4 and $50\%$ fr-MC4. Dotted, dashed, and solid orange curves correspond to the fitted scaling laws for different finetuning dataset sizes, $D_{f}=42M$, $D_{f}=210M$, $D_{f}=21B$ tokens, respectively. (right) WMT-16 en-to-ro translation task. Pretraining dataset has $50\%$ en-MC4 + $50\%$ ro-MC4. Dotted, dashed, and solid green curves correspond to the fitted scaling laws for different finetuning dataset sizes, $D_{f}=625K$, $D_{f}=3M$, $D_{f}=312M$ tokens, respectively. (bottom) Cross-entropy (CE) validation loss vs pretraining dataset size: $\mathbf{L(D_{p})=E+\frac{A}{D_{p}^{\alpha}}}.$ Same models as the top row. For all the plots, the markers are the actual experimental results and the black horizontal curves correspond to the non-pretrained model directly trained on the task dataset. The finetuning dataset size increases in the order of dotted-dashed-solid for all the curves including the black horizontal lines.
Refer to caption: /html/2402.04177/assets/figures/wmt17_en_de_50_50_bleu.png

#### Optimizing the scaling law coefficients.

To fit the coefficients in the scaling laws in ([1](#S3.E1)) and ([2](#S3.E2)), similar to Hoffmann et al., (), we use the Huber loss (Huber,, ) and
the L-BFGS algorithm (Nocedal,, ) to estimate the scaling law robustly in the presence of outliers. For the Huber loss, we use $\delta=0.1$ for the BLEU score and $\delta=1e-3$ for the *downstream* cross-entropy loss. We select the best fit among a grid of initializations and report the prediction error computed via the Huber loss in Appendix [B.2](#A2.SS2). To optimize the coefficients, we use the first four data points that require the smallest amount of pretraining data and leave the remaining data points as held-out data to evaluate the accuracy of the laws. We note that, ideally, three points should be enough since both laws have three coefficients to be optimized for. However, adding more points improves the fit by making the optimization more robust to outliers. We provide more details about how to optimize the scaling law coefficients in Appendix [A.2](#A1.SS2). We refer the reader to Appendix [B.2](#A2.SS2) for the list of optimized coefficients and the prediction errors for each law we present in the next section.

## 5 Results and Analysis

Figure: Figure 2: (top) BLEU score vs pretraining dataset size: $\mathbf{f(D_{p})=(\log(A\cdot D_{p}^{\alpha}))^{\beta}.}$ (left) WMT-17 en-to-de translation task. Dotted, dashed, and solid red curves correspond to the fitted scaling laws for different finetuning dataset sizes, $D_{f}=6M$, $D_{f}=31M$, $D_{f}=3B$ tokens, respectively. (center) WMT-15 en-to-fr translation task. Dotted, dashed, and solid red curves correspond to the fitted scaling laws for different finetuning dataset sizes, $D_{f}=42M$, $D_{f}=210M$, $D_{f}=21B$ tokens, respectively. (right) WMT-16 en-to-ro translation task. Dotted, dashed, and solid red curves correspond to the fitted scaling laws for different finetuning dataset sizes, $D_{f}=625K$, $D_{f}=3M$, $D_{f}=312M$ tokens, respectively. (bottom) Cross-entropy (CE) validation loss vs pretraining dataset size: $\mathbf{L(D_{p})=E+\frac{A}{D_{p}^{\alpha}}}.$ Same models as the top row. For all the plots, the markers are the actual experimental results and the black horizontal curves correspond to the non-pretrained model directly trained on the task dataset. The finetuning dataset size increases in the order of dotted-dashed-solid for all the curves including the black horizontal lines.
Refer to caption: /html/2402.04177/assets/figures/wmt17_en_de_100_0_bleu.png

In Figure [1](#S4.F1), we analyze the models that are pretrained on different portions of (left) a mixture of $50\%$ en-MC4 + $50\%$ de-MC4, (center) a mixture of $50\%$ en-MC4 + $50\%$ fr-MC4, and (right) a mixture of $50\%$ en-MC4 + $50\%$ ro-MC4. These models are then finetuned on different portions of (left) en-de, (center) en-fr, and (right) en-ro translation datasets. In the top row, we report the BLEU score and, in the bottom row, we report the *downstream* cross-entropy loss. The dotted, dashed, and solid lines correspond to the scaling laws in ([1](#S3.E1)) and ([2](#S3.E2)) for different finetuning dataset sizes $D_{f}$. The black lines correspond to “non-pretrained” models (randomly initialized) that are directly trained on different portions of the finetuning dataset. In all cases, the scaling laws fit well to the empirical results (the markers) with prediction error at most $0.061$ for the BLEU score ($\delta=0.1$) and $5.95e-12$ for the *downstream* cross-entropy ($\delta=1e-3$) (see Appendix [B.2](#A2.SS2) for more details). As expected, as the finetuning dataset size increases (e.g., going in the order of dotted-dashed-solid lines), the BLEU score increases and the cross-entropy loss decreases smoothly and monotonically. Similarly, as the pretraining dataset size $D_{p}$ increases (along the x-axis), we see improvements in both metrics. Notice that the improvements by an increase in the pretraining dataset size is more effective for smaller finetuning datasets. When the finetuning dataset is large enough (e.g., solid lines), BLEU score is more or less constant regardless of the pretraining dataset size. In fact, we see little to no improvement of pretraining compared to the non-pretrained models (black lines) when the finetuning dataset is large. This implies that, for these tasks, there is no need to pretrain the models when the finetuning dataset is large enough. Luckily, we can correctly predict whether this is going to be the case (i.e., whether the available finetuning data is enough to eliminate pretraining altogether) with the use of scaling laws. All we need to do is to pretrain the model on a small portion of the pretraining dataset with reasonable compute cost to optimize the coefficients of the scaling laws, and then follow the guideline provided in Section [3.4](#S3.SS4).

In Figure [2](#S5.F2), we change the pretraining dataset to $100\%$ en-MC4 in all plots. Intuitively, we expect this dataset to be less aligned with the translation tasks than the multilingual pairs in Figure [1](#S4.F1) since it does not include one of the languages in the translation tasks. Indeed, we see smaller BLEU score and higher cross-entropy loss in general for the same finetuning dataset size. Most of the conclusions from Figure [1](#S4.F1) carry over to the results in Figure [2](#S5.F2). For instance, the pretraining data matters less when the finetuning dataset is large enough. One noticeable difference is in the BLEU scores for the en-fr translation task (center). We see that, for $D_{f}=42M$ and $D_{f}=210M$, the scaling law for BLEU score actually breaks once the pretraining dataset size passes a threshold while the cross-entropy loss scales as expected. This is counter-intuitive because the BLEU score sometimes decreases for larger pretraining dataset. Notice that this break in scaling law does not happen in en-de or en-ro translation tasks as the scaling law fits well to the pretraining data with prediction error at most $0.025$ for these tasks ($\delta=0.1$). To better investigate this, in Figure [3](#S5.F3), we take a closer look at some less aligned pretraining datasets due to the choice of language.

Figure: Figure 3: Comparison of scaling behavior for different pretraining datasets. (top) BLEU score vs pretraining dataset size: $\mathbf{f(D_{p})=(\log(A\cdot D_{p}^{\alpha}))^{\beta}.}$ (left) WMT-17 en-de translation task. (right) WMT-15 en-fr translation task. (bottom) Cross-entropy (CE) validation loss vs pretraining dataset size: $\mathbf{L(D_{p})=E+\frac{A}{D_{p}^{\alpha}}}.$ Same as the top row but for CE loss instead of BLEU score. For all the plots, the markers are the actual experimental results and the black horizontal curves correspond to the non-pretrained model directly trained on the task dataset.
Refer to caption: /html/2402.04177/assets/figures/wmt17_en_de_02_compare_pretraining.png

In Figure [3](#S5.F3)-(left), we provide the scaling laws for en-de translation task where the pretraining datasets are $100\%$ en-MC4 (same as Figure [2](#S5.F2)-(left)), $50\%$ en-MC4 and $50\%$ de-MC4 (same as Figure [1](#S4.F1)-(left)), $100\%$ de-MC4, $100\%$ fr-MC4 (less aligned), and $100\%$ ro-MC4 (less aligned). Notice that the last two pretraining datasets are expected to be the least aligned with the translation task since the translation pair does not include these languages. We see that, despite this, the scaling laws consistently fit well for both the BLEU score and the cross-entropy loss. However, this is not always the case for the en-fr translation task. In Figure [3](#S5.F3)-(right), we provide the scaling laws for the en-fr translation task where the pretraining datasets are different mixtures of en-MC4 and fr-MC4 datasets. We also include the “less aligned” pretraining datasets such as $100\%$ de-MC4 and $100\%$ ro-MC4. Surprisingly, we see that the scaling law for the BLEU score breaks after some point for the only-English ($100\%$ en-MC4), only-German ($100\%$ de-MC4), and only-Romanian ($100\%$ ro-MC4) pretraining datasets while the cross-entropy loss always follows the scaling law in ([2](#S3.E2)). Interestingly, we do not observe such a break in the BLEU score scaling for the only-French ($100\%$ fr-MC4) pretraining dataset – hinting that not including French data in pretraining leads to poor scaling in the en-fr translation task but not including English does not have such an effect. We also notice that the BLEU score is the lowest for these three pretraining datasets where scaling breaks. This suggests that the scaling law in ([1](#S3.E1)) works well for the BLEU score as long as the pretraining dataset has the promise to give rise to a good performance. However, when the scaling law does not fit well, we may suspect the BLEU score to be low overall. Therefore, whether we can fit the scaling law for the BLEU score seems to give a good indication about the degree of alignment between the pretraining data and the particular translation task.

###### Remark 5.1 .

We observe another interesting phenomenon in Figure [3](#S5.F3). For both en-de and en-fr tasks, 100% en-MC4 leads to significantly worse BLEU score and *downstream* cross-entropy than the more aligned 50% en-MC4 + 50% de/fr-MC4 balanced datasets, respectively. However, de-MC4 and fr-MC4 perform almost as well as the balanced datasets in en-de and en-fr tasks. We leave the investigation of why pretraining on only German/French helps more than pretraining on only English for the given en-de and en-fr tasks to future work.

We also highlight that we cannot make any strong conclusion about the degree of alignment of the pretraining dataset with the task by only looking at the *downstream* cross-entropy loss because of the inconsistency with the BLEU score, a task-related metric, observed in the en-fr plots in Figures [2](#S5.F2) and [3](#S5.F3). This is a counter-example for the claim by Gordon et al., () that the two metrics have an exponential relation. To better demonstrate this, in Figure [4](#S5.F4), we provide a BLEU score vs. *downstream* cross-entropy $\log$-$\log$ plot for en-de and en-fr translation tasks, respectively. While the two metrics indeed seem correlated in Figure [4](#S5.F4)-(left) on the en-de task, we observe a somewhat arbitrary relation for the en-fr task in Figure [4](#S5.F4)-(right) in some cases – which clearly cannot be explained with an exponential relation. This suggest that *downstream* cross-entropy is not always a good indicator for BLEU score. This raises the question whether the scaling laws that have been developed for the *upstream* cross-entropy loss are actually useful predictors for models’ downstream behavior.

Figure: Figure 4: BLEU score vs. *downstream* cross-entropy loss. (left) For en-de translation task, we see a consistent correlation between the two metrics for all the pretraining datasets. This supports the findings of Gordon et al., (). (right) For en-fr translation task, the two metrics usually show an arbitrary relation. Sometimes, the BLEU score increases while the cross-entropy also increases. Unlike the en-de results in (left), the exponential relation in (Gordon et al.,, ) is not observed here.
Refer to caption: /html/2402.04177/assets/figures/wmt17_en_de_ce_loss_vs_bleu.png

###### Remark 5.2 .

We also revisit the definition of the BLEU score to better understand the root cause of the non-smooth behavior and check if we could see a smooth monotonic scale in at least some elements of the BLEU score calculation. Recall that the common form of BLEU score is defined as

$$ $\displaystyle\text{BLEU}=\text{brevity-penalty}\cdot\left(\prod_{i=1}^{4}\text{precision}_{i}\right)^{1/4},$ (3) $$

where $\text{precision}_{n}$ refers to the precision of n-grams, and the second term is the geometric mean of the precision when n is varied from 1 to 4. In all the experiments, we observe $\text{brevity-penalty}=1$, i.e., the non-smooth behavior can be attributed to the precision terms. Hence, our findings, including the scaling law in ([1](#S3.E1)), would also apply for precision–another *downstream* task metric.

###### Remark 5.1 .

###### Remark 5.2 .

## 6 Discussion and Conclusion

We study the scaling behavior of the downstream performance of LLMs as the pretraining data grows and propose scaling laws for both *downstream* cross-entropy and the BLEU score. We demonstrate through extensive experiments that the scaling behavior is significantly influenced by (1) the degree of alignment between the pretraining and the downstream data and (2) the finetuning dataset size. In favorable cases where the distributions are sufficiently aligned, we show that BLEU score can be accurately predicted using a log scaling law. However, with less alignment, there are cases where BLEU score fluctuates unpredictably whereas *downstream* cross-entropy improves monotonically. We also observe that when the finetuning dataset size is sufficiently large, pretraining has little to no value.

Our findings highlight the importance of studying downstream performance metrics and not making decisions solely based on cross-entropy (whether upstream or downstream). This echoes the findings of Schaeffer et al., () about the discrepancy in behavior between smooth and non-smooth metrics when models are scaled.

## Appendix A Additional Experimental Details

### A.1 Model Architectures

We provide the architecture details of the T5-3B and T5-770M models in Tables [1](#A1.T1) and [2](#A1.T2). These models were initially introduced by Raffel et al., ().

**Table 1: T5-3B Raffel et al., () architecture details.**
| Embedding Dimension | $1024$ |
| --- | --- |
| Number of Heads | $32$ |
| Number of Encoder Layers | $24$ |
| Number of Decoder Layers | $24$ |
| Head Dimension | $128$ |
| MLP Dimension | $16384$ |

**Table 2: T5-770M Raffel et al., () architecture details.**
| Embedding Dimension | $1024$ |
| --- | --- |
| Number of Heads | $16$ |
| Number of Encoder Layers | $24$ |
| Number of Decoder Layers | $24$ |
| Head Dimension | $64$ |
| MLP Dimension | $2816$ |

### A.2 Optimizing the Scaling Law Coefficients

In this section, we provide more details on how we optimize the coefficients of the scaling laws. Following Hoffmann et al., (), we use the Huber loss (Huber,, ) to minimize overfitting to the outliers. Huber loss is particularly useful to suppress the effect of the outlier data points in the optimization problem. More specifically, if the data point with value $r$ is predicted by the law as $\hat{r}$, the loss for that data point would be

$$ $\displaystyle\ell_{\delta}(r,\hat{r})=\begin{cases}\frac{1}{2}(r-\hat{r})^{2}&\text{for }|r-\hat{r}|\leq\delta,\\ \delta\cdot(|r-\hat{r}|-\frac{1}{2}\delta)&\text{otherwise.}\end{cases}$ (4) $$

Due to the numerical range difference between the BLEU score (between 0 0 and $100$) and the *downstream* cross-entropy typically taking much smaller values, we use $\delta=0.1$ for the BLEU score law in ([1](#S3.E1)) and $\delta=1e-3$ for the *downstream* cross-entropy law in ([2](#S3.E2)).

For optimization, we use the L-BFGS algorithm (Nocedal,, ). Specifically, for the BLEU score law in ([1](#S3.E1)), we solve

$$ $\displaystyle\min_{E,A,\alpha,\beta}\sum_{\text{Data point }i}\ell_{\delta}(\log f_{i},\log\hat{f}(D_{p_{i}})),$ (5) $$

where $D_{p_{i}}$ is the pretraining dataset size and $f_{i}$ is the BLEU score for the data point $i$, and $\hat{f}(\cdot)$ is the approximation for the optimal law $f(\cdot)$. Similarly, for the *downstream* cross-entropy loss law in ([2](#S3.E2)), we solve

$$ $\displaystyle\min_{E,A,\alpha}\sum_{\text{Data point }i}\ell_{\delta}(\log L_{i},\log\hat{L}(D_{p_{i}})),$ (6) $$

where $D_{p_{i}}$ is the pretraining dataset size and $L_{i}$ is the *downstream* cross-entropy loss for the data point $i$, and $\hat{L}(\cdot)$ is the approximation for the optimal law $L(\cdot)$.

## Appendix B Additional Experimental Results

In this section, we provide additional experimental results that we had to skip in the main body due to the page limit.

### B.1 Results on T5-770M

In Figures [5](#A2.F5) and [6](#A2.F6), we present results similar to Figures [1](#S4.F1) and [2](#S5.F2) in Section [5](#S5), but for T5-770M instead of T5-3B. In general, we observe a similar trend. The proposed scaling laws describe the downstream behavior well when the pretraining and downstream data are aligned. Similar to the results in T5-3B in the main body of the paper, in Figure [6](#A2.F6)-(top, right), we observe a break in the scaling law when the pretraining dataset is 100% en-MC4 and the task is en-fr translation – suggesting the same misalignment for this pretraining data and task that was also observed in Section [5](#S5) on the larger T5-3B model.

Figure: Figure 5: (top) BLEU score vs pretraining dataset size: $\mathbf{f(D_{p})=(\log(A\cdot D_{p}^{\alpha}))^{\beta}.}$ (left) WMT-17 en-to-de translation task. Pretraining dataset has $50\%$ en-MC4 + $50\%$ de-MC4. Dotted and dashed blue curves correspond to the fitted scaling laws for different finetuning dataset sizes, $D_{f}=6M$ and $D_{f}=31M$ tokens, respectively. (right) WMT-15 en-to-fr translation task. Pretraining dataset has $50\%$ en-MC4 and $50\%$ fr-MC4. Dotted and dashed orange curves correspond to the fitted scaling laws for different finetuning dataset sizes, $D_{f}=42M$ and $D_{f}=210M$ tokens, respectively. (bottom) Cross-entropy (CE) validation loss vs pretraining dataset size: $\mathbf{L(D_{p})=E+\frac{A}{D_{p}^{\alpha}}}.$ Same models as the top row. For all the plots, the markers are the actual experimental results and the black horizontal curves correspond to the non-pretrained model directly trained on the task dataset. The finetuning dataset size increases in the order of dotted-dashed for all the curves including the black horizontal lines.
Refer to caption: /html/2402.04177/assets/figures/t5_large_wmt17_en_de_50_50_bleu.png

Figure: Figure 6: (top) BLEU score vs pretraining dataset size: $\mathbf{f(D_{p})=(\log(A\cdot D_{p}^{\alpha}))^{\beta}.}$ (left) WMT-17 en-to-de translation task. Dotted and dashed red curves correspond to the fitted scaling laws for different finetuning dataset sizes, $D_{f}=6M$ and $D_{f}=31M$ tokens, respectively. (right) WMT-15 en-to-fr translation task. Dotted and dashed red curves correspond to the fitted scaling laws for different finetuning dataset sizes, $D_{f}=42M$ and $D_{f}=210M$ tokens, respectively. (bottom) Cross-entropy (CE) validation loss vs pretraining dataset size: $\mathbf{L(D_{p})=E+\frac{A}{D_{p}^{\alpha}}}.$ Same models as the top row. For all the plots, the markers are the actual experimental results and the black horizontal curves correspond to the non-pretrained model directly trained on the task dataset. The finetuning dataset size increases in the order of dotted-dashed for all the curves including the black horizontal lines.
Refer to caption: /html/2402.04177/assets/figures/t5_large_wmt17_en_de_100_0_bleu.png

### B.2 Optimized Coefficients and Prediction Errors of the Scaling Laws

In Tables [3](#A2.T3), [4](#A2.T4), [5](#A2.T5), and [6](#A2.T6), we provide the optimized coefficients for the scaling laws plotted in Figures [1](#S4.F1) and [2](#S5.F2) together with the prediction error.

**Table 3: The coefficients for the BLEU score law $f(D_{p})=(\log(A\cdot D_{p}^{\alpha}))^{\beta}$ for the results in Figure [1](#S4.F1)-(top). For the BLEU score laws, we use $\delta=0.1$ for the Huber Loss. We report $\log A$ instead of $A$ since $A$ typically takes very small and very large values.**
| Pretraining Dataset | Finetuning Dataset | Finetuning Dataset Size | $\log A$ | $\alpha$ | $\beta$ | Prediction Error |
| --- | --- | --- | --- | --- | --- | --- |
| $50\%$ en + $50\%$ de-MC4 | WMT-17 en-de | 6M | $-180.75$ | $9.00$ | $0.75$ | $0.034$ |
| $50\%$ en + $50\%$ de-MC4 | WMT-17 en-de | 31M | $-1.68\times 10^{3}$ | $84.04$ | $0.49$ | $0.050$ |
| $50\%$ en + $50\%$ de-MC4 | WMT-17 en-de | 3B | $-1.64\times 10^{8}$ | $9.91\times 10^{6}$ | $0.19$ | $0.048$ |
| $50\%$ en + $50\%$ fr-MC4 | WMT-15 en-fr | 42M | $-1.82\times 10^{4}$ | $8.98\times 10^{2}$ | $0.42$ | $0.061$ |
| $50\%$ en + $50\%$ fr-MC4 | WMT-15 en-fr | 210M | $-2.33\times 10^{4}$ | $1.21\times 10^{3}$ | $0.40$ | $0.013$ |
| $50\%$ en + $50\%$ fr-MC4 | WMT-15 en-fr | 21B | $5.08\times 10^{3}$ | $4.61\times 10^{8}$ | $0.16$ | $0.005$ |
| $50\%$ en + $50\%$ ro-MC4 | WMT-16 en-ro | 625K | $-36.02$ | $1.77$ | $1.28$ | $0.042$ |
| $50\%$ en + $50\%$ ro-MC4 | WMT-16 en-ro | 3M | $-0.115.03$ | $5.69$ | $0.89$ | $0.015$ |
| $50\%$ en + $50\%$ ro-MC4 | WMT-16 en-ro | 312M | $-1.82\times 10^{4}$ | $9.04\times 10^{2}$ | $0.40$ | $0.015$ |

**Table 4: The coefficients for the *downstream* cross-entropy law $L(D_{p})=E+\frac{A}{D_{p}^{\alpha}}$ for the results in Figure [1](#S4.F1)-(bottom). For the *downstream* cross-entropy laws, we use $\delta=10^{-5}$ for the Huber Loss.**
| Pretraining Dataset | Finetuning Dataset | Finetuning Dataset Size | $E$ | $A$ | $\alpha$ | Prediction Error |
| --- | --- | --- | --- | --- | --- | --- |
| $50\%$ en + $50\%$ de-MC4 | WMT-17 en-de | 6M | $3.21\times 10^{-5}$ | $35.45$ | $0.64$ | $1.36\times 10^{-12}$ |
| $50\%$ en + $50\%$ de-MC4 | WMT-17 en-de | 31M | $3.28\times 10^{-5}$ | $4.70\times 10^{2}$ | $0.78$ | $3.17\times 10^{-12}$ |
| $50\%$ en + $50\%$ de-MC4 | WMT-17 en-de | 3B | $2.24\times 10^{-5}$ | $2.56\times 10^{-2}$ | $0.36$ | $5.76\times 10^{-14}$ |
| $50\%$ en + $50\%$ fr-MC4 | WMT-15 en-fr | 42M | $2.72\times 10^{-5}$ | $2.01\times 10^{6}$ | $1.18$ | $7.52\times 10^{-13}$ |
| $50\%$ en + $50\%$ fr-MC4 | WMT-15 en-fr | 210M | $2.57\times 10^{-5}$ | $1.75\times 10^{7}$ | $1.30$ | $2.24\times 10^{-13}$ |
| $50\%$ en + $50\%$ fr-MC4 | WMT-15 en-fr | 21B | $1.11\times 10^{-7}$ | $3.41\times 10^{-5}$ | $1.82\times 10^{-2}$ | $5.20\times 10^{-14}$ |
| $50\%$ en + $50\%$ ro-MC4 | WMT-16 en-ro | 625K | $2.45\times 10^{-5}$ | $0.49$ | $0.41$ | $3.61\times 10^{-12}$ |
| $50\%$ en + $50\%$ ro-MC4 | WMT-16 en-ro | 3M | $2.62\times 10^{-5}$ | $2.40$ | $0.49$ | $2.19\times 10^{-12}$ |
| $50\%$ en + $50\%$ ro-MC4 | WMT-16 en-ro | 312M | $2.08\times 10^{-5}$ | $3.94$ | $0.53$ | $5.95\times 10^{-12}$ |

**Table 5: The coefficients for the BLEU score law $f(D_{p})=(\log(A\cdot D_{p}^{\alpha}))^{\beta}$ for the results in Figure [2](#S5.F2)-(top). For the BLEU score laws, we use $\delta=0.1$ for the Huber Loss. We report $\log A$ instead of $A$ since $A$ typically takes very small and very large values.**
| Pretraining Dataset | Finetuning Dataset | Finetuning Dataset Size | $\log A$ | $\alpha$ | $\beta$ | Prediction Error |
| --- | --- | --- | --- | --- | --- | --- |
| $100\%$ en-MC4 | WMT-17 en-de | 6M | $-1.88$ | $0.15$ | $3.30$ | $0.014$ |
| $100\%$ en-MC4 | WMT-17 en-de | 31M | $-1.81\times 10^{4}$ | $896.12$ | $0.28$ | $0.006$ |
| $100\%$ en-MC4 | WMT-17 en-de | 3B | $1.02\times 10^{-7}$ | $104.92$ | $0.42$ | $0.015$ |
| $100\%$ en-MC4 | WMT-15 en-fr | 42M | $1.00$ | $2.57\times 10^{-5}$ | $1.11\times 10^{4}$ | $0.042$ |
| $100\%$ en-MC4 | WMT-15 en-fr | 210M | $-6.38\times 10^{7}$ | $3.43\times 10^{6}$ | $0.20$ | $0.034$ |
| $100\%$ en-MC4 | WMT-15 en-fr | 21B | $204.81$ | $3.80\times 10^{14}$ | $9.97\times 10^{-3}$ | $0.004$ |
| $100\%$ en-MC4 | WMT-16 en-ro | 625K | $-10.54$ | $0.55$ | $1.12$ | $0.008$ |
| $100\%$ en-MC4 | WMT-16 en-ro | 3M | $-40.41$ | $2.11$ | $0.79$ | $0.025$ |
| $100\%$ en-MC4 | WMT-16 en-ro | 312M | $3.61$ | $8.17\times 10^{5}$ | $0.19$ | $0.018$ |

**Table 6: The coefficients for the *downstream* cross-entropy law $L(D_{p})=E+\frac{A}{D_{p}^{\alpha}}$ for the results in Figure [2](#S5.F2)-(bottom). For the *downstream* cross-entropy laws, we use $\delta=10^{-5}$ for the Huber Loss.**
| Pretraining Dataset | Finetuning Dataset | Finetuning Dataset Size | $E$ | $A$ | $\alpha$ | Prediction Error |
| --- | --- | --- | --- | --- | --- | --- |
| $100\%$ en-MC4 | WMT-17 en-de | 6M | $3.22\times 10^{-13}$ | $3.18\times 10^{-3}$ | $0.15$ | $5.79\times 10^{-12}$ |
| $100\%$ en-MC4 | WMT-17 en-de | 31M | $3.24\times 10^{-5}$ | $5.20\times 10^{-3}$ | $0.20$ | $9.25\times 10^{-13}$ |
| $100\%$ en-MC4 | WMT-17 en-de | 3B | $2.24\times 10^{-5}$ | $2.56\times 10^{-2}$ | $0.36$ | $5.76\times 10^{-14}$ |
| $100\%$ en-MC4 | WMT-15 en-fr | 42M | $3.49\times 10^{-5}$ | $1.05\times 10^{-2}$ | $0.25$ | $3.63\times 10^{-13}$ |
| $100\%$ en-MC4 | WMT-15 en-fr | 210M | $4.24\times 10^{-5}$ | $19.39$ | $0.66$ | $5.40\times 10^{-13}$ |
| $100\%$ en-MC4 | WMT-15 en-fr | 21B | $1.26\times 10^{-7}$ | $2.59\times 10^{-5}$ | $4.81\times 10^{-3}$ | $3.63\times 10^{-14}$ |
| $100\%$ en-MC4 | WMT-16 en-ro | 625K | $5.79\times 10^{-12}$ | $1.03\times 10^{-3}$ | $7.76\times 10^{-2}$ | $5.56\times 10^{-12}$ |
| $100\%$ en-MC4 | WMT-16 en-ro | 3M | $1.78\times 10^{-12}$ | $9.98\times 10^{-4}$ | $8.33\times 10^{-2}$ | $8.23\times 10^{-12}$ |
| $100\%$ en-MC4 | WMT-16 en-ro | 312M | $5.85\times 10^{-5}$ | $1.37\times 10^{3}$ | $0.88$ | $3.05\times 10^{-13}$ |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: N/A
soundness: 2
strengths: The paper conducts extensive experiments and present some interesting findings.
summary: The paper proposes scaling laws for pre-training data and downstream data in the context of machine translation. The authors demonstrate that when the pre-training data is sufficiently aligned with the downstream task, both the downstream cross-entropy (CE) loss and translation quality adhere to the proposed scaling laws. However, if the pre-training data is not well-aligned, the downstream CE may still align with the scaling laws, but other metrics of translation quality may exhibit misalignment.
weaknesses: 1. "Inaccurate" statement: The field of machine translation is increasingly adopting decoder-only architectures, whereas this paper focuses exclusively on encoder-decoder models. This focus leads to findings that conflict with those from SoTA decoder-only translation models. For example, in lines 355-359, the authors claim that "there is no need to pretrain the models when the fine-tuning data is large enough." This assertion totally contradicts the findings in [1], which states that "you only need a large amount of data for pre-training and a small number of data for continued fine-tuning." The key difference is that [1] employs a decoder-only model but it achieves top performance among other SoTA translation models. I am inclined to agree with the latter explanation because your training and test data are from the same domain (WMT). Training on WMT data usually yields better results on the WMT test set (especially BLEU), which might give the illusion that using only fine-tuning data is sufficient. However, this approach may lead to poor generalization abilities. A large training dataset may have less impact on in-domain test data, but it's essential to ensure that the model generalizes well to out-of-domain data. Therefore, the authors should also investigate the model's performance on other out-of-domain test datasets to provide a more comprehensive evaluation before make this statement.

2. Out-of-date metric: Most of the findings in this paper rely on BLEU scores. However, BLEU is increasingly regarded as an inaccurate metric that does not align well with human judgments, especially when compared to other metrics like COMET. The inclusion of ROUGE also seems weird, as ROUGE is primarily used for summarization evaluation rather than translation. While it might be applicable in certain cases, it is not a mainstream choice for assessing translation quality. The authors should consider using more suitable metrics, such as BLEURT, to provide a more accurate evaluation of their models.

3. Unclear writing: While this is a minor issue, the paper's writing feels disjointed and contains redundant information. There are several instances where the same points are reiterated, such as "when the distribution is well-aligned, both metrics and CE align with the scaling laws." Moreover, the frequent references to figures that are located several pages away disrupt the reading flow and make it difficult to follow the arguments being presented.

4. Lack of definition: I struggled to understand what the authors mean by the "distribution of pre-training and downstream task." It wasn't until midway through the paper that I realized this likely refers to the amount of monolingual target language data included in the pre-training dataset—where more target language data equates to a more "aligned distribution." One question here, if the authors were to conduct experiments on another low-resource language like Icelandic, where the fine-tuning data consists of a small amount of news content but the available pre-training monolingual data is primarily from games, the "distribution/alignment" becomes ambiguous. Alternatively, one might employ cross-lingual learning using a high-resource language like German, for which abundant news data is available. In this scenario, which dataset constitutes a "more aligned distribution"—the Icelandic data from a different domain or the German data from the same domain? I believe that the term "distribution" is too abstract and makes the paper challenging to understand. Providing a more precise definition or elaborating on this concept would greatly enhance the clarity of the paper.

5. A narrow scope in high-resource languages: The focus on high-resource languages is not particularly compelling, as SoTA models already perform exceptionally well in these languages, regardless of scaling laws. People can simply amass large amounts of data for training to achieve excellent results. The paper's emphasis on utilizing billions of tokens for pre-training and millions of parallel sentences for fine-tuning is not practical for many mid- or low-resource languages, where those languages do not have the resource.  A more interesting and valuable direction would be to explore scaling laws that examine how high-resource languages can be used to support low-resource languages. Investigating how data from other languages can enhance translation quality in languages with limited resources would address a critical need in the field of MT and contribute to more inclusive and effective translation models.

Update: I just found that I forgot adding the references:

References:

[1] A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: see above
soundness: 3
strengths: The paper seeks empirical answers to important questions -- scaling laws (even rough ones) for downstream transfer can provide useful guidance: When should one seek out different pretraining data vs more pretraining data? How much finetuning data is sufficient for good downstream performance?

The paper focuses specifically on machine translation -- an important NLP task that has not been the main focus of prior transfer learning scaling law work. This is a strength. The application focus lets the paper go into more depth in empirical analysis and leads to clearer takeaways. 

The paper concretely shows that downstream cross-entropy can be misleading. The paper finds that downstream cross-entropy monotonically improves with pretrain size, even when other machine translation metrics like BLEU do not. This is a useful result for future researchers.
summary: Past work on scaling laws has mostly focused on pretraining loss rather than downstream transfer learning task performance. This paper attempts to establish scaling laws for transfer learning for machine translation. The paper specifically investigates how pretraining data size affects downstream translation performance, measured in terms of downstream cross-entropy loss as well as in terms of downstream machine translation metrics like BLEU. Results show that that size of finetuning data, as well as the similarity between the pretraining and finetuning datasets, have a large impact on downstream results and scaling behavior. Most interestingly, the results indicate that when pretrain and finetune are well enough aligned, there is a clear log scaling law for pretraining size on downstream machine translation metric performance. However, when pretrain and finetune are dissimilar, more pretraining data can sometimes even hurt performance on downstream machine translation metrics. This stands in contrast with downstream cross-entropy, which improves monotonically with pretrain size regardless of pretrain/finetune mismatch.
weaknesses: Aspects of the presentation could be improved. Specifically, as a reader, I found it confusing that the paper essentially starts with a results / takeaways section before discussing experimental details. Then an experimental details section comes next, followed by a results / takeaways section. I think a more traditional ordering could be more effective here -- the intro could be used to discuss high-level takeaways up front, leaving the rest of the detailed results to be described after experimental setup is clear. Without having a clear understanding of experimental setup, I was left wondering in many cases how to interpret the early results discussion. 

The one experimental weakness that stood out to me: It's not entirely clear how "alignment" between pretrain and finetune is being defined. For the most part, it seems to mean in the context of this paper whether and to what extend that datasets share languages. This could be further clarified and formalized -- but, further, more nuanced measures of alignment could easily be reported. These might help clarify experimental takeaways having to do with alignment, which represent some of the more interesting results in this paper. 

Minor:

-Discussion of results on other tasks is odd given that paper is focused on MT.

-I couldn't follow remark 2 -- what was the main takeaway / hypothesis? Consider rephrasing this section for improved clarity. 

-"3.4 A Guide for ..." section is kind of confusing -- especially part 2.

---

confidence: 3
contribution: 2
initial_rating: 8
presentation: 3
questions: To what extent do you expect your findings to carry over to decoder-only models more commonly found in current LLMs?
soundness: 3
strengths: - well written paper
- addresses a problem that is relevant for a large community
- thourough empirical analysis
summary: The paper analyses to what extent scaling laws apply if the pre-training and downstream data are well or less well aligned. They find that iff the data is less aligned, task-specific metrics can fluctuate when increasing the pre-training data, while increasing more monotonically if the data is well aligned. Their findings are mostly based on machine translation experiments but also generalize to more general tasks such as the tasks in the SuperGlue benchmark set.
weaknesses: Although the concept of alignement is central to this work and many times the authors refer to "degree of" or "not sufficient" alignment, no definition of alignment is given. I understand that this is not trivial but I would appreciate at least an attempt to provide a---ideally formal---definition that could also be used to quantify alignment. Currently, it seems the only way is to determine the degree in hindsight (see lines 227--228) by failing to fit the scaling law.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: The alignment between the pre-training and downstream task is crucial to understanding the scaling law. I am not entirely clear on what “align” means in this context. Could you please explain in detail the meaning of “align” in this work?
soundness: 3
strengths: The paper's main contribution is its in-depth investigation of the impact of pre-training data on machine translation tasks and the proposal of a new logarithmic law to describe the change in downstream performance as the size of the pre-training data increases. This research is helpful to understand the performance of large-scale language models in specific downstream tasks and provides a new method for evaluating the value of pre-training data. From the perspectives of innovation and importance, the paper is highly valuable. Additionally, the quality and clarity of the paper are good, with detailed descriptions of experimental design and result analysis, and abundant charts and empirical evidence to support conclusions.
summary: The paper investigates the scaling behavior of downstream performance metrics, including translation scores and cross-entropy loss, as a function of the size of the pretraining dataset in machine translation tasks. The authors conduct experiments using T5 models trained on different subsets of the Multilingual C4 dataset and fine-tuned on various translation tasks. They find that when the pretraining and downstream tasks have well-aligned distributions, both translation scores and cross-entropy loss improve monotonically with more pretraining data. However, when the distributions are not well-aligned, translation scores can exhibit non-monotonic behavior, while cross-entropy loss still improves monotonically. The authors propose a practical guide for evaluating the value of pretraining data for translation tasks based on downstream translation scores and demonstrate the effectiveness of their approach on multiple natural language processing tasks.
weaknesses: One potential weakness of the paper is it is focusing solely on downstream performance changes in machine translation tasks. This limits the comprehensive evaluation of the applicability of the logarithmic law to a wider range of natural language processing tasks. 
For NLP tasks that use precision and recall as metrics, can these be effectively aligned with the pre-training procedure? Additionally, can cross-entropy be generalized to observe the scaling law?

It is recommended to include a reference to the inference scaling law from OpenAI’s o1 model and to discuss the scaling law within a broader framework.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: - This is in not a critique but I'm curious about your thoughts. You study pre-training with up to 2 languages. How do you think your findings translate to the more realistic setting of pre-training on a large number of languages, with severe misalignments in data size between the languages?

- Similarly, do you have any findings related to language similarity?
soundness: 4
strengths: - Exceptionally well-grounded in literature, with relevant prior work effectively woven throughout the paper's narrative.
- Strong experimental methodology with fully controlled experiments that isolate different variables' effects. In contrast to a lot of recent LLM works, the experiments are fully controlled which I appreciate a lot. The paper demonstrates comprehensive validation across multiple metrics and scenarios, with clear empirical support for the proposed log-law.
- The writing is clear and the structure effective. Clear and systematic presentation of results with thorough analysis.
- The proposed log-law for translation quality metrics is well-motivated and comprehensively validated.
- Makes important observations challenging common assumptions about cross-entropy and translation quality metrics.
summary: This paper investigates scaling laws for downstream task performance in machine translation, specifically examining how pretraining dataset size affects translation quality after finetuning. The authors propose a novel log-law to describe the scaling behavior of translation quality metrics (BLEU, ROUGE, COMET) and demonstrate that the relationship between pretraining data size and downstream performance heavily depends on distribution alignment and finetuning dataset size. A key finding is that while cross-entropy consistently follows power-law scaling, translation quality metrics may deviate from expected scaling patterns when pretraining and downstream distributions are not well-aligned.
weaknesses: - Limited to pairwise language combinations in pretraining - would benefit from discussion about generalization to realistic multilingual pretraining scenarios.

- It would be a great addition to have a short discussion that connects these findings to recent successes in LLM-based MT. For instance, some works like https://aclanthology.org/2024.tacl-1.32/ and https://aclanthology.org/2024.acl-long.336/ fine-tuned directly on parallel data, but these LLMs are heavily English-centric and therefore there is a low distribution alignment. More recent work like ALMA https://openreview.net/forum?id=farT6XXntP and TowerLLM https://openreview.net/forum?id=EHPns3hVkj#discussion circumvent this issue by continuing pre-training on multilingual data before fine-tuning on parallel data.

- Minor  issues
    - Lines 142-143: "Tran et al., 2019" is repeated 3 times
    - Lines 264-265: it'd be nice to include fine-tuning dataset sizes here

Meta-review: This paper investigates the scaling laws of the MT downstream performance. More importantly, when pretraining and downstream data are well-aligned, both downstream CE loss and translation quality metrics follow predictable scaling laws; when the distributions are misaligned, the CE loss continues to scale monotonically, but translation quality metrics deviate, sometimes worsening with more pretraining data. A novel logarithmic law describes the scaling behavior of translation metrics, which complements prior scaling laws based on CE loss

**Strengths** (1) Comprehensive empirical analysis, (2) reasonable but solid insights on the MT scaling laws based on the data alignment; (3) a new log-law for downstream metric scaling

**Weaknesses** (1) Some reviewers mentioned that "alignment" between pretraining and downstream data is insufficiently defined and unclear how to quantify (addressed during the discussion phase); (2) The study lacks experiments on low-resource, multilingual and cross-domain scenarios; (3) Reviewer argue the conclusion of the paper might be inaccurate as the paper focuses on traditional encoder-decoder architectures, while the mainstream SoTA MT research is shifting to decoder-only models; (4) Additional neural-based metrics can be added as supplement.

**Decision** 
This paper makes valuable contributions by exploring scaling laws for MT and proposing a new log law for downstream performance. With the additional improvements, including the definition of alignment, I am leaning toward acceptance.

It is also noted that part of the review (including the discussion) from 2Keq is a bit irrelevant from the focus of the paper. Therefore, I down-weigh the score from 2Keq.

Decision: Accept (Poster)