When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method
Authors: Biao Zhang
Zhongtao Liu
Colin Cherry
Orhan Firat
Venue: ICLR 2024 poster (2024)
Keywords: LLM finetuning
Scaling Laws
Full-model finetuning
Parameter efficient tuning
Machine Translation
Multilingual Summarization
Primary area: general machine learning (i.e., none of the above)
TL;DR: We explore the scaling for LLM finetuning and consider multiple factors, including LLM model size, pretraining data size, finetuning data size, and new finetuning parameter size.

Abstract: While large language models (LLMs) often adopt finetuning to unlock their capabilities for downstream applications, our understanding on the inductive biases (especially the scaling properties) of different finetuning methods is still limited. To fill this gap, we conduct systematic experiments studying whether and how different scaling factors, including LLM model size, pretraining data size, new finetuning parameter size and finetuning data size, affect the finetuning performance. We consider two types of finetuning – full-model tuning (FMT) and parameter efficient tuning (PET, including prompt tuning and LoRA), and explore their scaling behaviors in the data-limited regime where the LLM model size substantially outweighs the finetuning data size. Based on two sets of pretrained bilingual LLMs from 1B to 16B and experiments on bilingual machine translation and multilingual summarization benchmarks, we find that 1) LLM finetuning follows a powerbased multiplicative joint scaling law between finetuning data size and each other scaling factor; 2) LLM finetuning benefits more from LLM model scaling than pretraining data scaling, and PET parameter scaling is generally ineffective; and 3) the optimal finetuning method is highly task- and finetuning data-dependent. We hope our findings could shed light on understanding, selecting and developing LLM finetuning methods.

Published as a conference paper at ICLR 2024
WHEN SCALING MEETS LLM FINETUNING: THE EFFECT OF DATA, MODEL AND FINETUNING METHOD
Biao Zhang Zhongtao Liu Colin Cherry Orhan Firat
Google DeepMind
"Google Research
(biaojiaxian jixiing, aariggzooonn zhongtao, co www.lenncer,yc.m orhanf @ooogaee.eeooggeecocoommom
ABSTRACT
While large language models (LLMs) often adopt finetuming to unlock their ca- pabilities for downstream applications, our understanding on the inductive biases (especially the scaling properties) of different finetuning methods is still limited. To fill this gap, we conduct systematic experiments studying whether and how dif- ferent scaling factors, including LLM model size, pretraining data size, new fine- tuning parameter size and finetuning data size, affect the finetuning performance. We consider two types of finetuning full-model tuning (FMT) and parameter ef- ficient tuning (PET, including prompt tuning and LoRA), and explore their scaling behaviors in the data-limited regime where the LLM model size substantially out- weighs the finetuning data size. Based on two sets of pretrained bilingual LLMs from IB to 16B and experiments on bilingual machine translation and multilin- gual summarization benchmarks, we find that 1) LLM finetuning follows power- based multiplicative joint scaling law between finetuning data size and each other scaling factor; 2) L.LM finetuning benefits more from [.LM model scaling than pretraining data scaling, and PET parameter scaling is generally ineffective; and 3) the optimal finetuning method is highly task- and finetuning data-dependent. We hope our findings could shed light on understanding, selecting and developing LLM finetuning methods.
INTRODUCTION
Leveraging and transferring the knowledge encoded in large-scale pretrained models for downstream applications has become the standard paradigm underlying the recent success achieved in various
domains (Devlin al.) 2019 Lewis et al. 2020: Raffel et al. 2020 Dosovisskiy et al.) 2021 Baevski
ctt al. 2020), with the remarkable milestone set by large language models (LLMs) that have yielded
ground breukidb performance across language tasks (Brown et al. 2020 Zhang al. 2022b; Scao
cet al., 2022 Touvron et al.] 2023). Advanced LLMs, such as GPT-4 (OpenAll 2023 and PaLM 2 (Anil et al.. 2023), often show emergent capabilities and allow for in-context learning that could use just a few demonstration examples to perform complex reasoning and generation tasks (Wei) al. 2022, Zhang al., 2023] Fu et al.) 2023 Shen et al. 2023). Still, LLM finetuning is required and widely adopted to unlock new and robust capabilities for creative tasks, get the most for focused downstream tasks, and align its value with human preferences (Ouyang al.] 2022 Yang et al.
2023 Gong et al. 2023 Schick et al., (023). This becomes more significant in traditional industrial applications due the existence of large-scale annotated task speciicci data accumulated over years.
There are many potential factors affecting the performance of LLM finetuning, including but not limited 1) pretraining conditions, such as L.M model size and pretraining data size; and 2) fine- tuning conditions, such as downstream task, finetuning data size and finetuning methods. Intuitively, the pretraining controls the quality the learned representation and knowledge in pretrained LLMs, and the finetuning affects the degree of transfer to the donwstream task. While previous studies have well explored the scaling for LLM pretraining or training from scratch (Kaplan et al.) 2020; Hoff- mann et al. 2022) and the development of advanced efficient finetuning methods (Hu et al. 2021
He et al. 2022), the question of whether and how LLM finetuning scales with the above factors unfortunately receives very little attention (Hernandez et al. 2021), which is the focus of our study.

---

Published as a conference paper at ICLR 2024
Note, apart from improving finetuning performance, studying the scaling for LLM finetuning could help us to understand the impact of different pretraining factors from the perspective of finetuning, which may offer insights for LLM pretraining.
In this paper, we address the above question by systematically studying the scaling for two popular ways of LLM finetuning: full-model funing (FMT) that updates all LLM parameters and parameter- efficient tuning (PET) that only optimizes a small amount of (newly added) parameters, such as prompt tuning (Lester et al., 2021 Prompt) and low-rank adaptation (Hu et al. 2021 LoRA). We first examine finetuning data scaling (Hernandez et al. 202T). on top of which we further explore its scaling relationship with other scaling factors, including LLM model size, pretraining data size, and PET parameter size. We focus on the data-limited regime, where the finetuning data is much smaller than the L.MM model, better reflecting the situation in the era of L.M. For experiments, we pretrained two sets of bilingual LLMs (English&German Fnglish&Chines) with model size ranging from 1B 16B, and performed large-scale study on WMT machine translation (English- German, English-Chinese and multilingual summarization (English, German, French and Spanish) tasks with up to 20M finetuning examples. Our main findings are summarized below:
We propose the following multiplicative joint scaling law for LLM finetuning:
((), A
X * +E, D
(1)
where {A, E, 9) are data-specific parameters to be fitted, D, denotes finetuning data size, and X refer to each of the other scaling factors. We show empirical evidence that this joint law generalizes to different settings.
Scaling LLM model benefits LLM finetuming more than scaling pretraining data.
Increasing PET parameters doesn't 't scale well for LoRA and Prompt, although LoRA shows better training stability.
The scaling property for LLM finetuning is highly task- and data-dependent, making the selection of optimal finetuning method for a downstream task non-trivial.
L..M-based finetuning could encourage zero-shot generalization relevant tasks, and PET performs much better than FMT.
2 SETUP
Downstream Tasks We consider machine translation and multilingual summarization the downstream tasks for the finetuning, because 1) these tasks require resolving cross-lingual under- standing and generation, which represent high complexity and challenging; and 2) they are well established in NLP with rich amount of available finetuning corpora. Specially, we adopt WMT14 English-German (En-De) and WMT19 English-Chinese (En-Zh) (Koemi et al. 2022) for transla- tion. We combine the De, Spanish (Es) and French (Fr) portion of the multilingual summarization dataset (Scialom et al. 2020) with CNN/Daily-M and denote it as MLSum. Details about each task are listed in Table la Note for MLSum, we di- rectly concatenate the datasets of different languages for training and evaluation, where each article is prepended prompt indicating its language "Summarize the following document in (lang):".
(Hermann et al. 2015 En) for summarization
LLMs and Preraining We adopt the exact setup as Garcia al. (2023) for LLM pretraining. The model is a decoder-only Transformer with multi-query attention [Chowdhery et al. 2022) and trained with the modified UI.2 objective (Tay af.] 2022). Considering the focused downstream tasks and also to ensure the generalization of our study, we pretrained two sets of bilingual LLMs, ie. En-De LLM and En-Zh LLM. The pretraining data is a mix of monolingual data from two languages: we use En/De (En/Zh) data with about 280B (206B) tokens to pretrain the En-De (En- Zh) LLM. We train LLMs with parameter sizes from IB to 16B by varying model configurations as in Table 33] and keep all other settings intact. All LLMs are optimized using Adafactor (Shazeer & [ttern] 2018) for one training epoch under a cosine learning rate decay schedule (from 0.01 to 0.001). We refer the readers (Garcia et al. 2023) for more details about the pretraining.
2

---

Published as a conference paper at ICLR 2024
Table 1: Setups for finetuning. "K/B/M": thousand//illionmiilloor "#Train": the number of training examples; "Length"; maximum source/taret sequence length cut at training. Note pretraining data size for token count. Bold numbers denote the held-out settings we leave for scaling law verification.
(a) Details for finetuning tasks.
Task
#Train
Length
Dev
Test
Zero-Shot
Base LLM
WMT14 En-De
4.5M
256/256
newstest2013
Nwwstest2020,221111 2021,2022
Flores200
En-De LLM
WMT19 En-Zh
25M
256/256
newsdev2017
newstest2020, 20,0,022,2002,
Flores200
En Zn-Zh LLM
MLSum
1.1M
512/256
official dev sets
official test sets
En- LLM
(b) Scaling settings for different factors.
LLM Model Sizes
IB, 2B, 4B, 8B, 16B
En-De LLM
84B, 126B, 167B, 209B, 283B
Pretraining Data Sizes
En-Zh LLM
84B, 105B, 126B, 147B, 167B, 206B
Prompt Length
50, 100, 150, 200, 300, 400. 600
PET Parameter Sizes
LoRA Rank
4, 8, 16, 32, 48, 64, 128
Prompt LoRA
8K, 10K, 20K, 30K, 40K, 50K, 60K, 70K, 80K, 90K, 100K
FMT- WMT En-De
100K, 500K, 1M, 1.5M, 2M, 2.5M, 3M, 3.5M, 4M, 4.5M
Finetuning Data Sizes
FMT WMT En-Zh
IM, 2M, 3M, 4M, 5M, 10M, 15M, 20M, 25M
FMT- MLSum
100K, 200K, 300K, 400K, 500K, 600K, 700K, SO0K, 900K
Finetuning Settings
We mainly study the scaling for the following three finetuning methods:
Full-Model Tuning (FMT): This is the vanilla way of finetuning which simply optimizes
all LLM parameters;
Prompt Tuning (Prompt): Prompt prepends the input embedding X x with a tun- able "soft- perettppopp" P RlP|xd and feeds their concatenation [P: X] [(P++|x)x to LLM. |-| and d denote sequence length and model dimension, respectively. During finetun- optimized. We initialize P from sampled vocabulary, and set the prompt length Pl to 100 by default (Lester et al. 2021).
ing, only the prompt parameter
Low-Rank Adaptation (LoRA): Rather than modifying LLM inputs, LoRA updates pre- trained model weights W Th x with trainable pairs of rank decomposition matrices E mRx A E Rr and uses W + BA instead during finetuning. m, n are dimensions and is LoRA rank. Only Bs and As are optimized. We apply LoRA to both attention and feed-forward layers in LLMs, and set the rank r to 4 by default (Hu et al. 2222I).
We explore 4 different factors for the scaling, which are summarized in Table lb scaling, all experiments are based on the corresponding 1B LLM. For pretraining data scaling, we adopt intermediate pretrained checkpoints as the proxy due computational budget constraint while acknowledge its sub-optimality. Details for optimization are given in Appendix.
Except LLM model
Evaluation We use the best checkpoint based on token-level perplexity (PPL.) on the dev set for evaluation. For scaling laws, we report PPL on test sets; for general generation, we use greedy decoding, and report BLEURT (Sellam al., 2020) and RougeL (Lin) 2004) for translation and summarization, respectively. For zero-shot evaluation, we adopt Flores200 (NLLB Team 2022) and evaluate on (Fr, De, Hindi (Hi), Turkish (Tr), Polish (Po)~Zh) and (Fr, Zh, Hi, Tr, Po >De for En-Zh and En-De translation respectively. For scaling law evaluation, we split empirical data points into two sets, empirical fitting and held-out set, where the former is used for fitting scaling parameters and the latter is used for evaluation. We report mean absolute deviation. To reduce noise, we perform three runs, each with a different random subset the finetuning data, and report average performance. When sampling for MLSum, keep the mixing ratio over different languages fixed.
3 WHY MULTIPLICATIVE JOINT SCALING LAW?
We consider scaling factors in this study but jointly modeling all of them is time and resource consuming. Instead, we treat finetuning data as the pivoting factor and perform joint scaling analysis
3

---

Published as a conference paper at ICLR 2024
Figure 1: Fitted single-variable scaling laws for finetuning data scaling different LLM model sizes WMT14 En-De. Solid lines denote fitted scaling curves. Filled circles and triangles denote fitting and held-out data points. Au: mean absolute deviation on the held-out data.
FMT, 4. = 0.002
Prompt, 4 = -0.002
LoRA, 4e 0.003
1.3
1b.-0022
168-0.06
1010
p 0.09
p0 022
2b 2--- 22.5-0.3.
9-26
15
ibb p 0.05
0.91
üb 2 0.03
1.0
02
0.8
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
#Sentences tfinetuning 166
#SSentences dinetuning)
145
Senneences tfinetuning)
1e5
Table 2: Held-out fitting errors (4) for the additive and multiplicative scaling formulation over different fine- tuning methods on WMT14 En-De. Multiplicative scaling law generalizes better.
Multiplicative
Additive
Scaling Factor
FMT
Prompt
LoRA
Avg
FMT
Prompt
LoRA
Avg
LLM Model Size
0.0052
0.0043
0.0047
0.0048
0.012
0.0076
0.0045
0.0079
Pretraining Data Size
0.0057
0.0061
0.0084
0.0068
0.0048
0.0075
0.0082
0.0069
PET parameter size
0.005
0.0031
0.004
0.0069
0.0032
0.005
between it and every other factor separately. Below, we start with finetuning experiments for FMT, Prompt and LoRA WMT14 En-De, and then explore the formulation for the joint scaling.
Finetuning data scaling follows a power law. We first examine the scaling over finetuning data size for each LLM model size independently, with single variable formulation: Ł(D,) A/D3 + E. Following Hoffmann et al. 2022). we estimate {A, 3, E} using the Huber loss (6 = 0.001) and the L-BFGS algorithm, and select the best fit from grid of initializations. Figure ssswws that the above formulation well describes LLM finetuning data scaling with small predictive errors across model sizes and methods, echoing with the findings of Hernandez et al. (2021). Such scaling trend also implies that, while finetuning with small amount of examples could achieve decent results (Zhou al. [2023] Gao et al. 2002), larger scale finetuning data still contributes improved downstream performance, especially when the downstream application is well defined.
Additive or multiplicative joint scaling law for LLM finetuning? Figure I also shows some scaling pattern over LLM model sizes, suggesting the existence of joint scaling law. We explore two formulations: multiplicative in Eq. 11 and additive: L(X, DT) 4/x" + B/D o + E Hoff-
mann al. 2022). , and compare them via empirical experiments !
In both formulations, a and |3 reflect the impact of factor X and finetuning data size on the perfor- mance, respectively, which factor-specific. E' model- and task-dependent term, describing irreducible loss (Ghorbani et al.) (022)). We notice that the meaning for 3 and E generalizes over different factors X, and thus propose estimate them first based on results for both LLM model and pretraining data scaling 2 Such joint fitting could also reduce overfitting and improve extrapolation ability. We apply the following joint fitting loss:
min B.g runL factor X
Σ
Hubers (xxx -ci).
(2)
ax ax.bx
For LLM model scaling, we omitted the newly added parameters in PET because 1) the added parameters only take very tiny proportion, and the proportion across LLM model sizes is similar. Take the IB LLM example. | 100 in Prompt adds 0.017% parameters; r in LoRA adds 0.1% parameters. We also explored different formulations for the new parameters PET, which don't make substantial difference. We didn't 't consider PET parameter scaling when estimating and F. because this scaling is pretty weak and ineffective, as shown Section 4
4

---

Published as a conference paper at ICLR 2024
Figure 2: Fitted multiplicative joint scaling laws for LLM model size and finetuning data size WMT14 En-De. WMT19 En-Zh and MLSum. A./An: mean absolute deviation on fitting/held data. Ore, /btta scaling exponent for LLM model size/finetuning data size. We work on IB to 16B LLM.
FMT (WMT En De) Ag 0.004, A. = 0.005
Prompt (WMT En De) de 0.005, An 0.004
LoRA (NMT En De) Ae 0.006, A. 0.005
1-10
Om 0.52, 0.15
On 6- 8.0.51
Om 0.36,
1.2
0.8
0.0
0.4
0.6
0.0
0.6
(finetuning)
1ef
dinetuning)
165
(fineturing)
165
FMT (WMT En Zh)
Prompt (WMT En Zh)
LoRA (WMT En En-+Zh)
A. 0.004, 4 -0 0.007
4 = 0.005, As 0.019
A. 0.004, 4n 0.026
1+10
4m
0.34, 0.14
0.33, 0.015
D.31. 8- 0.025
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
#Sentences (finetuning)
(finttnnina
165
#onteencss (fineturing)
l05
FMT (MLSum) 0,003, A 0.007
Prompt (MLSum)
LoRA (MLSum) 4g = 0,004, 4 = 0.022
4g
Ag = 0.012, An = 0,013
Bm 0.1, - 0.025
dm 0.11, 8= 0.03
1010
0.0087
155
1.0
Test
1..
05
0.0
0.2
0.4
0.6 0.8 1.0
0.0
0.2
0.4
0.8
0.8
1.0
#Sentances (Tinstuning!
165
#Sentieccss (finetuning)
165
#Sentiences (finetnnin!
165
where we set Ax = exx Bx = Fx e., and X refers to LLM model size or pretraining data size. Note bx is only valid in the additive formulation. We then fix 3 and E and refit other parameters for each factor, separately.
Table (and Table 6 in Appendix) shows that both joint laws perform similarly while the multi- plicative one achieves slightly lower extrapolation error on average. Therefore, we adopt Eq. ① for follow-up analysis.
SCALING RESULTS FOR LLM FINETUNING
Here, we show the empirical results for LLM model, pretraining data and PET parameter scaling on WMT14 En-De, WMT19 En-Zh and MLSum Figures
3 and respectively. Results for BLEURT/RougeL are given in Appendix (Figures 7 8 and w which shows high correlation with the PPL scores in general (see Table Fitted scaling parameters summarized in Table
The proposed multiplicative scaling law captures the scaling relation between different fac- tors and finetuning data size. In each group of experiments, we leave several data points along each scaling dimension the held-out set. We report the mean absolute derivation the empir- ical fitting (A.) and held-out (Ai) sets to show the fitting and predictive ability, respectively. In general, observe that Eq. (T) captures the scaling trend of different factors under finetuning data scaling with small fitting and extrapolation errors. Note there are some mismatched cases, where the empirical data points themselves could be noisy mostly caused by unstable optimization and
5

---

Published as a conference paper at ICLR 2024
Figure 3: Fitted multiplicative joint scaling laws for pretraining data size and finetuning data size on WMT14 En-De, WMT19 En-Zh and MLSum(LLM model size: IB). ag. scaling exponent for pretraining data size.
FMT (WMT En De) Ag 0.005, A, 0.006
Prompt WWNT En En- De) de 0.007, An 0.006
LoRA (NMT En De) Ae 0,006, 4. 0.008
1-11
1.4
8,-
-
4 021,8-0.05
dp 0.18,8-0.000
322
Sizel
1.2
15
1.0
0.5
0.0
0.4
0.6
1.0
0.0
0.6
(fineuniig)
1e6
dinetuning)
165
(fineturing)
105
FMT (WMT En Zh) 4. = 0.000,.4, An -0 0.002
Prompt (WMT En Zh) 4 = 0.008, 4s 0.007
LoRA (WMT En Zh)
A 0.008, An = 0.006
1e11
Pretrain Data (4tokess))
0.11.
50
1.7
25
1.6
Teat
15
0.015
4.
B
1.4
0.5
1.3
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
#Sentences (finetuning)
le/
#Senterices (inntuninal
10b
Sontenoes (fineturing)
les
FMT (MLSum)
Prompt (MLSum) Aa = 0.007 An = 0. 0.008
LoRA (MLSum) 4g = 0,003, 4 = 0.004
4.
0,008, A = 0.009
2.3
C₂ 0.069, 8 0.025
1ell
0.073. 8 - 0.03
2.2
0.11, 6..087
3.0
2.1
225
2.0
bake
1.5
1.9
1.9
1.8
1.7
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
#Sentences (Tinatuning]
165
#Sentences (finetuning)
165
#Sentsinces (finstniin!
165
dev-set overfitting, challenging issues when tuning small datasets. We observe high mismatch when extrapolating 16B, particularly for LoRA and Prompt on WMT19 En-Zh in Figure 2 We ascribe this to 1) the insufficiency of empirical data over LLM model sizes (i.e. only 4 points) the prediction by the fitted scaling law makes sense intuitively based on IB-8B results, and 2) the inferior of the 16B En-Zh LLM due to pretraining instability, where its pretraining performance is not well predicted by even single-variable scaling laws as in Figure 10 Appendix.
LLM finetuning benefits more from LLM model scaling than pretraining data scaling across tasks and methods. While LLM model size and pretraining data size show similar impact on the pretraining scaling following the optimal scaling under a computational budget constraint (Hoff- mann al.) [2022] Muennighoff et al. 2023). they show slightly different roles finetuning scaling. Intuitively, finetuning heavily relies on the knowledge encoded in the LLM, where LLM model size and pretraining data size both matter. However, results in Figures 2 and Table 4 show that the scaling exponent for LLM model size am often outnumbers that for pretraining data size Op across finetuning methods and tasks, i.e. am > ap. This suggests that using larger LLM model is pre- ferred over pretraining on a larger dataset, but we also notice that the difference in scaling is highly nask-dependent. Our selection closed generation tasks, i.e. translation and summarization, might deliver biased observations and for more creative generation tasks, larger and diverse pretraining data could be more crucial.
Scaling PET parameters is ineffective, delivering limited gains for both LoRA and Prompt. The amount of newly added trainable parameters often forms bottleneck for the expressivity of

---

Published as a conference paper at ICLR 2024
Figure 4: Fitted multiplicative joint scaling laws for PET parameter size and finetuning data size WMT14 En-De. WMT19 En-Zh and MLSum(LLM model size: IB). ac: scaling exponent for PET parameter size.
Prompt
Prompt (WMMT
Prompt (MLSum)
de = 0. 008,008,00000 A = 0. 0.005
4. = 0.007 A = 0.009
de 0.005, A 0.014
1.30
0.0027,8
1:67
2.10
0.0026.6-00.0
1.26
1.24
2.05
1.65
1.64
2.00
1.63
150
1.18
1.62
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0,50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
# SSrrtencess (finetnning le5
#Sentences (finetuning)
165
Serteencss (finetuning)
le5
LoRA
LoRA (MLSum)
4. 0.004, A 0.003
4.
0.005, 2
A- 0.002, A.
2.10
1.68
0.0017,F
0.081
0.0044,
0.025
-
2.05
Test PPL
100
1.64
2.00
30
1.20
1.95
1.60
1.30
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.30
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Serttences Ifinetuning! le5
#Sentences (rinetuning) 165
@Settences (finetuning) 1e5
PET, controlled by the length [P] and rank r in Prompt and LoRA, respectively. However, Figure and Table 4 show that increasing PET parameter sizes (i.e. enlarging |P] and affects finetuning performance marginally as demonstrated by the small scaling exponents, layl < le 2, and even results in inverse scaling some settings, c.g. LoRA on En-De. Besides, we observe that scaling Prompt length suffers from training instability optimizing larger prompt embedding becomes non-trivial, which has also been seen in previous studies (Lester et al. 202T Hu al. [02T]. We expect that carefully optimizing finetuning hyperparameters and prompt initialization may alleviate it to some extent. In this respect, LoRA is more stable and reliable.
Finetuning data have more pronounced influence on FMT than PET, where LoRA scales bet- ter than Prompt. Different finetuning methods show different degrees of finetuning data scaling. Table 4 shows that the scaling exponent for FMT is often significantly higher than that for PET across settings, indicating that FMT is more data-hungry and also benefits more from increasing fine- tuning data. While the scaling exponents are quite similar across PET, 3 for LoRA often slightly surpasses that for Prompt. As shown in Figures and 4 LoRA often achieves better finetuning performance with more finetuning data than Prompt while Prompt behaves better with only few thousands of finetuning examples.
PET depends more LLM model and pretraining data scaling than finetuning data scaling across settings. Since the majority of LLM parameters is frozen during finetuning. PET relies heavily the encoded knowledge in pretrained LLMs when adapting them to downstream tasks. This is reflected by Table 4 that Omm and Op are clearly larger than / in PET. Figure and further further support the scaling of L.M model, where the performance gap between FMT and PET is substan- tially narrowed with larger LLMs.
5 DISCUSSION
Which finetuning method should apply for given task? Unfortunately, there is universal answer! Intuitively, there exists a critical point for finetuning data size beyond which one finetuning method performs better than another. However, the high non-linearity of the joint scaling law hinders us from identifying such points analytically, although the finetuning data size follows a power law when the performance difference between methods is fixed (see Appendix). We thus resort to

---

Published as a conference paper at ICLR 2024
Figure 5: Critical finetuning data sizes between diffternt finetuning methods estimated by the fitted joint scaling law on WMT14 En-De, WMT19 En-Zh and MLSum. We use scipy.optimize,,son for the estimation. Critical point for "A vs. B": the finetuning data size (y-axis) at which A performs equal to under the base model condition at x-axis. The value varies greatly across tasks.
WMT
WIMT Zh
MLSum
25000
3.25
10
Pronict LONA
Lo
32502
FHT LafA
Pronpe vi. Lokk
1.00
0.75
PMT LoBA
Pronect LORA
36-
04
5032
0.25
2500
0.00
0.
100
0.5
1.0
0.5
1,0
1/5
Modd Suc
2420
Modi S4u
1430
Model DAae
141.
WMI -De
MLSium
70300
60300
400000
va. hampt
PNT Prompt
50300
60300
30000
203000
20000
100000
10000
Figure 6: Zero-shot evaluation for LLM model size and finetuning data size scaling. The score is averaged over [Fr, De, Hi, Po- Po-+Zh] and (Fr. Zh, Hi, Tr, Po-DDe WMT19 En-Zh and WMT14 En-De, respectively.
FMT
Prompt (WIMI De)
LoRA (WMT
1e10
0.55
0.40
0.5
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
#Sechences (finetanin)
1o5
@Sentences (finetunin))
les
@Senhenee (fnetuning)
165
FMT (WNT Zhi
Promppt (WMT En 2h)
LoRA (WMT En Zh)
1w10
Zendhot BLEURE BLEUNT
0.55
0.50
0.40
0.0
3.5
1.0
1.5
2.0
2.5
0.0
0.2
0.4
0.5
0.3
1.0
3.0
0.2
0.4
0.5
0.5 3.0
#
Senteeness (fhetuning)
167
*Sentences (fnetuning
140
#Sentences Minetnning
165
empirical methods by extrapolating the fitted scaling law. Figure [5] shows the critical points as a function of L.LM model size and pretraining data size over different tasks.
The scaling trend and actual value are highly dependent on the downstream task: critical points for task hardly generalize to other tasks. Still, the existence of such points suggests that the selection of finetuning methods should be based on the availability of finetuning examples. When only few thousands of finetuning examples are available, PET should be considered first, either Prompt or LoRA. With sightly larger datasets, LoRA would be preferred due to its stability and slightly better finetuning data scalability. For million-seae scale datasets, FMT would be good.
How does finetuning affect the generalization capability of the base LLM? While finetuning on task-specific data improves task-specific performance, it may specialize the base LLM towards the task and hurt the models' generalization. We examine this for different finetuning methods by performing zhro translation for LLMs finetuned on WMT14 14 En-De and WMT19 En-Zh (Few- shot results in Appendix). We focus generalization to related tasks, where the target language shared, i.c. De and Zh, and generalization should be relatively easier (Johnson et al.) 2017). We report average performance for translation from diverse set of source languages other than English.
8

---

Published as a conference paper at ICLR 2024
Figure 6 shows the results. While specializing downstream task, finetuning could still elicit and improve the generalization for closely related tasks, although the overall zero-shot tramslation quality is inferior. Note whether finetuning benefits generalization is method- and task-dependent. Overall, Prompt and LoRA achieve relatively better results than FMT particularly when the base LLM is large, mostly because L.M parameters are frozen and the learned knowledge get inherited. This also suggests that when generalization capability is a big concern, PET should be considered.
6 RELATED WORK
LLM finetuning With the significant increase of model size, updating all LLM parameters be- comes computationally inefficient and unaffordable. Researchers thus resort to parameter efficient tuning methods that target achieving the best performance with minimal tunable parameters. Efforts in this direction mainly focus on developing efficient tunable modules for LLMs, such as adapters
that insert small feed-forward layers (Houlsby et al., 2019] |aapaa et al., 2019), prefix and prompt
tuning that appends tunable embeddings to the input (Li & Liang] [202] Lester et al., 2021 LoRA
and compacter that adopts low-rank decomposition (Hu et al., 2021 Mahabadi et al. [0221) Bitfit
that adds tunable bias vectors (Zaken et al.] (02I), IA. that scales model activations (Liu et al.) 2022) and QLoRA that leverages quantization (Dettmers et al.
2023), 3 to name a few. While pre-
vious studies reported encouraging performance with PET. c.g. reaching and even surpassing FMT
across various domains (He et al. 2022: Ding et al. 2022 Liu et al. 2022 Dettmers et al. 2023), they mainly focus performance of different finetuning methods under-explored.
or few experimental setups, leaving the question of how scaling affects the
Scaling Laws Recent research has shown that the performance of neural models can be predicted by power-law of model and/or data sizes (Hestness et al. 2017 Kaplan et al., 2020). Such pattern widely exists across different domains and model architectures, such as computer vision (Zhai et al., 202T), autoregressive generative modeling (Henighan et al. 2020). neural machine translation (Gor- don et al. 2021 Ghorbani et al. 2021 Bansal et al. [022]] Zhang et al.] 2022a). multilingual trans- lation (Fernandes et | [2023). multi-modal modeling (Aghajanyan et al. (023) and sparse neural architectures (Frantar al. (202)). These laws provide valuable tool for guiding training deci- sions (Hoffmann et al. 2022) and model development by understanding how model performance evolves with scale, which greatly facilitates the development of LLMs (OpenAT 2023). Unfortu- nately, the study of scaling for LLM finetuning lags behind badly, and our study fills this gap.
The most closely related work to ours is (Hernandez et al. 2021) which explored the scaling for knowledge transfer by comparing finetuning with training from scratch. Our study is orthogonal to theirs with significant difference as key focus is understanding the scaling of different factors for LLM finetuning, rather than the transfer.
7 CONCLUSION AND FUTURE WORK
In this paper, we systematically studied the scaling for L.M finetuning, considering different factors including LLM model size, pretraining data size, finetuning data size, PET parameter size and di- verse finetuning methods. To ensure the generality, we worked on two sets of LLMs, three different downstream tasks (translation and summarization), and three finetuning methods (FMT, Prompt and LoRA). We proposed a multiplicative joint scaling law that could describe the scaling relationship between finetuning data size and each other scaling factor. Extensive results show that increasing LLM model size has higher impact on finetuning than pretraining data scaling, and that scaling PET parameter is ineffective. In addition, finetuning scaling is highly task- and data-dependent,, making the selection of best finetuning method for a downstream task less conclusive.
We acknowledge that our work suffers from some limitations. The proposed joint scaling law is mostly based on empirical results on closed generation tasks without theoretical groundings. Whether it could generalize to different finetuning scenarios requires more experimentation, which however is beyond our current computing budget. Besides, we understand the imperfection of the optimization and evaluation for Prompt and LoRA in some setups. In the future, would like to extend our study to multi-modal LLMs, explore the impact of finetuning data quality and consider open and creative generation tasks well multi-task setup for finetuning.

---

Published as a conference paper at ICLR 2024
8 ACKNOWLEDGEMENTS
We thank the reviewers for their insightful comments. We thank Yamini Bansal for providing valu- able feedback on the scaling laws, Xavier Garcia for reviewing this work with constructive com- ments, Frederick Liu for helpful discussion on PET optimization, and Quoe Le, Apu Shah and Google Translate team for supporting this research.
We also thank the colleagues building the training infrastructure used in this paper: Brian Lester, Rami Al-Rfou and Noah Constant for prompt tuning, Chu-Cheng Lin for LoRA, Xavier Garcia and the T5X team (Roberts et al., (023) for the training framework.
REFERENCES
Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu. Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer. Scaling laws for gen- erative mixed-modal language models. arXiv preprint 1xXiv:2301.03722 2023.
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, al. Palm 2 technical report. arXiv preprint 230.::2233..00000... 2023.
Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. Wav2vec 2.0: A framework for self-supervised learning of speech representations. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
Yamini Bansal, Behrooz Ghorbani, Ankush Garg, Biao Zhang, Colin Cherry, Behnam Neyshabur, and Orhan Firat. Data scaling laws in NMT: The effect of noise and architecture. In Ka- malika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 1466-1482. PMLR, 17-23 2022. URL mlr wwwpsppepeeeeppeppp... vi wwwp////aaaaaaaa...aaaa .am.alm2mmmmmmmm
Ankur Bapna, Naveen Arivazhagan, and Orhan Firat. Simple, scalable adaptation for neural machine translation. arXiv preprint arXiv: 1909 00477, 2019.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal,
Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.03311 2022.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305. 14314, 2023.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-44 4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/N19-11 1423. URL https: Crc.lanchooooy..o///ll------
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang. Zhiyuan Liu, Hai- Tao Zheng, Jianfei Chen, Yang Liu, Jie Tang, Juanzi Li, and Maosong Sun. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogni- tion at scale. In International Conference on Learning Representations, 2021. URL hetps: 7/openrevie ew. net. wtt./fooooo formm d=Y cbFdNTTy
10

---

URL 1tt.//22221eeempee Published as a conference paper at ICLR 2024
Patrick Fernandes, Behrooz Ghorbani, Xavier Garcia, Markus Freitag, and Orhan Firat. Scaling laws for multilingual neural machine translation. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.). Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 10053- PMLR, 23-29 Jul 2023. URL http::: httpps://proceedinng mml.. press/v200 htw:///enandee2eae htm
Elias Frantar, Carlos Riquelme, Neil Houlsby, Dan Alistarh, and Utku Evci. sparsely-connected foundation models, 2023.
Scaling laws for
Yao Fu, Hao Peng, Tushar Khot, and Mirella Lapata. Improving language model negotiation with self-play and in-context learning from feedback. arXiv preprint arXiv:2305.10142 2023.
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Qiao. Llama-adapter v2: Parameter-efficient 2023.
visual instruction model. arXiv preprint arXiv:2304.
Xavier Garcia, Yamini Bansal, Colin Cherry, George Foster, Maxim Krikun, Melvin Johnson, and Orhan Firat. The unreasonable effectiveness of few-shot learning for machine translation. In International Conference on Machine Learning, pp. 10867-10878. PMLR, 2023.
Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Gar- cia, Ciprian Chelba, and Colin Cherry. Scaling laws for neural machine translation. CoRR, abs/2109.07740, 2021. htt.htts///xxii..rrrrg/ wwwpxii.org/ab//////// 07740
Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. Multimodal-gpi A vision and language model for dialogue with humans. arXiv preprint arXiv:2305.04799 2023.
Mitchell A Gordon, Kevin Duh, and Jared Kaplan. Data and parameter scaling laws for neural machine translation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5922, Online and Punta Cana, Dominican Republic, November 2021. Associatio for Computational Linguistics. doi: 10. httpss//aclaanthoooogy. org 2021 www.:22e....mmaa.mm.mm. www.p-main..n.....
Junxian He, Chunting Zhou, Xuezhe Ma. Taylor Kerg-Kirkpattick, and Graham Neubig. Towards unified view of parameter-efficient transfer learning, 2022.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, al. Scaling laws for autoregressive generative modeling. arXiv preprint 11Xiv:2010.14777 2020.
Karl Moritz Hermann, Tomás Kocisky. Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines read and compre- hend. In NIPS. pp. 1693-1701, 2015. URL /ttp:/r//ppp.p.//e http://pppppppp.coooooo/esoo
Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer.
Xrviv preprint arXiv:2102.01229 2021.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory F. Diamos, Heewoo Jun, Hassan Kianine- jad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically empirically. CoRR, tbb//172.00499, 2017. URL httt:://aax/iae hrg. 712. 00409
Jordan Hoffmman,, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203.15555 2022.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2790-2799 PMLR, 09-15 Jun 2019. URL https://proceecings .mr..
press/v97 / abyl 9a html

---

Published as a conference paper at ICLR 2024
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. CoRR, abs/2106.09685, 2021. httpbtttp:://xxttte
abs /tt.//1..... 09685
Melvin Johnson, Mike Schuster, Quoe V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al. Google's 's multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, 5:339-351, 2017.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 2020.
Tom Kocmi. Rachel Bawden, OndA Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel. Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Re- becca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal NovA,,, Martin Popel, Maja PopoviAf, and Mariya Shmatova. Findings of the 2022 conference on machine translation (wmt22). In Proceedings of the Seventh Conference on Machine Translation, pp. 1-45, Abu Dhabi, December 2022. Association for Computational Linguistics. URL https:
ac /ttl/aaaanntte www.ehnanthhaaoooo ogy. www.oogg/2222.. wmc
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parametee--effiieent prompt tuning. CoRR, abs/2104.088fl 2021. URL URL
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequence pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880, On- line, July 2020. Association for Computational Linguistis.. doi: 10. 11.1886/11/2220.a. acl.000..clllme main.703. URL htttps://aclaatthll.oooooo org.oooyy.oor////o wttp:////020...1mma...........
acl-mainmmaa 703
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuuus prompts for generation. CoRR, abs/2101
2021. URL http:ppss//arail...o////////0.0/
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain. July 2004, Association for Computational Linguis-
tics. URL MLL.Etttss//acaatttoloo...
N04
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learn- ing, 2022.
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. CoRR, abs/2106.046e 046447, 2021. URL https::/arxiv..ok abs/ moss/216..0 04647)
Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Noua- mane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models. arXiv preprint arXiv:2305. 162644 2023.
James Cross Onur Çelebi Maha Elbayad Kenneth Heafield Kevin Heffernan Elahe Kalbassi Jan- ice Lam Daniel Licht Jean Maillard Anna Sun Skyler Wang Guillaume Wenzek Al Youngblood
Bapi Akula Loic Barrault Gabriel Mejia Gonzalez Prangthip Hansanti John Hoffman Semarley Jarrett Kaushik Ram Sadagopan Dirk Rowe Shannon Spruit Chau Tran Pierre Andrews Necip Fazil Ayan Shruti Bhosale Sergey Edunov Angela Fan Cynthia Gao Vedanuj Goswami Fran- cisco Guzmán Philipp Koehn Alexandre Mourachko Christophe Ropers Safiyyah Saleem Holger Schwenk Jeff Wang NLLB Team, Marta R. Costa-jussà, No language left behind: Scaling human- centered machine translation. 2022.
OpenAI. Gpt-4 technical report, 2023.
12

---

Published as a conference paper at ICLR 2024
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Al- ice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https:://penzeeveew net forum?
TG8KACXEON
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. (1(1), jan 2020. ISSN 1532-4435.
Adam Roberts, Hyung Won Chung, Gaurav Mishra, Anselm Levskaya, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, et al. Scaling up models and data with t5x and seqio. Journal of Machine Learning Research, 24(377):1-88 2023.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint 22Xiv:22111001000 2022.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761 2023.
Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. MLSUM: The multilingual summarization corpus. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. ber 2020. Association for Computational Linguistics. doi: 10. 10.18655/1110002..mmm.m.mm URL hww.hhtpss/aaaaaato.ooo
Online, Novem-
org/2020 emnlp-main. 647
Thibault Sellam, Dipanjan Das, and Ankur Parikh. BLEURT:
Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pp. 7881-7892, Online, July 2020. Association for Computational Linguis- 704. URL hatpps:/accaantallog..ooo
tics. doi: 10.18653/v
ac maa 704
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pp. 4596-4604. PMLR, 2018.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint 1.Xiv:2303.17880 2023.
Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Steven Zheng, et al. U12: Unifying language learning paradigms. In The Eleventh International Conference on Learning Representations, 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307 (0t.:2337000288.. 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thougee prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 15:24824-248877 2022.
Wen Yang, Chong Li, Jiajun Zhang, and Chengqing Zong. Bigtrans: Augmenting large lan- guage models with multilingual translation capability over 100 languages. arXiv preprint 11ti::2305.18099 2023.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: tuning for transformer- masked inguage-model.
Simple parfmeter-efficient fine- CoRR, 1iss2100.10199 2021. URL
https: pamps://arxive wrw. /2tpg////////.....///// 10199
13

---

Published as a conference paper at ICLR 2024
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. CoRR, abs/2106.0456)) 2021. URL htttp:://axxxiy>> org/ http:/aba2221...000 04560
Biao Zhang, Behrooz Ghorbani, Ankur Bapna, Yong Cheng, Xavier Garcia, Jonathan Shen, and Orhan Firat. Examining scaling and transfer of language model architectures for machine trans-
lation. CoRR, abs/2202.00528,, 2022a. URL https:
httgg/bb//22222 00528
Biao Zhang, Barry Haddow, and Alexandra Birch. Prompting large language model for machine translation: case study. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 41092-41110. PMLR, 23-29 Jul 2023. URL https::/ppoceedaii .tt.s..sss.s.s..//.mmmmm
mlr
V2C2/2hang2 3m. html
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, al. Opt: Open pre-trained transformer language models. arXiv preprint HrXiv:200501066 2022b.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment, arXiv preprint arXiv:2305. 1206, 2023.
14

---

Published as a conference paper at ICLR 2024
A APPENDIX
Table 3: Hyperparameters for different-sized LLMs. "B": billion; "#Layers, #:eads": the number of layers and attention heads, respectively: "Head Dim, FFN Dim, Model Dim": the dimension for each attention head, the feed-forwan layer and the hidden representation, respectively.
LLM Model Size
#Layers
#H
Head Dim
FFN Dim
Model Dim
1B 2B
16
256
8192
2048
20
10
256
10240
2560
4B
24
12
256
12288
3072
8B
32
16
256
16384
4096
16B
40
20
256
20480
5120
Figure 7: Generation quality (BLEURR/RougeLL) for scaling LLM model size and finetuning data size WMT14 En-De, WMT19 En-Zh and MLSum. Overall, BLEURT/RougeL correlates positively with PPL with few exceptions.
FMT (WMT En De)
Prompt (WMT En De)
LORA (WMT - De)
0.72
1610
0.70
Toat RUAT
Sizee
0.68
1.0
0.5
0.0
0.2
0.4
0.6
0.81
1.0
0.0
0.2
0.4
0.8
0.8
3.0
4 Sertennes (fineturing)
1c6
aSentences ('inetuning)
165
#Sentences (fineturing)
165
-Zhh
Eh)
0.68
1e10
155
Text BL EURT
Sizee
1.0
0.60
0.59
0.0
0.5
1.5
2.5
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
# Sentenncss (fineburing)
1e7
#Sentenccss (finatuning)
lc5
#Sentences (finebunnng)
1e5
FMT (MLSum)
Prompt (MLSum)
LoRA (MLSum)
30
1030
Tast Rougel
#Sontences (finetuning) 165
#Sentences (fintuning)
#Sentences (fineturing) 165
Optimization for LLM finetuning. For optimization, we continue the pretraining from the given pretrained checkpoint on finetuning data but with the standard conditional log- likelihoooo loss. More specifically, for each finetuning example, we concatenate the input and target into a single sequence and compute the log- likelihoood on the target alone. Adafactor and cosine learning rate schedule are reused. Note En-De and En-Zh LLM are pretrained for 135K and 98K steps, respectively. All LLMS are further finetuned for up to 200K steps (except for WMT En-Zh (FMT) which is 300K steps) or 100 epochs, whichever comes first. To get the best performance, we optimize the initial learning rate and batch size for different finetuning methods based on the 1B LLM via grid search. Finally, and le 3 for Prompt, LoRA and FMT, respectively, and set the batch size to 16 and 128 for PET and FMT, respectively.
we set the learning rate to 3e
15

---

Published as a conference paper at ICLR 2024
Table 4: Fitted scaling parameters for different settings.
WMT14 En-De
WMT19 En-Zh
MLSum
Params
FMT
Prompt
LoRA
FMT
Prompt
LoRA
FMT
Prompt
LoRA
Scaling LLM model size and finetuning data size
Am
1.2 x 10 3.9 x 10 2.1 x 10 3.3 x 10
8.5 x 10
6.6 x 10?
3.3 x 10²
23
26 0.11
Con
0.52
0.4
0.36
0.34
0.33
0.31
0.24
0.1
Scaling for pretraining data size and finetuning data size
Ap
102
x 10
1.4
10
10
x 10
1.3 10
42
Op
0.21
0.21
0.17
0.11
0.073
Scaling for PET parameter size and finetuning data size
At
1.4
1.2 0.0044
2.6
2.4
0.0027
0.0017
0.0019
0.0026
0.000 22
0.75
0.62
0.62
0.77
0.73
0.98
0.000
0.15
0.051
0.081
0.14
0.015
0.025
0.087
0.025
0.03
Figure 8: Generation quality (BLEURT/RougLL) for scaling pretraining data size and finetuning data size WMT14 En-De, WMT19 En-Zh Zh and MLSum.
FMT (WMT En De)
Prompt (WMT En - De)
LoRA (WMT - De)
Text BLFUAT
0.64
2.0
1.0
0.5
0.58
SSertences (fineturing)
@Sentences (finetuning)
#Senteeccss (fineturing)
FMT (WMT En Zh)
Prompt (WMT En -Zh)
LoRA (WMT En Zhi)
3300
1e0 BLEURT
0.62
0.60
2.0
0.58
0.56
0.5
0.54
0.5
2.5
0.6
1.0
0.2
(fineburing)
1c7
(finetuning)
(finebuning)
FMT (MLSum)
Prompt (MLSum)
LoRA (MLSum)
bell
Text Rougel
2.5
3.5
1.5
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
#Sentences (Tinetuning)
165
#Sentances (iistuning]
165
#Sentences (fnntuuing)
165
Analyzing the critical finetuning data size DG. While Eq. (T) hinders us from computing DG di-
rectly, it still allows for theoretical analysis between two fineturing methods when their performance gap is constant:
? E E2
= H (Ai/A, P1 7
02
Q1
(3)
B
32
which follows another pawer-law. Intuitively, the exponent y captures the transferability difference of the methods the downstream task scaling factor X. We summarize the coefficients for
16

---

Published as a conference paper at ICLR 2024
Figure 9: Generation quality (BLEURT/RougeL) for scaling PET parameter size and finetuning data size on WMT14 En-De, WMT19 En-Zh and MLSum.
Prompt De)
Prompt En-22h
Prompt (MLSum)
0.655
25.5
0.5950
BLEVRTIRUOOL
25.0
Pron Lengght
0.5900
0.645
0.5875
24.0
300
23.3
200
0.640
0.5825
0,5800
22.5
0.50
Senttances (Instuning]ee5
#Sentences Mfnetaning]le5
#Sentences (Inetuning)1e5
LoRA (WMT En De)
LoRA (WMT En + 2h)
LoRA (ML)um)
26
150
0.605
0.650
25
0.645
0.600
100
LoRk
0.640
0.585
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
#eettences (fnnetniing]e5
#Sentences (fineturing:1e5
#Sentences (fnetuuingg]e5
Table 5: Coefficients in Eq.
by comparing different methods setups. "F/P/L" PMT//rompp/LoR.
WMT14 En-De
WMT19En-2
MLSum
Params
vvs.
F
Pvs.
Fvs.
Fvs.
Pvs.
Fvs. P
Fvs. L
P
Sculing LLM model size and finetuning data
3.7 10
2 10
1.6
10
10
16x 10
1.8 10
10th
1.2 10
0.000
-1.2
-24
1.5
-0.12
-0.3
1.9
-21
-1.8
2.3
Sculing pretraining data size and finetuning data size
H1
3.7 10
6.9
10"
102
8.6
10
3.7 10
10
1.6 10
-0.0015
-0.5
1.2
0.26
0.093
2.1
-1.63
-066
-0.63
different tasks in Table across settings.
5 where the value differs greatly over tasks and there are no clear patterns
Fitted single-variable scaling laws for En-De and En-Zh LLM pretraining. We evaluate the model on held-out validation set and fit the scaling law based on PPL. Note that the scaling law doesn't It well extrapolate 16B for En-Zh LLM whose actual performance is than the expectation (This might be caused by pretraining instabilities.). Such mismatch we argue amplified after finetuning shown in Figure
Figure 10:
De LLM
1.3
1.2
Valld PPL
1.1
1.0
0.9
0.8
0.25
0.50
0.75
1.00
1.25
1.50
Model Size
1e10
How does finetuning affect the few-shot capability of the base LLM? Apart from zero-shot translation, also explore LLM's few-shot capability after finetuning. Few-shot generation
17

---

Published as a conference paper at ICLR 2024
Table 6: Held-out litting emorr (L) for the additive and multiplicative scaling formulation over different tasks.
Overall, multiplicative scaling law generalizes better.
Multiplicative
Additive
Scaling Factor
FMT
Prompt
LoRA
Avg
FMT
Prompt
LoRA
Avg
LLM Model Size
0.0052
0.0043
0.0047
0.0048
0.012
0.0076
0.0045
0.0079
WMT En-De
Pretraining Data Size
0.0057
0.0061
0.0084
0.0068
0.0048
0.0075
0.0082
0.0069
PET parameter size
0.005
0.0031
0.004
0.0069
0.0032
0.005
LLM Model Size
0.0075
0.019
0.026
0.018
0.021
0.018
0.029
0.022
WMT En-Zh
Pretraining Data Size
0.002
0.0071
0.0056
0.0049
0.0026
0.0069
0.0058
0.0051
PET parameter size
0.0075
0.0051
0.0063
0.0076
0.0044
0.006
LLM Model Size
0.0066
0.013
0.022
0.014
0.0072
0.015
0.017
0.013
MLSum
Pretraining Data Size
0.009
0.0083
0.0039
0.007
0.0062
0.0046
0.0043
0.005
PET parameter size
0.0081
0.003
0.0055
0.0053
0.0027
0.004
Table Pearson correlation between PPL and BLEURT/RougeL different finetuning methods setups. " the correlation is significant at < 0.01. Note lower PPL and higher BLEURT/Rougel. denote better quality, thus their correlation values are negative. In general, PPL. and /LEURR/Rougel. are highly correlated.
Scaling Factor
FMT
Prompt
LoRA
LLM Model Size
-0.184
-0.986*
-0.988
WMT En-De
Pretraining Data Size
-0.792
-0.9677
-0.980
PET parameter size
-0881
-0.975
LLM Model Size
00984
-0.994*
-0.995*
WMT En-Zh
Pretraining Data Size
-0,9941
-0.9791
-0.978#
PET paramcter size
-0.643*
-0.9683
LLM Model Size
-0.955t
-0.909t
-0.8907
ML.Sum
Pretraining Data Size
-0.941
-0.8331
-0.838
PET parameter size
-0.924
-0.986
only offers a way to inspect LLM's capability but also is interest to downstream applications as it provides an effective way adapt models over domains. Figures [1 12 13 and 14 shows the impact of finetuning on few-shot generation.
We note that FMT degenerates LLM''s few-shot capability in most cases, where adding more fine- tuning data often reduces the few-shot performance. By contrast, PET behaves more robustly which retains most of LLM's sew-shot capability regardless of model size and pretraining data size.
18

---

Published as a conference paper at ICLR 2024
Figure II: One-shot performance (BLEURT/RougeL) for LLM model size and finetuning data size scaling on
WMT14 En-De, WMT19 En-Zh and MLSum. 'Baseline:: performance without finetuning.
Prompt En De)
T02 BLEUST tone Sto0
Modall Staee (log Scale)
2012
0.5
0.2
0.5
#SSnttnnees (finttuiing)
#Sonttncees (finctuiing)
#SSnntonces (finctuning!
#Sontonces (fncturing)
FMT (WMT En Zhh
Prompt (WMIT En Zh)
LoRA (WMT En Ehh
Baceline (WMT En En-
0.7
Text BLEURT (One Shee
0.6
Model Sizw tleg Scake)
201
0.5
0.3
0.2
200
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.2
0.4
0.6
0.8
3.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.5
1.0
1.5
2.0
Sonttomoes (finctuning)
1w7
@Sentonces (fincturing)
1e5
#Sentonces (fh ffnnotuiing
1ed
#Scrtonces (fincturing)
1e?
FMT (MLSum)
Prompt (MLSum)
LoRA (MLSum)
Basellne (MLSum)
Test RestRe (onc Shord
0.0
0.2
0.6
0.8
1.0
0.0
0.4
0.6
0.8
1.0
#Sentences (frnturinn)
165
#Sentences Minntuning!
165
#Sontences (fineturing)
165
#Senterces
1c5
Figure Five-shot performance (BLEURT/RougeL) LLM model size finetuning data size scaling on WMT14 En-De and WMT19 En-Zh.
(WMT En Del
Prompt En Del
Del
Dee
Text SLEURT (nve Shod
Modiel Sizw Unng Scaale)
2010
0.6
0.3
0.4
0.8
3.0
0.2
0.6
#Sontoooos (finctuning)
1e8
#Sentonces (fincburin))
#Sontorcos (finctuiing
145
#Sontonces (fincturing)
FMT (WMT En Zh)
Prompt (WMT En En++Zh)
LoRA (WMT En-ZZh
Baseline (WMT En En-ZZ)
BLTTBL live ShoE!
Model Size (lng Scriil Scrkke)
0.1
20"
0.0
0.5
1.0
1.
2.0
25
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
10
0.0
0.5
1.0
1.5
2.0
2.5
Sentamocs (Tinutuning)
167
@Sentances (fowturing)
165
#Sentences (Tratuuing!
165
#Santances (fineurrnng)
167
19

---

Published as a conference paper at ICLR 2024
Figure 13:
One-shot performance (BLEURT/RougeL) for pretraining and finetuning data size scaling on WMT14 En-De, WMT19 En-Zh and MLSum. 'Baseline:: performance without finetuning.
Prompt En De)
De)
De)
Precsin Size (ittken, Log Scale)
BLEUST tone Stoo
3
0.5
2x
0.4
0.3
6
4
1.0
0.2
0.6
Sonttences (finetuning)
1u5
#SSentences (fincturing)
#SSnntonces (finctuning!
#Sontonces (fnntuningg
FMT (WMT En Zhh
Prompt (WMMT En Zh)
LoRA (WMT En En-
Baseline (WMT En
Protrain Dat Sto DRookens Scate)
0.6
Test BLEURT (One Shee
2x12t
0.4
6x
4xx10
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.2
0.4
0.6
0.8
3.0 1e5
0.0
0.2
0.4
0.6
0.8
1.0 Ted
0.0
0.2
0.4
0.6
0.8
1.0
Sontomoess
(finttuning)
1w7
@Santonces (fincturing)
#Sontonoes (fin (finetuuing)
#Sontonces (fincturing)
1e5
FMT (MLSum)
Prompt (MLSum)
LoRA (MLSum)
Basellne (MLSum)
Test Rougel 1one Short
Sk
Preszain Data Size (4tobens, Log
2x12t
6x100t
4x12le
0.0
0.2
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
#Sentences (finttrinn)
165
#Sentences Minebuning!
105
#Sontencess (fineturing)
165
#Senterces
105
Five-shot performance (BLEURT/RougeL) pretraining finetuning data size scaling on WMT14 En-De and WMT19 En-Zh.
Figure 14:
FMT (WMT En De)
Prompt En De)
Precoin Data Sic (RRooOn Log Scalo)
Teak BLEURT IRve Shod
5x10f
0.3
1o
0.2
0.1
0.4
3.0
0.2
0.6
#Sontonoos (fincuunigg
1e8
#Sentonces (fincturing)
#Sontoroos (finctuiing
145
#Sontonces (fincturing)
FMT (WNT En Zh-
Prompt (WIMT En Zh)
LoRA (WMT En En-ZZh)
Baseline (WMT En
Pretoin Data Ste CPENenS. Log Saaee
Test BLEURT lfve Shot!
2x10f
0.4
6x1ck
0.1
0.0
0.5
1.0
1.5
2.0
25
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.5
0.8
100
0.0
0.2
0.4
0.6
0.8
1.0
Sentamoes (Tinutuning)
167
@Sentances (fiwturing)
165
#Sentences (fraturing!
165
#Sentinces (finturing))
165
20


Reviews:
[{"review_id": "8sEniik4Fo", "reviewer": "Reviewer_gFLk", "summary": "This paper proposes a scaling law for language model fine-tuning. The claim is supported by experiments with fine-tuning in machine translation and summarization tasks. The other scaling factors besides fine-tuning dataset size are LLM size, pretraining data size, and PET parameter sizes for the Prompt Tuning and LoRa fine-tuning regimes. The core finding is that a multiplicative scaling model achieves better fits than a closely additive scaling model.", "questions": "1. Table 2 shows that the multiplicative scaling is better than the additive one for WMT14 En-De. However, the multiplicative model seems strictly more expressive than the additive one, so I am not sure this is surprising. Why not add even more terms for even more expressivity? Can there be, and should there be, some controlling for the complexity of the law itself?\n\n2. Where are the counterparts of Table 2 for the other tasks and other metrics (besides perplexity)? My apologies if I am overlooking something in the paper. It seems like these other numbers would be given prominently.\n\n3. The paper does say that the BLEURT-RougeL picture \"shows high correlation with the PPL scores in general\", but eye-balling Figure 7 in the appendix doesn't support this too well, though it's easy to imagine that the quantitative picture is different. But what is the quantitative picture?\n\n4. The promise of the paper is that the scaling law will provide guidance for people seeking to fine-tune. However, the guidance seems to me that perplexity will generally go down for all methods, but that the best method and precisely ideal stopping point will be highly variable. This guidance is very familiar and doesn't need to be characterized with a \"scaling law\". Is there more specific guidance implicit in this work?\n\n__The authors gave thoughtful answers to the above questions, which helped me understand the work better, and I raised my score by 1 point in response.__", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "It's excellent to see scaling law investigations extended to fine-tuning, where they can potentially provide a lot of practical value.\n\nThe paper is generally clear and direct, and the experimental findings are quite rich.", "weaknesses": "The precise nature and strength of the findings is difficult for me to discern, and I am not sure of the value of this result for fine-tuning work. All of this leaves me concerned about the paper, but I am open-minded. I am going to express my concerns as questions and see what the answers are.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mvZRfcbiyM", "reviewer": "Reviewer_R4DF", "summary": "The work provides a set of simple but extensive scaling law experiments on comparing pretraining, fine-tuning, and parameter-efficient tuning LLMs on translation and summarization tasks.", "questions": "- What model sizes are used in Figures 2/3/4?\n- Figure 5 is unclear to me. Does the critical point refer to when A outperforms B in \"A vs B\"?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This work provides a set of straightforward and well-motivated set of experiments.  The authors are clear where there are reasonable scaling patterns, and cases where there is no discernible pattern.", "weaknesses": "- Some typographical/grammar mistakes. E.g. \"propmt\" in Figure 2, \"infers\" rather than \"implies\", \"there exits a critical point\" etc\n- It is unclear to me if the set of tasks chosen (translation, and multi-lingual summarization) are representative of broader applications of fine-tuning. However, I think the results stand on their own for this set of narrow applications at least.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KrkWpcsrLG", "reviewer": "Reviewer_wU43", "summary": "This paper describe a multiplicative joint scaling law, considering different factors including LLM model size, pretraining data size, finetuning data size, PET parameter size, on machine translation and summarization tasks.", "questions": "Please see weaknesses above.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces a multiplicative joint scaling law and offers comprehensive experiments to empirically demonstrate that this law applies to both machine translation and summarization.\n- This paper provides insightful observations regarding fine-tuning, especially parameter-efficient fine-tuning (PEFT) which I think is the most interesting part for the community now, and its relationship with parameter or data size scaling laws.", "weaknesses": "Minor:\n-The paper evaluates only MT and summarization. Including more tasks or languages, particularly low-resource translations, would enhance the paper's comprehensiveness.\nMajor:\n\nMajor:\n-Regrettably, it seems that the proposed scaling law may exhibit a significant mismatch for parameter-efficient fine-tuning when the model size is 16B. This raises concerns about the law's applicability to larger models, especially those of 70B or exceeding 100B in size.\n-The paper omits some details regarding model training. Conventionally, MT models employ an encoder-decoder architecture. I believe all models in this study are decoder-only. How did the authors approach training with a decoder-only architecture for MT tasks? How might this differ from the scaling laws when using an encoder-decoder model? What prompts were utilized for MT training?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TUg8BuZ9bR", "reviewer": "Reviewer_oDvP", "summary": "This paper examines scaling laws for LLM adaptation methods (finetuning, PET like LoRA and PT), where the LLMs were trained on two different machine translation corpora (En->ZH and EN->De) and adapted to either the other translation task, or article summarization in an unseen language.\nThe paper is primarily empirical, and the goal is to fit scaling laws and identify general trends that could transfer across problem domains (e.g. vision, multimodal, etc).", "questions": "- How does zero-shot evaluation work in Figure 7? If the source language is unseen, how is the model able to get nontrivial performance on the task?\n\n### Typos:\n- P4 footnote: is pretty week → pretty weak\n- Fig 2. center column: Propmt → Prompt", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "### Overall\nThe paper is clearly written. The methodology is generally explicit, and the authors do a good job of interpreting the (many) experiments in the paper.\n\nThe figures are clear and well-organized. \n\nExcellent references to existing work in identifying scaling laws.\n\n### Experiments\nThe experimental results are somewhat expected (given enough data, use FMT, with limited data, use PT or LoRA). The authors do a good job of explaining and justifying these conclusions given experimental evidence, with the appropriate amount of uncertainty given the noisiness of the data. I think that correctly calibrating the uncertainty is a great strength of the paper. \n\nExcellent analysis of scaling different axes (pretraining data, model size, finetuning data)", "weaknesses": "### Experiments\n#### Fig 2:\nGiven the poor extrapolation to the 16B model, I'd like to understand the cause -- i.e. is this the right form of the equation, or does this multiplicative scaling break down at larger parameter counts?\nOne way to test this might be to compare the fit by including 16B in the fitting procedure, vs excluding it. \n\n#### Fig 4:\nI liked the analysis of scaling to larger PET settings (LoRA and PT in Fig 4). Since the datasets seem to support up to 1e-6 and 1e-7 training sentences, I wish the authors had compared the scaling on even larger finetuning datasets to enable direct comparison to FMT in Figure 3. \n\n#### Fig 5\nI didn't really understand Figure 5. My my understanding is that the figure analyzes the fitted power laws, and the x-y- point indicates the estimated amount of finetuning necessary to achieve the performance parity between the two approaches at different model sizes. Does the ordering here matter (e.g. FMT vs LoRA, or LoRA vs FMT) -- my understanding is that it should not.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)