Predictable Scale (Part II) --- Farseer: A Refined Scaling Law in LLMs
Authors: Houyi Li
Wenzhen Zheng
Qiufeng Wang
Zhenyu Ding
Haoying Wang
Zili Wang
Shijie Xuyang
Ning Ding
Shuigeng Zhou
Xiangyu Zhang
Daxin Jiang
Venue: NeurIPS 2025 spotlight ()
Keywords: Large Language Model
Scaling Law
Primary area: deep_learning

Abstract: Training Large Language Models (LLMs) is prohibitively expensive, creating a critical scaling gap where insights from small-scale experiments often fail to transfer to resource-intensive production systems, thereby hindering efficient innovation.
To bridge this, we introduce Farseer, a novel and refined scaling law offering enhanced predictive accuracy across scales. By systematically constructing a model loss surface $L(N,D)$, Farseer achieves a significantly better fit to empirical data than prior laws (e.g., \Chinchilla's law). 
Our methodology yields accurate, robust, and highly generalizable predictions, demonstrating excellent extrapolation capabilities, outperforming Chinchilla's law, whose extrapolation error is 433\% higher. 
This allows for the reliable evaluation of competing training strategies across all $(N,D)$ settings, enabling conclusions from small-scale ablation studies to be confidently extrapolated to predict large-scale performance. 
Furthermore, Farseer provides new insights into optimal compute allocation, better reflecting the nuanced demands of modern LLM training. To validate our approach, we trained an extensive suite of approximately 1,000 LLMs across diverse scales and configurations, consuming roughly 3 million NVIDIA H100 GPU hours. To foster further research, we are comprehensively open-sourcing all code, data, results (https://github.com/Farseer-Scaling-Law/Farseer), all training logs (https://wandb.ai/billzid/Farseer?nw=nwuserbillzid), all models used in scaling law fitting (https://huggingface.co/Farseer-Scaling-Law).

Predictable Scale (Part II) Farseer: A Refined Scaling Law in LLMs
Houyi Li' Fudan University & StepFun, China
Wenzhen Zheng* StepFun, China
Qiufeng Wang StepFun, China
Zhenyu Ding
Haoying Wang
Zili Wang StepFun, China
Xi' an Jiaotong University, China
Xi'an an Jiaotong University, China
Shijie Xuyang Fudan University & StepFun, China
Ning Ding
Xi' an Jiaotong University, China
Shuigeng Zhout Fudan University, China
Xiangyu Zhang
Daxin Jiang StepFun, China
StepFun & Megvii Technology, China
Abstract
Training Large Language Models (LLMs) is prohibitively expensive, creating a critical scaling gap where insights from small-scale experiments often fail to trans- fer to resource-intensive production systems, thereby hindering efficient innovation. To bridge this, we introduce Farseer, a novel and refined scaling law offering enhanced predictive accuracy across scales. By systematically constructing a model loss surface L(N, D). Farseer achieves significantly better to empirical data than prior laws (e.g., Chinchilla's 's law). Our methodology yields accurate, robust, and highly generalizable predictions, demonstrating excellent extrapolation capabilities, outperforming Chinchilla's law, whose extrapolation error is 433% higher. This allows for the reliable evaluation of competing training strategies across all (N, D) settings, enabling conclusions from small-scale ablation studies be confidently extrapolated to predict large-scale performance. Furthermore, Farseer provides new insights into optimal compute allocation, better reflecting the nuanced demands of modern LLM training. To validate our approach, we trained extensive suite of approximately 1,000 LLMs across diverse scales and configurations, consuming roughly 3 million NVIDIA H100 GPU hours. To foster further research, comprehensively open-sourcing all code, data, results all training logs", all models used in scaling law fitting
" Equal contribution.
1 "oonespon author. Ning Ding: ding ding.ning@en wwing@xjtu.ed.ccom Shuigeng Zhou: sgzhou @ fudan.edu.cn pettts://githhn www.//////eerreeeeeee Scannnng-aaaaaaaaasseeee
phttps://wandee htt.a////aa//.llii////aeeeeeeemr hnwnnsssrredd
ct Scaling-
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Chinchilla ila fits
devistes from the middle, the Chirohilla's X error becommss divergint
Liama
#0B
Average Relative Error
Liama 3.1
*78
#14B
0.010
Optimal DIN Rato
Owen
#7B
higher
Qwen 2.5
0.008
x3.8
Farsear
w7B
0.006
Chinchllaa
w138
0.004
10
#72B
0.002
Farsear Chinchila
compergent
#708
0.000
10
10"
10
10"
10
10"
Model Size
Compute Budgel C (6ND)
(a) Comparison of Average Relative Error
(b) Optimal D/N Ratio vs. Compute Budget
Figure 1: Farseer beats Chichilla [20]. (a) Average relative error (BPC) vs. model size N for Farseer (red) and Chinchilla (blue). Chinchilla, lacking high-order cross terms, fits only near the central N and its error diverges as model size grows. In contrast, Farseer's error is 232% lower within the fitted range and remains stable across the full N range. (b) Chinchilla's rule of thumb (D/N x 20) is valid only at moderate budgets (C zs 1020-102h but it underestimates the requirements for larger scale regimes. In contrast, our analysis predicts steadily increasing optimal D/N, which consistent with the actual training configurations used in recent large language models (e.g., Llama 3.1 [16], Qwen3 [45]. etc.).
Introduction
Recent remarkable progress in Large Language Models (LLMs) such as GPT-3 [6]. GPT-4 [2], and Llama [39] is largely attributed to scaling laws, notably proposed by Kaplan et al. [22]. These laws demonstrate that model performance, typically measured by loss L, exhibits a predictable improvement trend as model parameters (N) and training data (D) increase. This relationship follows power-law dynamics expressed as:
ON
N N
Dr
DN =
+
D
Subsequently, DeepMind's s Chinchilla [20] pro- posed optimal compute-scaling strategies and a revised scaling law:
Farsear
Chinchila
A
B D
L(N. D)
E.
No
BPC
where all parameters other than N and D are fitted. While valuable, Chinchilla's formulation has limitations in modeling the interplay be- tween model size and data scaling. Specifically, we argue that its term DV. which describes how loss improves with data D, uses constant pa- rameters B and 3. This implies that rate of improvement with data is uniform across all model sizes N, thereby lacking adequate mod- cling of N's influence on data scaling dynamics. Consequently, Chinchilla's law tends to capture an average data scaling behavior across the N values used for fitting. As result, it performs best for models near the midpoint of this range, but less accurately at the extremes of N, as shown in Fig. (a). This characteristic inherently limits its extrapolation capabilities, especially for model sizes significantly different from calibration set, making cost-effective prediction across arbitrary (N, D) surfaces persistent challenge.
9 900%
0.5
4.10%
10
Dateset Size (tokens)
Figure 2: Empirical BPC values (Ground Truth) are plotted alongside fits from Farseer and Chinchilla for fixed model size of N = 6B. Farseer yields predictions that lie almost exactly on the ground truth curve, whereas Chinchilla's fit exhibits sys- tematic under- and over-estimations, particularly at small and large D.

---

These limitations in accurately predicting performance with existing scaling laws underscore broader difficulty: the very exploration of superior scaling laws severely hampered by a significant scaling gap. The immense computational cost of state-of-the-art LLM training (often >> 102s FLOPs) means insights from affordable, small-scale experiments frequently fail to transfer to production scales. This "scaling gap" refers to the phenomenon where conclusions drawn from small-scale experiments do not consistenlly hold when scaling up to larger models. A more accurate scaling law can mitigate this by providing more reliable predictions, making small-scale exploratory experiments more valuable (see Appendix E for an example).
To bridge this gap, we introduce Farseer, a refined scaling law (Eq. 3) and an experimental method- ology developed from training over 1, 1,000 LLMs. Farseer employs Differential Piecewise and Multi-round Iterative fitting to model the loss surface L(N, D):
L(N, D) 2aa-N"+ + e D=e N
where all parameters other than N and D are fitted. Our key contributions are:
Refined scaling law. We propose Farseer (Eq. 3), providing a significantly more accurate fit to empirical LLM data (Fig. 1 (a) and 2) through novel fitting approach where data scaling effects are explicitly Nedependent.
Superior extrapolation. Farseer enables reliable large-scale performance prediction from tractable scall-scale experiments, effectively bridging the scaling gap.
Improved compute guidance. Our analysis yields new, data-driven insights for optimal D/N allocation in modern LLM training, diverging from simpler heuristics (Fig. 1 (b)).
Comprehensive open-sourcing. We release all models, data from 11,00 trained LLMs, detailed logs, and the Farseer code to foster further research.
2 Preliminaries
2.1 General Loss Formulation
Our Farseer method systematically samples L(N, D) via small-scale experiments (smaller N, D) and applies a mathematical scaling formulation to predict performance at significantly larger scales. This enables robust assessment of training strategy's 's scaling potential.
For clarity, we decompose the loss L,N, D) as:
L(N, = E + LN(NN + LD(D) + LND(N. D)
(4) loss component, (D( is the
where E is constant term, LN (NN is the sonel-sizz-depende data-size-dependenn loss component, and LND(N, , D) represents the interaction effect between N and D. Farseer focuses on determining the functional forms and parameters of LN (D()) and LND(N, D) using only small-scale experimental data.
2.2 Basic Settings
We train approximately 1,000 models with standard language modeling objective [41, 11, 31j. The training data comprises a mix of web text, mathematical content, and code. This data processed using Byte Pair Encoding (BPE) [14] tokenizer with vocabulary size of 65,536. We evaluated two distinct data mixtures following [39, 25], with specific component weightings detailed in Appendix B.
Our model architecture design follows the Llama [39, 16] design, using the AdamW optimizer [29] with 3 values of [0.9, 0.95], an epsilon of 10- a weight decay of 0.1, and a gradient clipping norm of 1.0. We set the parameters N and 1) for these models using a geometric progression with a common ratio of 22, and specific details can be found in the Appendix F. A visualization of the experimental (N, D) grid provided in Fig. 7 (blue circles). Our learning rate schedule includes linear warmup for the first 2,000 steps, followed by cosine decay to 1 x 10-5 for the remainder of training. The model uses a fixed sequence length of 2,048 tokens. Ultimately, we define model size N by excluding embedding layer parameters (see Appendix G for details). Further elaboration on the metric computations, and considerations for optimal hyperparameter settings [25], including model aspect ratios, can be found in Appendix A.
3

---

10-2
AcUNN D5
AaLiN 03
N- 01++08
N-2 21++00
D-1. 1.100000 D-1.10e-08
Re3.080 98 66
D-4 762+09
N-3 88:+00
Ded 00c+00
No 98
De1 60c+10
N=7 58
Dw1 3te+10
Nw1 1i-1ele100
Dw5 3844-0
N-1 5341-09
De4 52e
Re2.18e11 66
Dwl 8121
N-3 18:100
D-1 52..22
Tp
T0P
10P
18
18
107
Data Size [takens)
Model Size (params) (b)
Data Size D (tokens) Ayd GIN)
Model Size N [params))
= 6IN) D
AM
N-100)
Figure 3: Log-Log analysis of the differential BPC terms ADL and A L as univariate functions of D and N. respectively: (a) ADL D fixed N (R² 0.9807); (b) ADL vs. N at fixed D (R² = 0.7457); (c) L D at fixed N (R² 0.8484); (d) AN vs. N at fixed ( R = 0.8733). The high R in (a) (0.9807) demonstrates consistent power-law relationship between A and D, so we adopt this form in our main analysis. The R value is the average across all fitted lines. As detailed in Appendix I, the associated terms A(N) and B(N) further offer improved numerical behavior.
For evaluation, utilize high-quality, specially constructed validation containing 30 million tokens. This dataset is entirely separate from our training data, ensuring that all validation samples are unseen. It comprises diverse mix of web pages, books, and academic papers, rigorously filtered be more distilled and of higher quality than the training corpus. Instead of validation loss, we employ Bits Per Character (BPC) our primary evaluation metric to measure the model's compression efficiency on this validation set. Further details are provided in Appendix D.
3 Methodology
This section summarizes our Differential Piecewise Fitting methodology the performance surface L(N, D) (D) + LND(N, D) + E + (N(N). Section 3.1 presents a differential details the modeling of the combined data-dependent terms LD(D) + LN LNN, D) (Stages & 2), guided by empirical observations to a power-law form in D. Appendix provides the full algorithm. Finally, Section 3.3 describes fitting the model-dependent term E + LN (Stage 3). The overall procedure yields fitted scaling law: L(N, D) fo (N: 0% (Fa(N())))))) fLw (N:
analysis establishing the necessity of the interaction term LND (N, D). Section
3.1 Differential Analysis for the Existence of Higher-Order Interaction Term LND(N, D)
To understand the contributions of the terms in Eq. (4), we employ a differential technique. We compute finite differences of the loss L(N, D):
ADL(N, D) = L(N, D) L(N, AD), this difference primarily reflects changes in the data- dependent terms: A
N, D) z [Lp(D) Lp(AD)] [NND(N, D) - LND(N, AD)). This operation cancels E and (NN).
AN D) L(N, L(AN, D). Similarly, this difference highlights changes in model-size dependent terms: LN
(LNND) D) s [Lv(N) - LN(AN)) + [LND(N,
LND(N, AD)]. This cancels E and LD(D).
Empirical analysis of these differential terms, as shown in Fig. 3, compellingly demonstrates the necessity of non-degenerate interaction term LND(N, D):
AN, D) exhibits systematic dependence both N and D (Fig. 3 (a) and (b)).
AN,(N, D) similarly depends systematically on both N and D (Fig. (c) and (d)).

---

This dual dependence confirmss that (ND(N. D) cannot be simplified to depend only on N or only on D. nor can it be additively separated. This motivates a fitting approach that can capture such coupled interactions.
3.2 Functional Form for Data-Dependent Term Ln(D) + LxD(N,
This section details the fitting of the data-dependent terms (D( + LND(N, D), guided by the differential analysis in Sec. 3.1, with the goal of deriving parametric forms for the functions La and fB in Eq. 5. The insights from the differential analysis in Section 3.1 guide the modeling of the data-dependent terms (D) + (N D). Specifically, the behavior of ( D) when plotted against D on log-log axes (Fig. (a)) reveals a consistent and striking linear trend across various model sizes N. This indicates robust power-law relationship for A ADL(N, D) with respect to D.
This empirical power-law behavior of the difference term AnL(N, D) strongly motivates modeling the integrated data-dependent loss component, Ln(D) + (ND(N, D), using a power law D. We therefore propose that this component can be approximated as:
LD(D) + (NV D) B(N)D A(N) + ( D) = D +ER(N, D) (5)
D-
The functions A(V) (A) and B(N) fB (N: :B) capture the model-size dependeneies of the exponent and coefficient, respectively. Where, e [N, D) is stochastic term jointly dependent on N and D that aggregates two sources of variability: the experimental measurement noise inherent to each (N, D) setting and the residual error arising from the fit of the parametric model.
Stage 1: Initial estimation. For each unique model size N, first compute the observed quantity of loss difference, it also can be approximated as:
RN (D) L(N, D) L(N, B(N) X A RN (D),
(6)
where À = V in experiments. And R,y (D) is predict value of (D) which is projected to log-log space for linear regression:
log (D (D) = log( BN) - AA * log(D),
where By BN (1--11----0 Bx Axx). Using Normal Equation [15] for minimizing the error of the loss differences of each N, respectively, as RR = ED(Ry (D) (D)))
Consequently, for each model size N we obtain discrete pair of parameters- the exponent Ay and the coefficient B -yhih constitute the the best linear fit for that particular model size. Collecting these pairs yields the arrays [An] } and { {.
Stage 2: Parameterization and Iterative Refinement. Building on the discrete estimates (AN: Br} obtained in Stage 1, now seek continuous functions f (N; O and I' (N; 0p) that simultaneously (i) admit a compact analytical form and (ii) minimize the global error of the loss differences & Ex(.R.N)
Let g = {9(1). .r.)....} denote a small dictionary of simple, monotone transformations (identity, logarithm, and power functions were found sufficient in practice). For every ordered quadruple (g., 95 gkv 9m) E g* we perform the coordinate projection
9(AN) ai 9, (N) bi + EAAN,
(n( 02 gm (N) + b2 +
(8)
where the coefficients (a1, b1) and (ag, b2) are obtained via the least squares method, yielding best estimates.
For every candidate transform pair we compute the projection-space residual sums of squares la Ex=EN E2,N and lg = ΣN e The transform quadruple that minimizes & + lg is selected. As results, Gis 9k is log function and gj.9m 9m is power-law function, minimized the residuals, implying that both fA and fg follow a frrechhd--epooeent
form:
fA(N:0A) = exp(aINa + +b), fB(N:0B) = exp(a2N8 + bz).
(9)
(10)
5

---

Actual AIN)
1000
Actual B(k)
0.40
Predicted A(N)
Predicted B(N)
Value
0.35
B(NN) Value
0.30
0.25
200
01
10a
10"
N
Predicted A(N)
Predicted
Figure 4: Fits of the terms A(N) (the data scaling exponent) and B(N) (the data scaling coefficient) using stretched exponential functions. Blue dots denote actual values derived from discrete fits, while orange crosses denote predictions from the continuous functions. The close aligmment demonstrates the accuracy of the stretched exponential parameterization across a wide range of model sizes N.
First, we determine initial parameters for (N; 0n) by fitting the function log( Bx) - a2 b2 the discrete data points {(N, Bx)) obtained in Stage This provides an initial estimate for the parameter set 8g (az, b2, B). An analogous procedure is performed for (N(N;0A) and its corresponding parameters.
Subsequently, we refine the exponents a and / using an iterative refinement strategy: (i) holding / fixed, update minimize the global residual error εn; (ii) holding the new fixed, update 3 according to the same criterion. This process is repeated until convergence, which empirically takes only 1-2 iterations. Implementation details, well systematic comparison of alternative functional families and their empirical errors, are provided in Appendix C. Fig illustrates the quality of the resulting fits for (NV and fa (N) using these stretched exponential forms.
3.3 Functional Form for Model-Dependent Residual E (N(NN
(LD(D)+ (LD(D) LND(N, D)), where L(N, D) is directly obtained from experimental data whereas the exact form of LD(D) + LND(N, is not experi- mentally accessible. Hence E + Ln (N) cannot be directly observed. It can, however, be accessed indirectly through
By construction, E' (N = L(N,
ONN, D) & L(N, D) D-N)) E LN(N) (RN,NN)RRNNDD))))
ER(N
(11)
where RNN, D) is the residual defined in Section 3.2. Grouping by model size and averaging over D yields
G(N) & Avg(O(N, E LN(N) Avglen((,
(12)
D
The difference
O(N, =n D) Avg(eR(N, D)]
(13)
diagnostic of the accuracy of the approximation B(N)D-A(M) If the fit B(N))--(NN) accurately approximates (D) + LND(N, D), the residual variable (R D) should behave like white noise with respect to N and D. As shown in Fig. 5 (b), this quantity is distributed as nearly Gaussian white noise across all N, with its amplitude shrinking from 2 x 10-3 to 4 x 10-4 as N increases. Given that G(N) >> 0.21, these fluctuations are negligible. From Fig. (a), we conclude: (a) the power-law model B(N) A-AN) provides sufficiently accurate approximation to LD(D) + LND(N, D); (b) the observable G(N) is therefore reliable estimate of E LN
T.D(D) + IND(N, D).
Fitting E + (NV. We now model G(N) fLW (N: (LN) by repeating the transformation-fi fit- selection procedure of Section 3.2. For each pair Jt. Uÿ E G we regress gi( (.(G(N) a3 9j (N) by via the normal equation and choose the transform pair that minimizes
residual sum of squares The optimal form identified was a logarithmic transformation for g: and power-law transformation

---

X10
5
Actual
Difference (E + UIN) -Mean)
Predicted
0.004
0.50
++ U(N) Value
0.002
39
0.45
0.09%
0.000
2
0.40
-0.002
10a
N
(a) Actual E + U(N) vs Predicted Values
(b) Difference between E U(N) and Mean
Figure 5: Predicted vs. actual values for the model-dependent residual term G(N) E + Lx (N). (a) Comparison of actual G(N) values (blue dots) and the fitted prediction exp(a3 (a + b3) (orange crosses) over model size N. (b) Residuals O(N, D) G(N) versus N, colored by data size D, confirming negligible variance across and validating G(N) as a reliable estimator for E + (N(N).
0.006
<250
0.06
0.005
Relative Error
0.04
0.004
0.003
264
0.02
0.002
292
305
0.01
0.001
329
104
0.00
1.9
2.3
3.8
N Upper Bound for Fitting
1e9
(a) Robustness of Baseline redipe
(b) Robustness of EN-ZH recipe
Figure 6: Robustness and data distribution generalizability of Farseer. (a) Relative error on excluded 6.4 B models as a function of the largest model size used in fitting. assessing robustness to fitting data volume. (b) Relative error on excluded 3.2 models trained with an English-Chinese data recipe, demonstrating structural generalizability to different data mixes. Circle size and adjacent numbers indicate the number of model-size points used for fitting in each case.
g)(N) = N for Jj. The optimal exponent y is identified via grid search minimizing the fitting error, leading to stretched-exponential form
E (N(N) G(N) exp(as ag N by)-
(14)
Fig. (a) shows that this fit attains relative error of only 0.09%. Given the negligible estimation error of G(N) in approximating E Ly we have thus completed robust modeling and fitting of the model-dependent residual. Combining these stages yields the fully specified scaling law in Eq. 3, with all functional forms and parameters fixed. Further details, ablation studies, and noise analyses are provided Appendix C.
Farseer's Properties: Robustness, Generalization, and Extrapolation
This section evaluates key properties of Farseer: its robustness to fitting data amount, its gen- eralization across different training data recipes, and its extrapolation ability beyond the fitting range.
4.1 Robustness to Fitting Data
We measure how prediction accuracy changes with data size to assess Farseer's robustness, parameter stability, and prediction consistency more model-size points become available. Specifically, fit Farseer using subsets of models with progressively increasing upper bounds on model size N. For each fitting process based on subset, we consistently measure the relative error the 6.4

---

B models, which is deliberately excluded from all fitting subsets. As illustrated in Fig 6 (a), the predicted relative error on the excluded 6.4 B models decreases significantly as more data points are included in the fit. When fitting only models up to 1.9 B parameters, the relative error is 6.01 x 10-3; expanding the fit to include models up to 5.4 B reduces it to only 5.87 x 10-1, Crucially, the relative error decreases nearly monotonically as additional model-size points are incorporated. This trend highlights Farseer's accuracy and robustness to fitting-data volume, yielding reliable predictions with limited data and predictable gains as more data are added.
4.2 Data Distribution Generalization
To investigate the generalizability of Farseer, we evaluate it different training data setup. All prior experiments used the Baseline data recipe, specified in Appendix B. Fig 6 (b) extends this analysis evaluate Farseer using data from models trained with the English-Chinese (EN-ZH) recipe (also detailed in Appendix B), which represents dramatically different bilingual data mix. Similar to the robustness analysis, as the fitting data increases, the relative error on deliberately excluded 3,2 B models trained with the EN-ZH recipe steadily converges and stays consistently low, reaching 7.60 10-4 the final point. This shows that Farseer captures key scaling trends and delivers reliable predictions even under dramatically altered bilingual data mix. This result underscores Farseer's s structural generalizability across diverse training-daie -ata compositions and suggests its applicability to models trained on varied datasets. As an illustration, detailed surface comparison of different English-Chines data mixture ratios provided in Appendix E.
4.3 Extrapolation Capabilities
To quantify the extrapolative capacity of Farseer, we fit its parameters on a /2-spaced sampling grid of small model sizes and dataset sizes, then predict BPC values for substan- tially larger and ofi- combinations. Fig. 7 presents six such extrapolation targets (red
x10
0.30% * 0.50%
0.2%
Fittin Extropolated points
Dateset Size D (tokenss
0.63%
stars) that situate well outside the calibration
0.20%
region (blue circles). Most notably, the 25.1 B model extends the evaluation domain beyond
0.2% 25.1 B
the largest calibrated scale by more than an order of magnitude. Despite this, Farseer's s pre- diction for this model exhibits a relative error of merely 0.47 %. The remaining validation points selected to assess extrapolation at both increased dataset sizes and off- grid (N, D) con- figurations demonstrate similarly low relative errors, ranging from 0.26 % to 0.72 %. Across these extrapolation targets. Farseer's average relative error is just 0.50 %, whereas the Chinchilla scaling law exhibits an average relative error of 2.68 % a 433 % increase.
10"
2.5 10
Model Size N (parameters)
Figure 7: Extrapolation of Farseer. Blue circles represent the grid of (N, D) employed to fit. Red stars denote validation points beyond that distribu- tion, including 25.1 B model, larger dataset sizes, and off-grid combinations. Annotated percentages give the relative errors of each extrapolated point.
These consistently low errors demonstrate that Farseer accurately captures the smooth functional dependence of BPC on both model and dataset scale. In particular, the highly accurate extrapolation at the 25.1 B model provides compelling evidence that Farseer generalizes robustly to previously unfitted scales, thereby serving as a reliable instrument to forecast the performance under yet larger computational budgets.
4.4 Formula and Monotonicity
As formulated in Eq. (3) and derived via our fitting method, Farseer possesses intrinsic mathematical properties that align with the theoretical expectations for loss functions in machine learning. This section details the fitted formula and its fundamental property of monotonicity.
Formula. The specific fitted law for Farseer is given by:
L(N, D) -.002--00 169 000091
88.01 1..01---.1111 66287 D
0. 0.444
(15)
8

---

Monotonicity. A fundamental expectation is that model performance should improve (i.e., loss should decrease) with more resources. Farseer inherently satisfies this: L(N, D) monotonically decreasing with respect to both increasing N and D. This behavior is a direct consequence of its functional form and the parameter constraints imposed by our fitting procedure. A detailed analysis of the partial derivatives confirming this monotonicity is provided in Appendix I.
5 Comparative Analysis of Farseer and Chinchilla
We compare our proposed scaling law Farseer against the widely recognized Chinchilla [20] in Eq. (2). This comparison highlights the advantages of our approach terms of predictive accuracy, extrapolation, and guidance optimal resource allocation.
5.1 Formula Comparison: Predictive Power
We compared the predictive power of Chinchilla and Farseer using our comprehensive dataset (Appendix B). Standard non-linear regression was used for fitting both, and Chinchilla was also evaluated with our multi-round iterative method (Appendix H). As shown Fig. (a) and Fig. 2. Farseer's s predicted BPC aligns remarkably closely with empirical values across various N and D. Conversely, Chinchilla systematically deviates, underestimating at low Dand overestimating at high D. This superior fit underscores Farseer's higher expressive capacity, enabled by its A(N) and ((N) that capture decay rates specific to each model size, unlike Chinchilla's ' single average trend.
5.2 Property Comparison: Robustness and Extrapolation
To assess the robustness and extrapolation, examine the generalization beyond the training domain in two scenarios (see Fig. 8), plotting relative error as the upper bound on model size used for fitting increases. As shown in Fig. 8 (a), with model size fixed at 6.4 B parameters, 1.9 x 109 to 5.4 x 109 and observe that Farseer formula yields both lower average relative error and smaller variance than the Chinchilla fit, indicating enhanced robustness changes in the fitting range. Fig. 8 (b) considers a fixed model size of 25.1 B, which is far beyond the fitting range. As N increases, Farseer's relative error shows clear downward trend and reaches a low level, while Chinchilla's error remains persistently high. These results collectively demonstrate that Farseer only offers greater robustness within the fitting range but also generalizes more reliably when extrapolating to larger model scales.
vary N from
0.025
Farseer
Relative Erro
0.020
Chinchilla
0.015
0.010
0.000
1.9
2.7
3.8
5/4
1.9
2.3
2.7
3.2
4.5 5.4
N Upper Bound for Fitting
1e9
(a) Robustness fixed 6.4B
(b) Extrapolation
25.1B
Figure 8: Furseer vs Chinchilla robustness and extrapolation. The x-axis "Largest model size used in fitting". (a) For 6.4 B models, Farseer shows lower and steadier errors as N upper bound increases. (b) For 25.1 B model, far beyond the fitting range, Farseer achieves a clear error reduction as N increases, while Chinchilla stays large.
5.3 Application Comparison: Optimal Computing Resource Allocation
Efficient training requires balancing model size and dataset size D, typically reflected in the optimal D/N ratio. we compare the practical guidance offered by Farseer and Chinchilla on optimal compute allocation during training, where the total budget is taken as = G [22, 20). Fig. (b) presents the optimal D/N ratio predicted by both Farseer and Chinchilla as a instrument of the total compute budget C. Chinchilla, suggesting an optimal D D/N ratio around 20 (typically 10-30), primarily derived from and applicable to training at moderate compute budgets corresponding to smaller model

---

sizes. This constant ratio provides inaccurate guidance for larger scale. In contrast, Farseer predicts steadily increasing optimal D/N ratio as the compute budget (and consequently, the optimal model size) grows. This predicted trend aligns remarkably well with the actual training configurations adopted for recent oate-of-the-art large language models (e.g., Qwen [46, 45], Llama [39, 40, 16)), indicating that Farseer offers more accurate and relevant guidance for optimizing resource allocation in current and future large-scale model training.
6 Related Work
Training LLMs necessitates optimizing compute allocation via scaling laws. The discovery and application of these laws have provided the confidence invest massive compute resources, knowing they can be predictably converted into model capability [22]. OpenAl's foundational work established power-law relationship between cross-entropy loss and model/data scale (Eq. 1), advocating for training large models on moderate data with early stopping. However, this assumes oversimplified dynamics and fixed hyperparameters, limiting extrapolation reliability.
DeepMind's Chinchilla [20] refined this by proposing proportional scaling of model and data size based on an updated methodology (Eq. 2), empirically demonstrating improved efficiency. Under the same compute budget as Gopher [32], Chinchilla adopts a smaller model paired with more training data, yielding substantial gains on downstream tasks. Nevertheless, recent work [4] has raised concerns about the reproducibility of Chinchilla's s findings and demonstrated that optimal token-to-parn to--t-r--aamterr ratios sensitive to data quality, suggesting current scaling laws are valuable heuristics but require calibration and lack universal applicability.
The landscape of scaling law research is rapidly evolving beyond classical formulations, exploring multifaceted avenues for enhanced efficiency and generalization. Recent empirical work includes deriving observational scaling laws from existing models to unify performance patterns and explain emergent phenomena [33, 3, 18, 42], while other studies apply scaling principles to improve data curation, e.g., by strategically filtering datasets [27, 26, 35, 30, 48, 13]. Theoretical inquiries concurrently advancing our comprehension of core mechanisms: some demonstrate how sophisti- cated feature learning can potentially double scaling rates for intricate functions [5], while others establish direct connections between generalization error exponents and data manifold dimensional- ity [18]. Furthermore, the purview of scaling laws now extends significantly beyond pre- -re-training, with investigations into their implications for inference dynamics [43, 44, 24, 43, 34], parameter and communication efficiency [1, 8], and downstream task [21, 9, 44, 19, 36, 10, 12, 37, 38. 28, 23].
7 Limitations
This study has several limitations. First, our empirical validation is primarily based Llama-style decoder-only Transformers. While the core methodology may be generalizable, its applicability to other architectures, such Mixture-of offuure-of-Epppeee of-Expetts (MoE), requires further investigation. Second, our largest validated model has 25. IB parameters. Extrapolating to trillion-parameterr models remains an open question, constrained by computational resources and the engineering complexities of large- scale parallel training. Third, in line with other work in this area, our scaling law is empirically derived and lacks first-rricipples theoretical justification. We have open-sourced our data to encourage community efforts on this front. Finally, this work focuses on pre-training loss (BPC); extending scaling laws to predict downstream task performance is a complex but important direction for future work. A more detailed discussion of limitations is available in Appendix J.
8 Conclusion
We propose Farseer, refined scaling law and methodology for Large Language Models. By accurately modeling the loss surface L(N, D) through a novel fitting approach, Farseer provides significantly better empirical fit and superior extrapolation capabilities compared to prior scaling laws like Chinchilla's. This work enables reliable prediction of large-scale model performance from tractable small-scale experiments, bridging the critical scaling gap and facilitating more efficient evaluation of training strategies and compute allocation. Validated on large corpus trained models, Farseer offers valuable insights for LLM development.
10

---

Acknowledgments
This research supported in part by the National Key Research and Development Program of China under Grant 20232D0121300, and in part by the Natinonal Natural Scinece Foundation of China(Grant No. 62495092).
References
[1] Samira Abnar, Harshay Shah, Dan Busbridge, Alaacldin Mohamed Elnouby Ali, Josh Susskind, and Vimal Thilak. Parameters vs flops: Scaling laws for optimal sparsity for mixture-of-experts language models. arXiv preprint arXiv:2501. 12370, 2025.
[2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[3] Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Proceedings of the National Academy of Sciences, 121(27)x2311877111 2024.
[4] Tamay Besiroglu, Ege Erdil, Matthew Barnett, and Josh You. Chinchilla scaling: A replication attempt. arXiv preprint arXiv:2404. 10102, 2024.
[5] Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. How feature learning can improve neural scaling laws. arXiv preprint arXiv:2409. 17858, 2024.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:18 33:1877-1901,, 2020.
[7] Weilin Cai, Juyong Jiang, Fan Wang. Jing Tang, Sunghun Kim, and Jiayi Huang. A survey on mixture of experts. arXiv preprint arXiv:2407.0622 2024.
[8] Zachary Charles, Gabriel Teston, Lucio Dery, Keith Rush, Nova Fallen, Zachary Garrett, Arthur Szlam, and Arthur Douillard. Communication-cffie effficeent language model training scales reliably and robustly: Scaling laws for diloco. arXiv preprint arXiv:2503.09799 2025.
[9] Yangyi Chen, Binxuan Huang, Yifan Gao, Zhengyang Wang. Jingfeng Yang, and Heng Ji. Scaling laws for predicting downstream performance in llms. arXiv preprint arXiv:2410.08527. 2024.
[10] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language Linguage-image learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1818-2829, 2023.
[11] Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171-4188 2019.
[12] Fernando Diaz and Michael Madaio. Scaling laws do not scale. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, volume 7, pages 341 341-357, 2024.
[13] Yan Fang, Jingtao Zhan, Qingyao Ai, Jiaxin Mao, Weihang Su, Jia Chen, and Yiqun Liu. Scaling laws for dense retrieval. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1339-1349, 2024.
[14] Philip Gage. A new algorithm for data compression. The C Users Journal archive, 12:23-38. 1994.
[15] Carl Friedrich Gauss. Theoria motus corporum coelestium sectionibus conicis solem ambien- tium. Frid. Perthes et I. H. Besser, Hamburg, 1809,

---

[16] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and Amy Yang et al. The llama 3 herd of models. 2024.
[17] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00755 2023.
[18] Alexander Havrilla and Wenjing Liao. Understanding scaling laws with statistical and approxi- mation theory for transformer neural networks on intrinsically low-dimensional data. Advances in Neural Information Processing Systems, 01421622-22110 2024.
[19] Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint arXiv:2 102.0 2021.
[20] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training Compute-Optimal Large Language Models, March 2022. arXiv:2203. [es].
[21] Berivan Isik, Natalia Ponomareva, Hussein Hazimeh, Dimitris Paparas, Sergei Vassilvitskii, and Sanmi Koyejo. Scaling laws for downstream task performance of large language models. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2024.
[22] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint vrXv::2001.08611 2020.
[23] Tanishq Kumar, Zachary Ankner, Benjamin F Spector, Blake Bordelon, Niklas Muennighoff, Mansheej Paul, Cengiz Pehlevan, Christopher Ré, and Aditi Raghunathan. Scaling laws for precision. arXiv preprint 2..VV3331 2024.
[24] Noam Levi. A simple model of inference scaling laws. arXiv preprinl arXiv:2410. 16377, 2024.
[25] Houyi Li, Wenzhen Zheng, Jingeheng Hu, Qiufeng Wang, Hanshan Zhang, Zili Wang, Shi- jie Xuyang, Yuantao Fan, Shuigeng Zhou, Xiangyu Zhang, et al. Predictable scale: Part i-optimal hyperparameter scaling law in large language model pretraining. arXiv preprint 2xXiv:5503.04755 2025.
(26) Margaret Li, Sneha Kudugunta, and Luke Zettlemoyer. (mis) fitting scaling laws: A survey of scaling law fitting techniques in deep learning. In The Thirteenth International Conference on Learning Representations, 2024.
[27] Ruihang Li, Yixuan Wei, Miaosen Zhang, Nenghai Yu, Han Hu, and Houwen Peng. Scal- ingfilter: Assessing data quality through inverse utilization of scaling laws. arXiv preprins arXiv:2408.085 083 10, 2024.
[28] Chuan Liu, Huanran Chen, Yichi Zhang, Yinpeng Dong, and Jun Zhu. Scaling laws for black box adversarial attacks. arXiv preprint arXiv:2411. 2024.
[29] llya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[30] Eric Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling. Advances in Neural Information Processing Systems, 36:28699-28722, 2023.
[31] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018
[32] Jack Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:21 1446, 2021.
12

---

[33] Yangjun Ruan, Chris J Maddison, and Tatsunori Hashimoto. Observational scaling laws and the predictability of language model performance. arXiv preprint arXiv:2405.1093. 2024.
[34] Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. Beyond chinchilla-optimal: accounting for inference in language model scaling laws. In Proceedings of the 41 Interna- tional Conference on Machine Learning, ICML'24. 24. JoLR.org, 2024.
[35] Utkarsh Sharma and Jared Kaplan. Scaling laws from the data manifold dimension. Journal of Machine Learning Research, 23(9):1-34, 2022.
[36] Kyuyong Shin, Hanock Kwak, Su Young Kim, Max Nihlén Ramström, Jisu Jeong, Jung-Wor Woo Ha, and Kyung-Min Kim. Scaling law for recommendation models: Towards general-purpont user representations. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 4596 2023.
[37] Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 15:19993-9 2022.
[38] Hui Su, Zhi Tian, Xiaoyu Shen, and Xunliang Cai. Unraveling the mystery of scaling laws: Part
i. arXiv preprint arXiv:2403.066 066663,..6333. 2024.
[39] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. 2 2023.
[40] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, and Shruti Bhosale et al. Llama 2: Open foundation and fine-tuned chat models. 7 2023.
[41] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[42] Peihao Wang, Rameswar Panda, and Zhangyang Wang. Data efficient neural scaling law via model reusing. In International Conference on Machine Learning, pages 36193-36204. PMLR, 2023.
[43] Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for IIm problem-solving. In The Thirteenth International Conference Learning Representations, 2025.
[44] Chengyin Xu, Kaiyuan Chen, Xiao Li, Ke Shen, and Chenggang Li. Unveiling downstream performance scaling of llms: A clustering- perspective. arXiv preprint arXiv:2502.17262, 2025.
[45] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang. Chenxu Lv. Chujie Zheng. Dayiheng Liu, Fan Zhou, Fei Huang. Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghho Deng, Mei I.i, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang. Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. arXiv preprint arXiv:2505.093550
2025.
[46] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Qwen Team, and Alibaba Group et al. Qwen2 technical report. 2024.
[47] Greg Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. 2022.
13

---

[48] Wenxuan Yang. Qingqu Wei, Chenxi Ma. Weimin Tan, and Bo Yan. Scaling laws for data-
efficient visual transfer learning. arXiv preprint arXiv:2504.33219 2025.
14

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's s contributions and scope?
Question:
Answer: [Yes]
Justification: Please see Section 1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
Please see Section 2.1.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification: Please see Abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/ for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips. ptpps.cc/puut cc/publ
srriiisssooooooioo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please see Section 2.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Please see Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
17

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please Section 1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http:///eeriicccoooeo http::/pp.pppp..ttttttttstueeeee
Answer: [Yes]
Justification: Justification: Please see Section
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
18

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Please see Abstract.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
19

---

Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
20

---

Justification:
Editing (e.g., grammar, spelling, word choice)
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer our LLM policy what should or should not be described.
for
21

---

Appendix
Content
A General Loss Function Metric Formulas
23
A.1 Standardizing Hyperparameter Settings
24
Defining an Architecturally Standardized I.I.M Family
................................
B Distribution Training Dataset
25
C Ablation Study of Fitting Methods
26
C.I Algorithm Workflow: Differential Piecewise Fitting
26
C.2 Core Insight: Power-Law Scaling with Data Size (D)
27
C.3 Functional Form Selection for A(N) and B(N)
30
Functional Form Selection Model-Dependent Residual L
30
D Validation Set and Compression Rate Metrics
34
D.1 Validation Set
34
D.2 Compression Rate Metrics
34
E Point, Line, and Surface Comparison
34
Model Parameter Configuration Table
G Consideration of Parameters with Embedding Layers
37
H Non-lincar End-to-End Fitting
38
Analysis of Formula Properties
39
Discussion and Future Work
41
22

---

A General Loss Function Metric Formulas
To systematically evaluate and predict the scalability of various LLM training approaches, introduce a general loss formulation expressed as loss surface L(N,, D). This surface captures the model's s performance metrics (BPC), function of model size (N) and training data size (D). The loss surface L(N, D) is implicitly conditioned on fixed underlying training strategy, serving as foundational tool for scalability analysis.
To formalize our analysis, we define specific Large Language Model (LLM) instance M & NA. Here, & denotes a LLM family, characterizing models that share common architectural features or design principles, such as Transformer decoder-only architectures with fixed aspect ratios. Given parameter count N, a deterministic LLM instance M. is realized from the family A.
The performance of an LLM instance, determined through specific training regimen and measured by metries, depends on intrinsic model properties (e.g., N) and various training- related factors. We formalize this aggregate performance metric L as function:
L(A. D. N, LR, 8S
(16)
where:
A: The LLM family.
ID:: The dataset or data distribution used for training.
N: The parameter count of the model instance.
D: A measure of the training data scale (e.g.. number of tokens).
LR, BS: The learning rate and batch size used during training.
8: A term encapsulating other influential hyperparameters training specifies, such as optimizer type, regularization strategies, or training duration.
Furthermore, in studying the Scaling Law, we assume that other hyperparameters (e.g., learning rate, batch size) are set to appropriate values that prevent the model training from deviating excessively or performing too poorly, or they follow scale-aware heuristics [47, 25J. Under this assumption, performance L becomes primarily a function N and D, and our central objective is to precisely characterize the loss surface L(N, D) for given model family A and data recipe D.
As established in 2.1, our primary objective is to accurately characterize and predict the loss surface L(N, D), which describes how performance scales with model size (N) and data size (D) for specific model family A and data recipe DD However, the overall performance metric L,, as indicated by our general formulation (Eq. (16)), is also sensitive to factors such as learning rate (Ir), batch size (bs), and specific architectural choices within the family A.
To ensure a stable and deterministic relationship between the primary variables (N, D) and the performance L. thereby enhancing the robustness and accuracy of subsequent scaling law fitting, is crucial to control for or standardize these additional sources of variation. This typically involves preliminary experiments and adherence to fixed protocols.
In this study, we utilize a fixed data recipe ID (details on its construction and characteristics are deferred to table 1). Consequently, the main efforts in controlling variability focus standardizing the following two categories of factors:
1. Optimization Hyperparameters: Primarily the learning rate (Ir) and batch size (bs), which are set to suitable values (potentially following scale-aware schedules or determined through preliminary sweeps) to ensure model training does not deviate significantly or exhibit markedly poor performance.
2. Architectural Hyperparameters:
The specific configuration defining the LLM family A (e.g.. layer counts, hidden dimensions relative to N, activation functions) must be consis- tently defined scaled according precise rules.
By carefully controlling these factors, we ensure that for a given experimental setup defined by to well-defined expected performance value T.. This satisfies fundamental prerequisite for reliably fitting the loss surface L D) and accurately assessing scaling behavior.
and D and the chosen hyperparameter protocols, each (N, D) pair maps
23

---

A.1 Standardizing Hyperparameter Settings
We first address the influence of training hyperparameters, primarily learning rate (Ir) and batch size (bs), on performance L. Prior work [25] and our preliminary investigations confirm their significant impact. Crucially, for given model scale (N) and data size (D); while one might pursue strictly optimal hyperparameters (pr N, D). bsppp( N, D)), our actual requirement is for appropriate hyperparameters. These are settings that ensure the model training does not deviate excessively or perform markedly poorly, rather than needing to be strictly optimal. Furthermore, the performance landscape L is often relatively flat near regions of good hyperparameter choices.
This flatness implies that using such appropriate hyperparameters yields stable and sufficiently good results, effectively mitigating performance fluctuations due to minor hyperparameter variations, without the need to pinpoint strict optima. Since the goal of scaling law research is to understand model potential under competent, rather than necessarily strictly optimized, training conditions, we operate by selecting hyperparameters pprinarily determined by N and D) for a fixed model family A and data recipe IT2 Ih>-tatt ensure they are adequate for effective training and do not cause the model to perform substantially worse than it otherwise could. This allows us to simplify the loss function, treating L primarily as a function of and D by implicitly using these chosen, appropriate hyperparameters (denoted Tapee, apppp)):
LN,((N, D,Ir, bs) Re La,n(N, D, I, Lappr, bsspp)) = LA, D)
In practice, we leverage methods like the Step Law [25] to help select suitable ir and bs values for our experiments, applying these standardized settings consistently across different (N, D) points to ensure model performance is not significantly hindered by hyperparameter selection.
A.2 Defining an Architecturally Standardized LLM Family
The concept of an LLM family provides a robust framework for systematic analysis and comparison. In a broad sense, a family can encompass models sharing a common set of training methods, data processing pipelines, or fundamental architectural paradigms. The core idea is that each distinct family is hypothesized to follow its own unique scaling law. By first defining family and then empirically determining its scaling behavior, we can create a principled basis for comparing the efficiency and potential of different approaches-forr instance, to see which family scales more effectively with increased compute or data.
We define an LLM family such that, given a parameter budget N, one can uniquely determine every architectural detail of the model: including, for example, the aspect ratio, the FFN ratio, the type of attention mechanism (multi-head attention or group-query attention), whether the network is pre-norm or post-norm, the activation function used in the FFN, the head dimension/number of heads, and whether a mixture-of-exxpre (MoE) architecture is employed.
With this definition, models of different sizes within the same LLM family share the same scaling- law characteristics. Consequently, once an LLM family A and a data recipe ID are fixed, they uniquely determine loss surface (N, N, D). Under the same data distribution, two different L.M families yield two distinct surfaces- LA, n (N, D) and LA,, whihhich allows us to compare how architectural choices affect scaling behaviour. Similarly, for fixed LLM family trained on different data distributions, the respective surfaces LA.D, (N, D) and LA.D, (N, D) reveal how data characteristics influence the scaling law.
Example. Even within a fixed architecture for instance, a LLaMA-style dense LLM (pre-norm + MHA) variatinss in model shape can still impact performance. We take two critical structural ratios as examples that govern model shape and are known to influence performance and computational efficiency:
* Aspect Ratio: The ratio of model width to depth (dmodal Number of Layers).
FFN Ratio: The ratio of the feed-forw nerdwrd network's 's intermediate size to the hidden dimen- sion (dmn/ /dmode)).
We conducted controlled experiments to study the independent effects of these ratios on perfor- mance T., holding either total parameters or estimated computational cost approximately constant. As shown in Fig. 9, our results consistently indicate that performance exhibits relatively flat optimum
24

---

with respect to both aspect ratio and FFN ratio across different experimental constraints. Motivated by the patterns observed Fig. 9, all experiments this paper therefore fix both the aspect ratio and the FFN ratio. The concrete values we adopt lie in the sub-optimal plateau region identified in the figure. This example illustrates that when using FARSEER define custom L.M family A, one must not only keep the architectural blueprint unchanged, but also hold model-shape-rellle relaped hyperparameters fixed. These configuration variables need not be globally optimal, but they must remain constant as N varies.
(a) Validation BPC vs. FFN Ratio
(b) Validation BPC vs. Aspect Ralio
sibliinb Min BPC:
0.590
46 1N 5341
0.553
in 5% 1795
+55, 0.5320
-1% 0,5329
Valiletbo RPC:
0.34
Validesbon BPC
0.54
0.540
0.540
0.55
0:511
0.590
0.53
1000
2006
3000
4300
5000
100
200
360
400
502
FEN Ratio (%)
Aupuzt Ratio
Figure 9: Validation BPC (see appendix D) under a fixed parameter versus (a) FFN ratio (interme- diate size hidden dimension) and (b) aspect ratio (hidden dimension / layer count). Blue circles: no embeddings; orange squares: with embeddings. Dashed lines mark min BPC (green), +0.5% (orange) and +1% (purple). Both curves exhibit broad, flat optimum.
B Distribution of the Training Dataset
This appendix details the composition of the data recipes used for training our models. Fach recipe represents a specific mix of datasets, weighted to target distinct capabilities. All prior experiments utilized the Baseline data recipe. Furthermore, evaluate Farseer under dramatically different bilingual data mix, experiments were also conducted using data from models trained with the Anglish-Chinese (EN-ZH) recipe. The table below specifies the percentage weights of the constituent datasets for these two primary recipes.
Table 1: Distribution of dataset weights (in percent) across various training strategies. Each strategy targets distinct capability: baseline performance, enhanced coding and mathematical reasoning, or English- Chinese bilingual proficiency.
Dataset
Baseline
EN-ZH
weh-data-cn
79.53
44,99
weh-data-cn
34.52
code-the-stack
4.62
4.63
web-data-nan math
book-non-novel-en
4.35
4.35
paper
3.38
3,38
wikipedia-mit
3.24
3.25
stackexchange
2.21
2.22
wikipodia-en
1.69
1.69
book-novel-en
0.83
0.83
wikipedia-cn
0.13
0.13
25

---

C Ablation Study of Fitting Methods
C.1 Algorithm Workflow: Differential Piecewise Fitting
The Differential Piecewise Fitting procedure, detailed in Algorithm 2, systematically models data L(N, Many steps in this procedure involve fitting functional forms, for example, to model how parameters Av and BN depend on N. Algorithm 1 describes a general method, Optimal Transformation Selection, for choosing appropriate data transformations (e.g.. logarithmic, power- law) from dictionary G to linearize relationships and subsequently fit parameters. The overall Differential Piecewise Fitting procedure, as implemented in Algorithm using specific stretched- exponential forms, comprises three main stages:
1. Stage 1: Initial Estimation of AN and Bx: For each model size N, the parameters An and By of the data-dependent term B(N)D-A(N) are estimated by analyzing the finite difference (DL(N, D), following the specific regression steps detailed in Algorithm
2. Stage 2: Parameterization and Iterative Refinement of fA Continuous functions fA (N; 0,)) and [NN are derived by fitting the discrete {Ax and (Bx) } estimates. While Algorithm 2 specifies particular functional forms (stretched- exponentials), the underlying task selecting optimal transformations and fitting parameters is generally addressed by the methodology in Algorithm 1. This stage also includes iterative refinement of power-law exponents (e.g., Ot, 3) as specified in Algorithm 2.
and fB(N;; (B);
3. Stage 3: Fitting the Model-Depender Residual E Lx (N): The residual term (NV is estimated (typically by averaging L(NN, D) - B(N; ( D A(N:0)) over to get G(N)) and modeled as fLx (N; (L Similar Stage 2, Algorithm 2 employs a specific secched-xxpoonntial functional form for fLx (N), and the transformation selection and fitting process determine its parameters can be understood in the general context of the methods presented in Algorithm 1.
Algorithm 1: Optimal Transformation Selection for Y = f(X)
Input: Discrete data points (Xx, Y&) for k = ,,...,,,..
Input: Dictionary of candidate transformation functions G 92)........
// Each G is function, e.g., identity, logari thm.
Output: Optimal transformations g E G.
Output: Coefficients (a" b²) for the linear model g (Yr) R (Xx) br
Output: Minimum residual sum of squares lmin-
Initialize miin < 0O
Initialize g <- null, 9 +- null, a" +- null, d" +- null, pd +- null
forall gy E do
forall gx G do
Define gy Define X = (Xx).
Perform linear regression: Y = acand X% beand
Calculate current residual sum of squares leurent + (X% - (AcandX) + beand))2
if
Ceurrent Emin then Emin
Leurent
(gy
g (gy, 9x
10
6" f- (Aeand: Deand)
return 9x (a", b,), min
26

---

Algorithm 2: Differential Piecewise Fitting (with Stretched-Exponential Forms)
Input: Loss Data points L(N, D), scale factor X. Output: Parameters 0 = (ai: ai,a") ") 0B = (azz 12, ), 6LN (ag. Dig 7")
Output: Final fit L(N, D) x exp(a + D explai N" +b1) exp(a32V7 + +b)).
// Stage 1: Initial estimation of discrete Av, Bx
foreach model size N do
Compute Ry (D) L(N, - ()N,
// From text: (D) s Dx
Estimate AN, BN via linear fit on
log( Rv (D)) log Bxx) AN log(D)
// Linear fit parame ters can be found using the Normal Equation.
BN + Bx/(1 A~)
6 Collect discrete sets {AN} } and {Bv]
// Stage 2: Parameterization and Iterative Refinement of (a( and fN(N;
// Assumed forms: fa(N; = exp(a1 N bi), ==) exp(a2 b2).
7 Fit (og(B)) N2 by to {(N, Bx)) to find initial 02 b2, B
// Similarly, for each 3, (az, b2) are found via linear regression (e.g., Normal Equation) minimizing Ex (log( Bx) (a2N2 + b2))
// Iterative refinement of exponents 0,
Let Rx OD; = fa(N;; (B) A=
Let global residual error ε = EN (D(NN(D) Rx(D:,A.0B))
10 repeat
11
Fix (3,02 22,02,22 Update a (and re-estimate 01, b1) to minimize ε // This involves finding (e.g., via grid search). For each candidate a, bi) are re-calculated by fitting log(AN) No + bi. The set (1)) that
minimizes the global residual εR is chosen.
12
Fix updated a, a1, by. Update (and re-estimate a2, b2) to minimize &R // Similarly, this involves finding B. For each candidate B, (az, by) are re-calculated by fitting (og(Bx)= a2 b2. The set (B, b2) that minimizes the global residual ER is chosen.
until convergence (e.g., 1-2 iterations small change in Cn) 14 Obtain refined parameters 0 = (ai, b","" and 8B = (až, 05, from the best fit
// Stage 3: Fit model-dependent residual E + Lx(N)
// Assumed form: fax (N; (,, e bb).
Compute (N,, D) L(N, D) fo -- (N(30)))
16 G(N) +- Avg D)]
Fit log(G(NN) = N² by to {(N, G(N))) to find 03, b3.7
// This involves finding (e.g., grid search); for each y, (as, by) are found via linear regression (e.g., Normal Equation) minimizing (lg((G(N)) (asN7 + ba))
18 Obtain final parameters 07,- (ag, 7")
// Final fitted Scaling Law
19 The final scaling law
D) exp(a;N9 + bb)D "op(a)""'+87) expla + bz)
return 84 02N
C.2
Core Insight: Power-Law Scaling with Data Size (D)
The entire methodology hinges on a key empirical observation: the finite difference of the loss with respect data size, An,(N, D) = XD) - L(N, D), exhibits a robust power-law relationship with D when plotted on log-log axes. This is illustrated in Figures 12. 13, and 14, which shows
27

---

striking linear trend for ADL(N, D) vs. D across various model sizes N. The left panel shows the power-law fit on log-log scale, while the right panel shows the relative residuals of the fit, which are small and randomly distributed, indicating good fit. This observation is consistent across all studied ranges of model size N, achieving high average R2 value of 0.9807.
This observation, ADL(N, D) D A(N), constitutes the central empirical insight derived directly from the data analysis concerning the functional form of the data-dependent loss components L (D)- D (N D). This insight allows us to approximate (D()) + (N D) = B A(N)) where A(N) and (NN) capture model-size dependencies. The cancellation of E and LN (NV terms in ADL(N, D) simplifies the analysis, focusing it on data-dependent effects. This power-law relationship is pivotal, all subsequent fitting stages for A(N), B(N), and consequently Ly (N), build upon this initial characterization.
To establish the most reliable functional forms for the components of the loss L(N, D), we analyzed various differential perspectives. The general form of the loss is L(N, D) - Ln (D) (NN(N, E + LN (N). We consider two primary types of finite differences:
ADL(N, D) L(N,)) D) A(N,
(LD(D) LD(AD)] + [LND(N, D)
LND(N, XD)]
(N, D) = D(, - D) & [LN(N) - LN(AN)] + [LND(N, D) LND(N, XD)]
Each of these differences can be analyzed as function of N (with D fixed) or D (with N fixed), potentially revealing power-law relationships. This gives four primary perspectives to model the interaction term L (NN, D) D) and its associated main effects:
AnL(N, D) vs. D (for fixed N): Leads to fitting ( The terms (D(N) and (D(N) are coefficients and exponents that depend on N.
2. LD,,N D) vs. N (for fixed D): Leads to fitting Æ (D)N-Cx(P) Here, (D) and (D()) are coefficients and exponents that depend on D.
3. L(N,)) vs. D (for fixed N): Leads to fitting G (NNND--FDMM) Here, Gn(N) and (NNN) are coefficients and exponents that depend N.
4. ANL(NN D) vs. N (for fixed D): Leads to fitting IN (DD)N-NH(N) Here, (D( and (D) are coefficients and exponents that depend on D.
Figures 10 and 11 illustrate the behavior of the coefficient and exponent functions derived from these four perspectives.
-..0
Commoin 8on
-0.25
Confeeieet Roi
0.35
-2.40
10
10
10
12
10'
10tH
10"
(pacametees)
Jo/CCeffieent ded
Col Coefficion Bi det
Coefficiont 510 dut &im
Coofficint del -RD) sctt0
Figure 10: Log-Log projections of the coefficient functions derived from four differential loss perspectives: (a) B(N) in ApL = B(N) A(N(N) vs. N: (b) E(D) in ApI. N vs. D: (c) G(N) in A = G(N) D- F(N) vs. N; (d) Í(D) in AyL = ((D) H(D)
N-H(D) vs. D.
Perspective (a), B(N) vs. N, exhibits the most consistent and regular scaling, motivating its use in the primary analysis.
As indicated in Figure 10(a), the coefficient ((N) (derived from ADL(N, D) vs. D) displays ex- ceptionally tight and consistent scaling with N. Similarly, Figure 11 shows that the corresponding exponent A(N) varies smoothly and predictably over wide range of N. This regularity and stability are significantly more pronounced than those observed for analogous coefficient and exponent terms derived from the other three perspectives (Figures 10(b-d) and (b-d)..
The superior numerical behavior and clearer trends of A(N) and B(N) obtained from the ADL(N, D) vs. D analysis (corroborated by the high R²2 in Figure 3(a)) strongly justify prior-
28

---

Expereent CID)
LEpo HMD)
Exponer
13
Exprremnt
107 Stze [paremetens)
102
10at
126 Model Sizw (ownametens
1320
Date [tooem))
(toleens)
(al Expooent del #220 gatt
(bI Expooent CID) In dad N.
Expenent FING in del 6100
(10 Exponent MCDI in dul (yot
Figure 11: Log-Log projections of the exponent functions derived from four differential loss perspec- vs. D; (c) F(N) in Ay L G(N) D-F(M) vs. N: (d) (D) in AvL = I(D) N vs. D. Perspective (a), A(N) vs. N. exhibits the most stable trend, reinforeing the choice of modeling AnL
tives: (a) A(N) in ApL B(N) D-A(N) vs. N; (b) C(D) ApL = É(D)
E N-C(D)
function of D.
itizing this specific differential view. Thus, we adopt the form ADL(N, D) =e B(N)D A(N) as the basis for modeling the data data-dependent terms.
29

---

C.3 Functional Form Selection for A(N) and (NN)
Once the discrete sets of parameters { and {By} are obtained for each model size N (Stage 1), continuous functions (NN AA) and (N 0g) are determined (Stage 2). This se- lection guided by a systematic search over a dictionary of simple, monotone transformations g = (identity, logarithm, power functions).
For every ordered quadruple of transformations (gi, 9ÿ 9k. (0m) E G we perform coordinate projec- tions:
9.(AN) = ai (2 + b1 +EA,N
(17)
9k (Bx) 02 9m (N) + + + EBNN
(18)
The coefficients (a),bi and (a2, b2) are found via least squares for each combination. The transform quadruple (gi 97. yk Im) that minimizes the sum of projection-space residual sums of squares, l + fB = Ex EN + +Ex E2 is selected.
As stated in Section 3.2, empirical results indicated that applying a logarithmic transformation to AN and Bv. and a power-law transformation to N, yielded the best fit. That is, 97 (Av) = log(Av), 9j (N) No, (Bx) = log (v)), and Im, (NV N These choices lead the stretched-
exponential forms:
JA(N: = exp(a1 No + b1) (N exp(ag No + b2)
(19)
(20)
The exponents a and 3 are subsequently refined via an iterative strategy to minimize the global error of the loss differences L = Σn ED(Nx - R2 (D))².
wide range functional form combinations from G was evaluated. Table 2 presents a comparison of the pep-pefformin candidates, thereby illustrating the types of transformations considered and highlighting the combination that empirically minimized the l + LB error.
Table 2: Summary of the functional form selection strategy for A(N) and B(N) using the transforma- tion dictionary G. The combination (g: log, 9 power, 9b log, Im power) was empirically found to minimize & + LB, leading to stretched-exponeniie forms.
g (NV (An))
gx(Bx)
9m(N)
Functional Form Type
Relative LA + EB
Identity
Power
Identity
Power
Power-law + Power-law
0.1322
Log
Power
Log
Power
Stretched Exp + Stretched Exp
0.1125
Log
Power
Identity
Power
Stretched Exp + Power-law
0.1
Identity
Power
Log
Power
Power-law + Stretched Exp
0.
The quality of fit using these stretched exponential forms is demonstrated in Figure 4 in the main paper.
C.4 Functional Form Selection for Model-Dependent Residual E + Ln (N)
After fitting LD(D) + LND(N, D) R fn( DD the model-dependent residual term E + (x(N) is estimated. This is done by first computing O(N, D) L(N, D) fa(N: 0z) D-((N16)) and then averaging over D to get G(N) = A D) = F (N).
The functional form for fLy (N; ) (which models G(N), absorbing E as a constant within the fit) procedure as for A(N) and B(N).
is determined by repeating similar transformation-fi--sde
For each pair of transformations (gp- 9q) G², we regress:
(G()) as 9y ( (N) by EU,N
(21)
The pair (9p, 97) that minimizes the residual sum of squares Eu ΣN EZ.N is chosen.
As detailed in Section 3.3, the optimal transformations identified were a logarithmic transformation for G(N) (o (G(N) log(G(N)) and a power-law transformation for N (g (N) = N7). This selection results in stretched-exponential form for E + I (N):
E (N G(N) exp(a, +bs)
(22)
30

---

10"3
Rinua
-0.1
224100180U
2581851640
338048/260
32804572880
2841061A60
3399481350
Lart WiL.ettteeeeeee
3.0
10-7
Menun am
-0.1
-0.3
109
1011
1e11
0.15
476872704.0
476972704.0
207721000.0
07687270 (687270040
0.05
10-2
-0.05
-0.15
-0 25
1014
2x
568403568.0
568403968.0
6750383040
675898304.0
588403068 0
67 3750363040
Wirr LoIf
-005
-0.10
3x10
1.99 2011
Figure 12: Example of the law relationship between the finite difference of the loss, denoted here as Minus L diiff f. and the data size D for model sizes (N) from 201228288 to 676012032. (a) The left panel log-log plot shows a clear linear trend, indicative of power law. (b) The right panel relative residuals of the fit shown function of D, demonstrating the quality of the power-law approximation.
31

---

2563872000
O
798367206
61676 376
95626137660
2503072000
Whzr. LOF
10
6x
-0.5
4x300
-0.10
-0.15
1014
1014
2x
31258347520
L125834/520
13:81622800
1338163200.0
01258347520
1388
0.00
0.10
4.
1014
101
lell
0.075
3.000
1013746302.00
1607191206
3.025
3.000
10"1
-0.025
-0.075
1011
2x
0.05
2373829153.0
2657507872.00
2667807672.0
22181201520
10-1
-0.0
6x
-0.55
Figure 13: Example the pawer-law relationship between the finite difference of the loss, denoted here Minus L d_diff and the data size D for model sizes (N) from 798470400 to 2697992704, (a) The left panel log-log plot shows clear linear trend, indicative of a power law. (b) The right panel relative residuals of the fit shown function of D, demonstrating the quality of the power-law approximation.
32

---

0.075
N
3816121344.0
3816121344.D
3184230400.0
M-Minus L diff rolative residual
0816121344.0
L.
10-2
Nxx y
-0.025
-0.075
-0.100
1
10
1e11
0.08
4503859200.0
4503859200.0
5353758720
--- 5353758720.0
4503059200.0
N.+DD NU--D_-Aiinus__ (.r.y-kiinns... J. LiIf relattveee rreiiuulll
5353758720.0
0.04
L
0.00
10
-0.04
9 10
-0.06
10
10
10
0.6
D
18
0.04
17 10
6369259920.0
6369259520.0
6369259520.0
16
_dlffrellotivee relolivee resicual
15x 10-2
Minus dilt
x.4
0.00
N 1.x+D y.Dinu,-inus__ Lddiff
1.
12x
-0.02
10-1
-0.04
10
10
Figure 14: Example the law relationship between the finite difference of the loss, denoted here as Minus L dif f, and the data size D for model sizes (N) from 3184435200 to 6369572352. (a) The left panel log-log plot shows clear linear trend, indicative of a power law. (b) The right panel relative residuals of the fit shown function of D, demonstrating the quality of the power-law approximation.
33

---

The exponent is identified via grid search minimizing the fitting error.
Table 3 summarizes this selection process, illustrating the types of transformations explored and highlighting the empirically optimal choice. The "Minimized Error" column refers to the to criterion.
Table 3: Summary of the functional form selection strategy for G(N) ss E + (NV using the transformation dictionary G. The combination (gp log, 9y power) was empirically found to minimize ly. leading to a atretched-exponentee
form.
9p (G())) (NV)
Resulting Functional Form Type
Relative tu
Identity Log
Power
Power-law
0.0083
Power
Stretched Exponential
0.0011
D
Validation Set and Compression Rate Metrics
D.1
Validation Set
validation set must be based on the following principles: (1) Unseen integrity: All validation samples are rigorously excluded from the training data of any model. (2) Bias-free composition: Strict prohibition of model-generated or model-processed content to prevent overfitting towards specific model families. (3) Diversity: Ensuring coverage in various fields through extensive collection and random sampling protocols. (4) High quality: Validation samples must maintain strict syntactic correctness and logical integrity to avoid distortion of measurements. We construct a specialized English validation set comprising web pages, books, and academic papers, with each data category containing approximately 10 million tokens. The web data originated from SERP (Search Engine Results Page) pages crawled via Google searches targeting Wikipedia entities. Book data was digitized from newly published textbooks through our proprictary PDF parser pipeline, while academic papers are sourced from arXiv publications after 2024. This composite dataset is systematically organized into 55 distinct domains spanning culture, arts, history, entertainment, and other fields. For each data category, we implemented rigorous filtering pipeline: preliminary data pruning including removal of books/papers with character lengths <2,000 and webpages scoring below 0.6 via our defect detection model, domain-stratified downsampling, and GPT-4o-based quality scoring with subsequent selection of top-ranked samples. Through the multi-stage refinement process, we ultimately develop a high-qualis validation set containing 30 million tokens.
D.2 Compression Rate Metrics
We adopt bits per character (BPC) as the quantitative metric for evaluating LLM compression efficiency. BPC measures the cross-entropy between the model's predicted distribution and the true character distribution. Crucially, when the predicted distribution aligns perfectly with the true distribution, this cross-entropy formulation becomes mathematically equivalent to lossless compression. This equivalence establishes BPC as theoretically grounded metric for quantifying how effectively LLMs compress given text in our validation set. Formally, BPC is defined as:
N
1
1
N M
BPC = log
(- log(p:))
R1 R₂ = R
pi
M
N
(23)
1=1
1=1
train
where M, N denote corpus size and vocabulary size, respectively. R1, the first term of the BPC formula, captures model compression through next-token prediction loss, which is equivalent to the training loss upon first exposure. R2, the second term of the BPC formula, represents vocabulary compression rate. is worth noting that due to differences in vocabulary size, cross-family model comparisons need to consider R, Rg, while analysis within the same family can focus solely Rj-
E Point, Line, and Surface Comparison
One of the key applications of Farseer to enable systematic comparison across different set- tings sungs-such alternative model architectures or pre-training data ceaturres-using only modest
34

---

Point Comparison
Line Comparison
0.22
0.22
10080900 oegeing
-002
00
-004
0000
11 E0g Bogiish Tomens Dj
10
Moddel Persmettrr
10c
11.0
12
11.5
124
Figure 15: Farseer's normalized 3D surface of relative BPC difference between datasets with 85% and 50% English proportions. The translucent pink plane marks A = 0: above it, the 50%-English configuration outperforms; below the 85%-English configuration outperforms. Green squares show small-scale experiments at individual (N, D) points, and the yellow dashed curve connects several such coints-conclusioonn from these point/line comparisons do not hold larger scales.
computational resources. By fitting two Farseer curves on relatively small-scale experiments, we obtain predicted loss surfaces L.N,, D) that faithfully capture performance across the full (N, D) plane for each setting. This gives rise to a comprehensive surface comparison that subsumes and extends traditional point and line analyses: point comparison highlights performance individual (N, D) coordinates; a line comparison reveals trends along fixed N or D slices; and the surface comparison synthesizes these insights into continuous two-dimensional map of relative behavior.
To illustrate, we apply this to two pre- pre-training mixtures with 85% versus 50% English data (balanced by Chinese for the remainder). We sample modest grid of model sizes N and English token counts D. fit Farseer to each subset, and compute the normalized relative BPC difference
BPCs5% BPC50%
BPC50%
Fig. 15 shows the 3D surface of A over log10 N and log1o D, with the zero plane marking parity between the two mixtures. Regions above the plane indicate that the 50% English mixture yields lower eπтоr, while regions below favor the 85% mixture. Green squares denote individual point comparison experiments at specific (N, D) coordinates, and the yellow line connects several such points for a line comparison. Although these smaller scale analyses suggest mixture is better, the surface comparison reveals how those conclusions can reverse at larger scales. Farseer's ss exhibits power for low-cost, high-fidelity extrapolation across any training recipes model designs.
F
Model Parameter Configuration Table
Our model configurations are built upon the architectural principles of the LLaaa [39, 40), scaling depth, width, and attention mechanisms in systematic grid to explore performance across broad spectrum of model sizes. Table 4 provides an overview of the 21 distinct configurations (Model through Model 21) used to fit the Farseer with Baseline training data in Tab. B. Each model (from the most compact Model to the most expansive Model 21) was trained on an exceptionally large and diverse corpus, ensuring robust generalization and comprehensive coverage of the underlying physical phenomena:
Parameter scale:
The second column reports the number of trainable parameters outside of the embedding layers, ranging from 2.01 x 10 up to 6.37 x 10 This systematic scaling of model size enables us rigorously examine how parameter count N influences the behavior captured by Farseer for modeling scaling laws.
35

---

Table 4: Model configurations used in the Farseer study with Baseline data recipe, detailing architec- tural and training dataset properties across 21 systematically scaled transformer models.
Model
N
D couns
D range
dmodel
dg
N Nned
N Mayer
Nwish emb
201228288
17
1.41 x 10" 3.62 x 10
1024
2728
16
16
335446016
2
239005312
18
1.19 x 10 4.31
1041
1088
2856
17
17
381611644 1668
284207616
18
1.00 x 10.. 3.62
10t1"
1152
3032
18
18
435202560
336994944
18
1.19 10 109, 4.31
10"
1216
3240
19
496378496 19
397772800
1.41 10 3.62
10th*
1280
3472
20
565544960 20
6
476931840
18
1.19 x 10 4.31
10
1344
3584
21
22
653092608
7
568468736
18
1.00 X 10" 3.62
10t1
1472
3888
23
22
761406720
8
676012032
18
1.19 X 10 4.31
104F
1536
4064
24
24
877338624
798470400
18
1.41 X 10 5.12 10
1600
4264
25
26
1008 85600)
10
956354688
18
1.19 10 4.31 10tH
1728
4528
27
27
182844114
112593868
1.00 10 3.62 10TH
1792
4832
28
29
1,668 1971
12
1338278400
1.19 X 10 4.31 10TH
1920
30
30
1589936640
13
1608 122368
17
1.41 X 10 3.62 X 10tH
2048
5448
32
32
1876557824
14
1911894528
17
1.19 X 109, 3.04 10
2176
5712
34
34
2197 107200
15
2273495040
14
1.41 X 10 1.81 X 10tH
2304
6064
36
36
2575484928
16
2697992704
15
[.68 10 2.15 X 10
2432
6488
38
38
3016759808
3184435200)
2.00 10 1.28 10tH
2560
6952
40
40
3519979520
3816352512
2.38 10 1.52 10t
2752
42
4177062656
19
4504118400
2.00 x 109, 9.05
x 10
2880
7744
45
45
488 4881605760
20
5354047488
11
3.36 x 10 1.08 104
3072
8264
48
47
5756700672
21
6369572352
2.00 x 10 9.05
10
3328
9136
52
47
6805779968
Data coverage: Column Doount indicates the number of distinct data subsets incorporated into each model's training process. The corresponding column Drange (expressed in sci- entific notation) reports the minimum and maximum token counts used for training each model. Across all models, the training corpus spans roughly 1.00 10 to 5.12 x 10"l tokens, highlighting the substantial scale and heterogeneity of the data leveraged to capture Farseer's complex / scaling behavior.
Architectural details: Following the L.aama design [39, 40), each configuration specifies the transformer hidden dimension dmooell feed-forward sublayer size de, number of atten- tion heads Nhedd number of transformer layers Mayerr and total embedding parameters Nwith_ emb. This structured sweep follows common design principles established by existing models, helping ensure that Farseer's results remain compatible and comparable with prior scaling studies.
Data-rich training:
By combining extensive token budgets with carefully designed models, we ensure that even our largest architectures remain data-saturated, avoiding under-fitting in any scale regime. This abundance of training examples is critical for capturing the multi-scale interactions inherent the Farseer phenomena.
This systematic sweep model scales, grounded in widely adopted architectural designs and supported by substantial and diverse training data, provides consistent and robust foundation for analyzing the scaling behavior captured by Farseer.
Additionally, Section 4.2 investigates the impact of different dataset compositions on Farseer. Under the EN-ZH configuration in Tab. B, conducted experiments using the model settings presented in Table to demonstrate the data generalization capabilities of Farseer, with settings analogous to those described above.
36

---

Table 5: Model configurations used in the Farseer study with EN-ZH data recipe, detailing architec- tural and training dataset properties across systematically scaled transformer models.
Model
N
D couns
range
dmodel
dg
N Nhed
N Mayer
Nwish_emb emb
201228288
13
[..00% x 10" 1.28 x 10
1024
2728
16
16
335446016
2
284207616
13
2.00 x 10 1.28 x 10
1152
3032
18
18
435202560
397772800
13
12.00 x 10 1.28 x 10
1280
3472
20
20
565544960
585641088
2.00 10 1.28
10"
1472
3888
22
23
778579072
778000000
12.00 10", 1.28
10
1600
4264
26
25
9877
1099958272
13
12.00 x 10", 1.28 x 10
1792
4832
29
28
1334839296
7
1608 122368
13
12.00 x 10", 1.28 x 10t1
2048
5448
32
32
1876557824
8
2273495040
13
2.00 x 10 1.28 x 104F
2304
6064
36
36
2575484928
3184445200
13
[2.00 x 10 1.28
10
2560
6952
40
40
3519979520
G Consideration of Parameters with Embedding Layers
persistent question in the formulation and analysis of scaling laws for language models revolves around the inclusion of embedding layer parameters when defining the total model size, denoted as This is not a trivial accounting detail, as the embedding layer can constitute a significant fraction of model's parameters, particularly for models with large vocabularies. Different perspectives on this issue have been adopted seminal works. For instance, the scaling laws proposed by OpenAl often focused on con-embedding parameters, whereas the Chinchilla scaling laws by DeepMind typically accounted for the total number of parameters, including embeddings. This divergence in methodology highlights underlying uncertainty regarding the effective contribution of embedding parameters to model capacity as characterized by scaling laws.
To investigate this ambiguity and determine a more empirically grounded approach, we conducted targeted ablation study. The study aimed to directly compare the predictive power of scaling laws fitted using two distinct definitions of N: one that includes the parameters of the embedding layer and another that excludes them. We fitted separate scaling law functions to our experimental data under these two conditions.
The findings, as illustrated Fig 16. In the vicinity of the data points used for fitting the scaling laws, both formulations - N with and without embedding parameters yield comparable predictions, exhibiting good local fits. However, a crucial distinetion emerges when extrapolating to model sizes significantly larger than those in the fitting set. We validated the fitted laws at 25.1 billion parameter scale (denoted as Neat = 25.15 B). At this larger validation point, the discrepancy between the two an N with embedding layers scaled law, the error in prediction increased substantially to
0.070
w/o Embedding (Farseer)
0.060
Relative Error
N with Embedding
0.040
0.030
0.020
0.010
0.000
1.9
2.3
2.7
3.2
3.8
4.5
5.4
1.9
2.3
2.7
3.2
3.8
4.5
5.4
N Upper Bound for Fitting
1e9
(a) Validation on Nyal 6.4B
(b) Validation Nvai = 25.1B
Figure 16: Scaling law extrapolation with and without counting embedding parameters in model size N. While both versions fit the training range well, only the formulation excluding embeddings extrapolates accurately to Neat 25.1B. This indicates that embedding parameters may not scale with model capacity the same way as transformer parameters, and excluding them yield more predictive scaling laws.
37

---

0.029. In contrast, the scaling law derived using N exclusive of embedding parameters demonstrated significantly better extrapolation, with an error more than four times smaller than its counterpart.
This ablation study provides compelling evidence that, from the perspective of achieving robust and generalizable scaling laws, it is preferable exclude embedding layer parameters from the calculation of N. The improved extrapolation accuracy observed when omitting these parameters suggests that they may not contribute the model's scalable capacity the same manner as the core "compute" parameters of the transformer architecture. This finding has important implications for accurately predicting the performance of much larger models and for optimally allocating parameter budgets in future language model development.
H Non-linear End-to-End Fitting
To ensure a robust and equitable comparison of the predictive capabilities of the Farseer and Chinchilla scaling laws, meticulously evaluated two fitting methodologies for both. This appendix details the procedures undertaken and the rationale behind the specific fitting choices reported in the main text.
Initially, our Differential Piecewise Fitting method (as detailed in Section 3 and Appendix C) was applied both the Farseer and Chinchilla formulations using our comprehensive dataset (Appendix B). This allowed for a direct comparison of the models when subjected to the same fitting paradigm.
Subsequently, also employed standard end-to-end non-linear regression for both models, mirroring the conventional approach used for Chinchilla. For the Farseer model, this process was particularly intensive due its more complex functional form. To thoroughly explore the parameter space and mitigate issues with local minima, we conducted an extensive search involving over 30,000 optimization runs, each initiated with different prior-informed parameter initialization.
0.0200
Standard Non Linear Regression Piecewise Fitting
Standers NennLinear
0.0167
Diferretial PRocowbee filking
0.015
0.0150
Relative error
0.0125
0.025
0.0100
0.020
0.0075
0.0056
0.015
0.0050
0.0028
0.100
0.0025
0.005
0.0000
0.000
Farseer
Chinchilla
Chinchila
Chinchila
Farswer
(a) Relative errors of Chinchilla and Farseer models fitted with standard non-linear regression and differ- ential piecewise fitting.
(b) Relative extrapolation error 6B 25B for Chin- chilla and Farseer, comparing standard end-to-end re- gression v.s. Differential Piecewise fitting.
Figure 17:
(a) Chinchilla performs slightly better with standard regression, while Farseer shows significant gain from piecewise fitting. (b) For the Farseer, differential piecewise fitting reduces the extrapolation error by an order of magnitude.
Our evaluations revealed distinct behaviors for the two models under these fitting regimes. For the Chinchilla model, the standard non-linear regression yielded marginally better fit compared to method, as illustrated by Fig 17.
In contrast, the Farseer model exhibited significantly different results. Due to its inherent complexity, the standard non-linear regression approach struggled converge to an optimal global solution, even with the extensive initialization strategy. The resulting fit produced errors approximately an order of magnitude (10 times) higher than those achieved with our method. The superior performance the Differential Piecewise Fitting method for Farseer suggests that its piecewise or iterative refinement better suited navigate intricate loss landscape of our formulation, indirectly validating the structural accuracy of the Farseer model itself.
38

---

Given these findings, and to ensure each model was represented by its most effective and accurately fitted version, the comparisons presented in the main body of this paper are based on:
The Chinchilla model fitted using standard non-linear regression. The Farseer model fitted using our Differential Piecewise Fitting method. This approach allows for fair comparison by leveraging the fitting technique that best captures the predictive power of each respective scaling law according to our empirical evaluations.
I Analysis of Formula Properties
As defined by the fitted form
L(N, D) 0.001-10000000 N0-19 -0.091 88.01-N 1.8.01-N-0-166220 -.:26..00
(24)
Fig. 18 exhibited the trend of BPC L(N, D) with model size N and data size D. Farseer exhibits a fundamental property: strictly decreases with both N and D. This guarantees that additional compute data never degrades performance. This feature makes Farseer both intuitively and formally well-behaved as large-scale scaling law.
Starting from the general form in Eq. (4) and other mentioned definitions, we group terms as
L(N, D) LN (N) + B(N) D D(N),
with
Lx(N)= = -..0211) 10 0.091
(NN) = 88.01.N 0.1 6.287 A(N) = -0.124-N 40.444
Each of these terms strictly positive for all admissible values of N and D.
1. Partial derivative w.r.t. D.
Differentiating with respect to D, obtain:
aD OD [N(NN) + B(N) D D-A(N) A(N) B(N) D- A(N)-1
This expression is strictly negative since all multiplicative components-A(NV) B(N), and D-A(N)-1 are positive for admissible values of N and D. Consequently, the loss L(N, D) monotonically decreases with increasing data size D, aligning with the intuitive expectation that more data improves generalization. This behavior is visualized in Fig. 19, which shows the surface of aL/OD over a wide range of (log N, log D).
2. Partial derivative w.r.t. N.
By the product and chain rules,
ØL ON (N(N++ AIN) + B(N)
aN (D-A(N)).
Because the analytical form of aT./N is prohibitively complex, we resort to numerical verification determine its sign. As illustrated in Fig. 20, the numerically computed 0T./ON remains negative across the entire range of N. Therefore, L is confirmed to decrease monotonically as N increases.
39

---

1.0
0.9
1.0
0.8
0.7
0.6
0.6
0.5
0.5
9
-0.
12
10
11
log
10
logs0
Figure 18: Visualization of L(N, D) as function of model size N and data size D.
1e-9
0.00
0.25
0.0
0.5
-0.75
1.0
aL/aD
-1.00
1.5
11.25
2.0
-1.50
1.75
13
10
11
11
log10(N)
10
log10(D)
13
14
8
15
Figure 19: Visualization of àL/0D as a function model size N and data size D, showing consistently negative gradient across the domain.
40

---

1c 10
00.5
0
-1.0
-1.5
2
-20
aL/aN
-2.5
3.0
4
3.5
13
-4.0
12
9
11
log10(N)
10
12
log10(D)
13
Figure 20: Visualization of aL/aN over varying N and D, also showing consistently negative gradient across the domain.
Discussion and Future Work
Through rigorous experimental design, massive number of experiments, and extensive experimen- tal data collected over 18 months experimentation and analysis, we have discovered law that is more fundamental than the Chinchilla law, which we name the Farseer. On one hand, we believe this paper opens up new field with a vast amount of potential work, such deeper explorations of the Law based on Farseer, or comprehensive comparisons of various model architectures and data methodologies using Farseer. On the other hand, Farseer has some limitations. We will now delve into a detailed discussion of these two aspects.
Computational power limitations. Due computational constraints, evaluations have been restricted to Llama-style, decoder-only Transformers. Different model architecture paradigm' distinct coupling and optimization dynamics may alter the scaling coefficients and predictive accuracy. We have already begun experiments on MoE-based LLMs. Furthermore, due to computational limitations, this work only evaluates the applicability on two pre-training data distributions (an -aglisn-dominant distribution and Chinese- English bilingual one). As this research began in early 2024, there was less focus on code data than there is now, hence our primary consideration of Chinese and English. However, we are currently investigating its applicability on aede/maat--omminaa data distributions, validating on code validation set. Please follow our subsequent work for updates. We are confident that the Farseer Scaling Law is applicable across different data distributions.
Larger-scale validation. This paper provides validation points up 25 B parameter LLM but does not include experiments on larger models, primarily due to computational power limitations. Other reasons include: (a) Farseer has maintained consistent prediction accuracy without any trend of increasing еттоr across various extrapolation points of different sizes (3B/6.5B/25P) Therefore, there is reason believe can be generalized larger models. (b) Larger models inherently have significant engineering implementation differences compared to smaller models. For instance, our models below 6.5B used only Data-Parallel distributed configurations, whereas for the 25B model training, we employed Pipeline Parallelism (Rank 8) and Virtual Pipeline Parallelism (Rank 5). Validating even larger models would require adopting further distributed training techniques, such as Tensor Parallelism. These distributed techniques themselves can introduce noise into the model's final performance. Therefore, larger validation points (e.g., 135B), it would be necessary to run multiple different engineering implementations reduce this noise, which poses further challenges
41

---

to computational resources. Balancing these complexities and benefits led to our decision to forgo larger-scale validations. (c) In numerous previous studies, 25 B dense model is already past the threshold where various intelligent emergent phenomena appear. We believe this size is sufficient to support the adequacy of our extrapolation validation.
BPC & downstream tasks. We chose to use Bits Per Character (BPC) instead of specific tasks for evaluation based on the following reasons: (a) Compression rates measured on a large and diverse validation set reflect model's general intelligence level, aligning with our goal of evaluating overall intelligence rather than task-specife performance. (b) As part of our experimental rigor, we meticulously constructed validation set characterized by four properties: Unseen Integrity, Bias-free Composition, Diversity, and High Quality. Compared individual benchmarks, it offers a more comprehensive assessment of intelligence and is guaranteed to be isolated from the training set. (c) Specific downstream tasks are highly susceptible to contamination and overfitting from training sets. In terms of magnitude, these benchmarks are more easily overfitted as evaluation targets. Consequently, pre-training datasets with varying degrees of contamination would exhibit different scaling laws, rendering such laws non-universal. (d) BPC possesses deep physical meaning related directly to compressibility, and numerous studies correlate model's compressibility strongly with downstream task performance. Although our carefully constructed validation set has numerous advantages over downstream task benchmarks, with the continuous advancement of post-training techniques, whether a base model with low BPC the validation set necessarily implies good model remains an open question. In summary, using BPC testing on a meticulously constructed validation set is the best method available to at the current level of public technology. Its limitations are primarily constrained by the state of contemporary technology.
Theoretical explanation. This work is a characterization of scaling laws based on massive observa- tional experimental data, and it does not provide a theoretical explanation. This is similar to all other work in the field of scaling laws, none of which have derived the specific formula of a scaling law from fundamental machine learning theory. This limitation, common across scaling law research, remains significant open challenge in machine learning theory.
Broader generalization. Our research primarily investigates decoder-only Transformers optimized via Next Token Prediction. Generalizing Farseer- formulations to broader machine learning unexplored in this work.
tasks-sure -uch as multimodal or diffusion-
Open source. We believe that our experimental data can be used for much further analysis, inter- pretation, discovery of new patterns. Therefore, we have decided open source our massive experimental data, worth millions of dollars, including 1000 LLM models trained from scratch and their entire training processes. We may even open source the training dataset and the meticulously constructed validation set. Based on this, We anticipate further research in at least the following directions: 1) Formula and fitting. We believe there may exist a more concise scaling law or a better fitting method. 2) More applications. While compute-optima guidance is valuable for model training. the entire L(N, D) surface obtained from Farseer certainly holds more valuable conclusions to be mined, such as the analysis of first-order partial derivatives, etc. 3) Training dynamics. This paper only evaluates the final performance of the models. The training dynamics of 1000 models of different model sizes and token sizes could be further studied to reveal underlying patterns.
This paper pioneers new experimental methodology for the era of large models. Consider two experimental settings: base A and experimental B. Within each group, all settings are kept identical, including but not limited to LLM Architecture Family, Data Distribution, and Training Strategy. Under settings A and B, two sets of small models, Models A and Models B. are trained. Using the method provided in this paper, two bivariate functions, LA (N,, D) and L (N, D), are fitted. These two functions share the same functional form (Farseer) but have different parameters. We can then compare the superiority of the two settings under infinite and arbitrary configurations of N and D. Under the previous methodology, if both Setting A and Setting B use 3B parameter size and 100B tokens, and the model from Setting A outperforms the model from Setting B, only proves that is better than B at that specific N, D size. Unlike conventional methods limited to specific configurations (e.g., fixed at 3B parameters, 100B tokens), our method identifies precise conditions under which one setting outperforms another. This extensibility fundamentally reshapes key insights, examples including:
Code/Math. What is the performance of code and math data with different mixture ratios on the scaling law? Will the Code Scaling Law differ from the Nature Language Scaling Law in various
42

---

parameters, thus leading to completely different compute-optimal character or different first-order derivatives with respect to N and D? Does a good Code/Math model require an ultra-large model size like nature language models do? We have already completed a portion of the relevant experiments, and this research is ongoing and will be published in subsequent paper.
Sparse LLMs. Do LLMs based on the MoE architecture [7] also follow the same functional form and predictive accuracy? And how should "Architecture Family" be defined within the MoE architecture? We have already run some experiments, and this research is ongoing and will be published in subsequent paper.
This paper uses Multi-Heaa Attention (MHA), a full attention paradigm. However, some studies suggest that certain types of Linear Attention-beds based LLMs perform better than full attention on smaller models [17]. The Farseer experimental methodology allows for a comprehensive comparison of the performance of different types of linear attention and full attention under various and D configurations. This could answer the key question, such as whether linear attention is only suitable for smaller D/N ratio. We look forward to the community building upon Farseer to complete this type of work.
Linear attention.
Model architecture. There are many more innovations at the model architecture level that need to be compared from a comprehensive scaling law perspective, such as Loop Transformer, etc. From this viewpoint, the focus is whether the relevant innovations can maintain performance as N scales up and whether performance saturates quickly as increases.
Data quality and repetition. Using high-quality data and employing different data repetition strategies, can performance be maintained N and increase? From the perspective of scaling laws, what is the trade-off relationship with larger amounts of middle-quality data?
Synthetic data. What patterns does synthetic data exhibit from scaling law perspective? Long CoT-like synthetic data more difficult to saturate with learning as D increases compared to general synthetic data?
43


Reviews:
[{"review_id": "B1GLBvuDW5", "reviewer": "Reviewer_XmBY", "summary": "This paper introduces Farseer, a novel scaling law for large language models (LLMs) that significantly improves prediction accuracy by refining the modeling of the loss surface  L(N, D). The core innovation of Farseer lies in its unique coupling mechanism between model size  N and data size D , which demonstrates a clear advantage over the fixed parameter assumption of the Chinchilla scaling law. The research team validated the effectiveness of this method through large-scale experiments on approximately 1,000 LLMs, consuming 3 million H100 GPU hours. The results show that Farseer achieves an error of less than 0.5% when extrapolating large-scale performance from small-scale experiments. Furthermore, the study reveals the variation in the optimal D/N ratio as the computational budget increases, providing crucial guidance for resource allocation in practical training scenarios.", "questions": "1. The loss surface modeling of Farseer adopts an exponential nesting form, but the paper does not theoretically explain why this form is most suitable for describing the scaling behavior of LLMs. Are there other theoretical bases supporting this design? It is recommended to supplement the theoretical analysis, for example: through the convergence analysis of gradient descent, illustrate how the exponential terms reflect the variation in optimization difficulty with scale; or compare with other functional forms (e.g., polynomial, logarithmic combinations) to prove the necessity of the current design.  \n2. The largest model validated in the experiments has 25.1B parameters, but the current mainstream LLMs have reached the trillion scale. Will the prediction error of Farseer increase under ultra-large scales?  \n3. The experiments are based solely on the Transformer architecture and bilingual (Chinese-English) data, but the LLM field has seen the emergence of new architectures such as MoE and multimodal systems. It is recommended to supplement the experiments across architectures or data types.", "limitations": "The authors clarified the primary reason for limiting the validation to LLMs with a maximum of 25B parameters and not including experiments with larger models. They outlined follow-up research plans focusing on MoE architectures and code data. The authors also explained the rationale for choosing BPC instead of downstream task evaluations. They acknowledged the common issue of lacking theoretical explanations in the scaling law domain. However, while mentioning that large-scale training requires complex parallelization strategies that introduce performance noise, they did not quantify the impact of this noise on Farseer's prediction error.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Advantages:\n1.Highly innovative: Farseer scaling law explicitly couples model size (N) and data size (D), significantly improving prediction accuracy compared to existing methods such as Chinchilla.\n2.Extensive experimental validation: Based on experiments with 1,000 LLMs of varying scales (3 million H100 GPU hours), the conclusions are highly credible due to the abundance of data. When extrapolating large-scale performance from small-scale experiments, the prediction error is less than 0.5%, which is markedly superior to benchmark methods like Chinchilla.\n3.Rigorous methodology: Employing differential analysis and multi-stage fitting strategies avoids overfitting and ensures model generalizability.\n4.Outstanding practical value: Provides quantifiable resource optimization schemes for LLM training, reducing trial-and-error costs.\nDisadvantages:\n1.The formula design relies on empirical fitting and lacks a solid theoretical explanation.\n2.The maximum validated model size is only 25.1B parameters, without covering the scale of hundreds of billions, raising concerns about the reliability of extrapolation to ultra-large scales. Additionally, the research is limited by computational capacity, with validation restricted to Llama-style decoder Transformers, failing to encompass a broader range of model architectures.\nI look forward to further research based on Farseer!", "was_revised": true, "final_justification": "Thanks for the solid work and nice responses by the authors. I have carefully read the authors' responses to my concerns and comments, as well as the interactions between the authors and all reviewers. Almost all reviewers' concerns have been satisfactorily resolved. Therefore, in my opinion, it meets the standards required by the conference.", "rebuttal": "We sincerely thank you for your thorough review and positive evaluation of our work. We appreciate your recognition of Farseer's innovation, extensive validation, and practical value. Your questions are insightful, and we would like to address them below.\n\n---\n**Disadvantage 1 & Q1: Regarding the theoretical explanation for the formula's exponential form**\n\nYou correctly pointed out that our paper lacks a deep theoretical explanation for the chosen exponential form. We agree that a complete theoretical justification for a scaling law's functional form is a challenging but important goal. While a first-principles derivation remains an open problem, we did empirically validate our choice. To help the community tackle this open problem, we have open-sourced our extensive experimental data, models, codes and training curves.\nAs you suggested, we compared our exponential form against other functional forms. Due to space constraints, these results were included in Appendix C.3, Table 2. The results clearly demonstrate that our proposed formula provides the best fit among the alternatives we tested, justifying our choice from an empirical standpoint.\n\n**Disadvantage 2 & Q2: Regarding the maximum validation scale of 25.1B parameters**\n\nYou raised a valid concern about the extrapolation of Farseer to models larger than our maximum validated size of 25.1B parameters. We acknowledge this limitation. This was primarily due to constraints on computational resources and the increasing engineering complexity of stable, large-scale parallel training. We believe our experiments up to 25.1B provide strong evidence for Farseer's effectiveness, and we acknowledge that validating on even larger models is a critical next step as resources permit.\n\n**Q3: Regarding the scope of model architectures and data types**\n\nThank you for this excellent suggestion. We agree that testing across diverse architectures and data modalities is crucial for generalizability. We are pleased to clarify two points:\n- Data Types: In Section 4.1 and Figure 6, we demonstrate Farseer's robustness across different data distributions. Furthermore, as part of our ongoing work, we have conducted a detailed study of the code domain, including the effects of various code-mixing ratios. These experiments confirm that the Farseer scaling law continues to yield highly accurate predictions. We believe a full analysis of these results is an important direction for future work and plan to report them separately.\n- Architectures: As for architectures like Mixture-of-Experts (MoE), they introduce additional complexities and variables (e.g., sparsity, fine-grained routing) that require more detailed modeling and experimentation. This is an exciting direction that we have planned for our future work.\n\nWe hope these clarifications address your concerns and further highlight the contributions of our work. Thank you once again for your valuable feedback."}, {"review_id": "o7TqaGxofY", "reviewer": "Reviewer_B5rQ", "summary": "This work proposes a new functional form and method for fitting tends in the loss surface of pretrained language models, L(N, D). Unlike the common baseline “Chinchilla” approach, this work considers the higher-order interaction between #params (N) and #data (D). They train a very large sweep of models up to ~10^10 params and ~10^11 tokens, and observed finite differences in loss wrt D lead a paramerizing the power law of D with N. They then use a differential piecewise and iterative fitting technique. The result is an estimate of L(N,D) that: 1) achieves better error than Chinchilla for small and large models, 2) suggests an optimal token-param ratio more dependent on compute size, 3) is robust across both an English and English+Chinese pretraining mix, 4) extrapolates well, e.g. Farseer overtakes Chinchilla on predictions of a 25.1B param model when using at least a 2.3B param model for fitting (though with 1.9B param model as the largest fitting model Chinchilla performs slightly better). Finally note that the authors release all “models, data from ∼1,000 trained LLMs, 61 detailed logs, and the Farseer code.”", "questions": "- Will you release training code or just Farseer fitting code? Glancing at the supplement the code there does not seem to include training code.\n    \n- Really like figure 7! Might be helpful to put earlier to get a quick summary of all the experiments you ran. Early on I was wondering if you needed some kind of specialized set of experiments to fit your method, but seeing this figure I understand you just need a grid of models with the grid step determined by lambda.\n    \n- The curated validation set in appendix D sounds very interesting! I would love to know more. Will it be released? Is the 2024 data cutoff intended to be after the data cutoff of the train data? Does the English+Chinese data also have a val set like this?\n    \n- I like that the various scaling law functional forms are listed directly in the intro! A small but nice though would be to just say that all other params than N and D are fitted.\n    \n- figure 2: What are the N values here? fixed? Why show this specific slice?\n    \n- Line 20: LM progress “attributed to scaling laws”; does this mean the discovery and use of scaling laws has largely driven progress in the field or does this mean that improvements in scale which are described by scaling laws are the mechanism of most improvement. The latter seems like a much less contentious assertion.\n    \n- line 45: “the very exploration for superior scaling laws is itself severely hampered by a significant scaling gap.” ([pdf](zotero://open-pdf/library/items/8LZPGLQW?page=2)) This seems to suggest that the issue with the scaling gap is that it inhibits research on how to fit scaling trends. But the rest of the paragraph describes how the scaling gap causes problems with applying predictions from existing scaling law methods to model development. The latter makes sense to me, but I’m not sure I follow the quoted section. What exactly do you mean by the scaling gap?\n    \n- figure 4: saying what A(N) and B(N) represent in direct language (the model-size dependencies of the data scale exponent and coefficient) would help the caption a lot and make the figure self contained. Its also hard to understand what it means to have an “actual” values for this until you read the running text.\n    \n- Figure 5: likewise give some explanation in the caption of G(.) U(.) etc…\n    \n- figure 6: the x axis label on the figure is confusing. The caption description is better. Maybe try just: “Largest model size used in fitting”.\n    \n    - This figure seems redundant with figure 8. It’s more helpful to see it with the chinchilla results as baseline.\n        \n    - This seems like a somewhat modest gap in compute (from 1B to 5B, I’m assuming D is comparable?), do you have results farther on the left of the x axis?\n        \n- eq 4: consider maybe using more mnemonic names for the components, e.g., “L_N(N)” instead of “U(N)”\n    \n- line 86: I think this is supposed to point to Appendix F not A right?\n    \n- figure 3: the ticks names are too sparse (sometimes just 1 per axis). Also maybe mark what Y axis direction indicates faster improving loss? Also the caption is a bit unclear to me. What does it mean to “adobt this form in our main analysis”? Also A(.) and B(.) are reffed before definition here. Also is the r^2 here is over all the points not just per line?\n    \n- line 130: typo in latex hat\n    \n- The motivation for what is being developed in 3.2 is hard to follow. Maybe start clarifying that you are trying to fit the lines in Figure 3 (a) and that stage 1 and 2 are about fitting the F_a and F_b. This would be clearer if instead of \\hatR_N(D) you used like `\\hat\\delta_DL(N,D)`\n    \n- line 162: I think this ref should be fig5 (b)?", "limitations": "The paper does not have a dedicated limitations section in the body of the paper but does discuss limitations in the appendix", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. This is a significant open source scientific release. To my knowledge this is the largest sweep over #params and #tokens, and will likely become a standard resource for future scaling law studies. They also release their data (hopefully both pretraining and validation) which is still relatively rare for works of this scale and can help all other kinds of research such as membership inference. And they also release their code, though the submission does not clarify if this includes training code or just loss surface fitting code.\n    \n2. I’m particularly impressed that they did a robustness test over a different pretraining data distribution!\n    \n3. Beyond having improved error over chinchilla scaling laws, their results also give us a more fundamental insight into optimal compute allocation, which raises an interesting question as to why optimal token/param ratio appears to increase with total compute budget. Similarly staring at their observed finite differences in loss in figure 3 is fascinating for starting to puzzle through the deeper dynamics here.\n    \n4. Provides a practical insight for developers that we should abandon the chinchila rule of thumb where optimal token-param ratio is often assumed as a constant 20.  \n    \n\nWeaknesses:\n\n1. It is difficult to judge how significant the improvement over existing methods is in practical terms. The paper doesn’t show a comparison between the proposed approach and baselines in terms of cost; it would be important to see what loss prediction error is achieved for a certain ratio of small experimental compute cost to large final run cost. We can see this partially in figure 8 which has the largest fitted model parameter size as an x axis. I am convinced that Farseer is better than Chinchilla for relatively large experiment/final cost ratios but there is some hint from this figure that Chinchilla might be better for cheaper predictions which may be more practical for large sweeps of development experiments.\n    \n2. It’s also difficult to judge what is a “low enough” loss prediction error. In so far as loss predictions are used to make practical model development decisions it would be important to see if Farseer can make better decisions than Chinchilla on predictive model selection tasks like DataDecide ([https://arxiv.org/abs/2504.11393](https://arxiv.org/abs/2504.11393)).\n    \n3. The notation is unnecessarily terse and arbitrary (a whole alphabet of function names). It would be clearer if these single letter function names were expand into short italicized words or to use color and text underneath terms in the equations to clarify what these are. For instance the full schematic of terms “L(N, D) = V (D) + H(N, D) + E + U (N )” ([pdf](zotero://open-pdf/library/items/8LZPGLQW?page=3)) could mark each of the terms (or sets of terms) with the section number in the paper that details them. Likewise explaining “what kind of data is needed for this fitting” up front would be helpful before explaining the whole fitting method. Is this data substantially more expensive to collect than previous scaling laws?", "was_revised": true, "final_justification": "The authors' response reassured me that they have a good plan for revising the clarity issues I raised. I don't have any other significant concerns remaining. I think the paper will be very impactful as it convincingly unseats a fundamental assumption about base language modeling scaling by showing that optimal tokens to parameter ratio is dependent on compute scale rather than being captured by the fixed \"Chinchilla\" rule of thumb. This finding should not just change practices in industry but will help clarify many scientific findings as understanding when an experiment is or is not overtrained is essential to reaching accurate conclusions to many questions.", "rebuttal": "Thank you for your detailed, thoughtful, and constructive review. We are encouraged by your positive assessment of our work's significance, originality, and the value of our open-source contributions. We appreciate the specific questions and suggestions for improvement, which will help us strengthen the paper.\nBelow, we address the weaknesses and questions you raised.\n\n---\n\n**W1: Prediction precision and cost.**\n\nYou are right to point out that in Figure 8, Farseer's error is comparable to Chinchilla's when fitting on a very small number of models. This is an expected behavior. Because our formula is designed to capture the learning dynamics across a wider range of model sizes (N), it requires more data points across N to realize its full predictive power. When using only a few fitting points from smaller models, its performance naturally converges to the Chinchilla baseline. Its performance is largely comparable to the Chinchilla baseline when using only a few fitting points, and while it shows a very slight disadvantage at the initial point in Figure 8b, its superiority becomes clear as the number and scale of models used for fitting increase.\nFurthermore, a key contribution of our work is the empirical demonstration that Chinchilla's formulation, which lacks the higher-order interaction term H(N, D), leads to divergent prediction errors, as shown in Figure 1a and Figure 2. In contrast, Farseer's error remains convergent and stable across the full range of N.\nThe primary reason for this is that the Chinchilla model extrapolates using a single, average rate of data scaling derived from its fitting set to predict performance for all other model sizes. This means that as the target model size N for prediction grows larger and deviates from the fitting range, Chinchilla's error is expected to increase and diverge. Farseer, however, is designed to capture this variance, leading to higher predictive confidence for larger models. This is critical because in most industrial and research applications, the goal of using scaling laws is precisely to forecast the performance of these larger, more resource-intensive models.\nFrom another perspective, Chinchilla does not inherently benefit from being fitted on a wider range of model sizes (N). Farseer, on the other hand, leverages the additional data points across N to build a more accurate model of how data scaling dynamics change with model size. Therefore, this is not a simple trade-off between prediction accuracy and fitting cost. Instead, it concerns the fundamental capability of the scaling law to generalize reliably to the large-scale regimes that are of greatest practical interest.\n\n**W2: On defining a \"low enough\" loss prediction error and performance on model selection tasks like DataDecide.**\n\nOur definition of \"low enough\" is based on the relative error at scale, which approaches the inherent noise from factors like training stability or data parallelism. The key finding is the significant improvement over the Chinchilla baseline within this context. Thank you for the excellent suggestion to evaluate on predictive model selection tasks like DataDecide. This is a great way to measure the practical impact of improved loss prediction, and we will explore incorporating such an evaluation to more directly demonstrate the benefits of our method for development decisions.\nTo supplement this, we can share an observation from our industrial practice. When making critical architectural decisions, such as comparing different Attention structures, experiments are typically conducted with models in the scale of tens of billions of parameters trained on hundreds of billions of tokens. In this context, a performance gap between two methods often needs to exceed 0.6% to be considered significant, while the entire performance difference between two competing approaches is rarely more than 3%. However, as shown in Figure 8b, Chinchilla's relative prediction error at the 25.1B scale already exceeds 2.5%. This level of inaccuracy severely undermines its practical value, as the prediction error is nearly as large as the performance margin one hopes to distinguish.\n\n**W3, Q2, Q4, Q8, Q9, Q11, Q15: On notation, clarity, and presentation of the formulas and methodology.**\n\nThank you for this collection of detailed and very helpful suggestions regarding the clarity of our paper. We sincerely appreciate your careful reading and feedback. We agree that the presentation can be significantly improved. In the revised version, we will perform a thorough update on our notation and presentation based on your feedback. This includes:\n- Adopting more mnemonic function names (e.g., your suggestion of L_N(N) instead of U(N)).\n- Adding direct, plain-language explanations of terms like A(N), B(N), G(.), and U(.) to the figure captions to make them self-contained.\n- Moving the overview of experiments (Figure 7) earlier to clarify the data requirements for our fitting method from the outset.\n- Explicitly stating that all parameters other than N and D are fitted.\n- Restructuring the beginning of Section 3.2 to more clearly state the motivation and logic of our fitting procedure.\nWe believe these changes will substantially improve the paper's readability and logical flow.\n\n**Q1 & Q3: On the release of training code and the curated validation set.**\n\nThank you for your interest. We are open-sourcing the Farseer fitting code. The training code, which is a modified version of the open-source project Megatron-LM, will not be released. Our primary modifications involve custom implementations to accelerate Data Parallelism (DP) and Tensor Parallelism (TP). For the smaller models in our study, which do not require complex parallel strategies, we believe that training with the standard community version of the codebase would not significantly impact the results.\n\nFurthermore, ensuring the validity and unbiased nature of the validation set was a resource-intensive endeavor, requiring substantial manual effort to maintain impartiality in the data curation process. For instance, the book portion of the set was fully annotated by human reviewers, rather than relying solely on machine-based OCR. We understand the value that releasing this validation set would provide to the community for reproducing our work. Consequently, we are actively seeking approval to make this dataset publicly available in the future.\n\nThe 2024 data cutoff for the English validation set is indeed intended to ensure it contains data unseen by the models, as its academic paper sources are from arXiv publications after 2024. We can confirm that we constructed similar validation sets for Chinese and code, and the fitting performance was also strong on these, though the English validation set was the primary one used for the results reported in the paper.\n\n**Q5: Regarding the specific slice shown in Figure 2.**\n\nIn Figure 2, the model size is fixed at N=6B. We chose this specific slice to provide a clear, direct comparison between the Farseer and Chinchilla functional forms against the ground-truth loss points. This visualization effectively highlights the improved fit our proposed form provides for a given model size across varying data sizes.\n\n**Q6 & Q7: On the role of scaling laws and the meaning of the \"scaling gap\".**\n\nTo clarify our statement that LM progress is \"attributed to scaling laws,\" we mean that the discovery and use of these laws have given researchers the confidence to invest massive compute resources, knowing they can be predictably converted into model capability. Scaling laws have been essential for guiding and evaluating this process.\nBy \"scaling gap,\" we refer to the phenomenon where insights and conclusions drawn from small-scale experiments (e.g., identifying a superior dataset or config) do not consistently hold true when scaling up to much larger models. A more accurate scaling law, like Farseer, can help mitigate this by providing more reliable predictions of large-scale performance, thereby making small-scale exploratory experiments more predictive and valuable. In fact, Appendix E provides a simple example from this scaling-law perspective.\n\n**Q10: Regarding the x-axis in Figure 6 and its relation to Figure 8.**\n\nYou are correct. The x-axis label in both Figure 6 and Figure 8 is \"Largest model size used in fitting\". The figures do show similar trends but are complementary as they show extrapolation to different target model sizes. We agree Figure 8 is more informative with the Chinchilla baseline and will consider merging or refining the presentation to avoid redundancy.\n\n**Q12, Q13, Q14, Q16: On various typos and clarifications.**\n\nThank you for spotting these. You are correct on all points.\n- Line 86: The reference should indeed point to Appendix A, which details hyperparameter and architecture ablations. Appendix F contains the specific model settings.\n- Figure 3: We will increase the tick density on the axes. The y-axis direction indicating faster loss improvement will be clarified. The R² value shown is the average R² across each of the fitted lines. We will clarify the caption and define terms before they are used.\n- Line 130: We will correct the LaTeX typo.\n- Line 162: You are right, the reference should be to Figure 5(b).\n\nWe will correct all of these in the revision. Thank you again for your meticulous review."}, {"review_id": "lhVZjdkyPi", "reviewer": "Reviewer_NtZX", "summary": "The paper revisits the data and model size dependent scaling laws in the literature and improves them by adding an additional term that is both data- and model size-dependent. This way the scaling rate as data/model size increases differs according to the particular model size)data.", "questions": "Around line 96, why isn't H(N, S) included in model-dependent terms?", "limitations": "I don't think there is enough (or any) coverage of limitations. The proposed law brings some improvements over the Chinchilla laws but it's still a model that comes with limitations. It would be great to see a discussion on this \n\nAnother obvious drawback that deserves a discussion is the increased number of parameters in the law compared to Chinchilla. This would require more small-scale models to be trained.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- Modeling the interrelations between the data and model size at scale is very important. The paper does a good job demonstrating the existing laws' inability to capture such relations and show empirical gains over them \n\nWeaknesses:\n\n- It's not clear how the scaling law form on line 92 changes to the one on line 97. The definition of \"f\" comes later but it's confusing to use it on line 97 without an explanation.\n\n- Around line 96, why isn't H(N, S) included in model-dependent terms?\n\n- The proposed law, Farseer, has 6 degrees of freedom as opposed Chinchilla's 5 degrees of freedom. While it makes sense that this additional parameter is needed to capture the dependency between the effect of N and D, it also increases the number of small-scale models needed to be trained to fit the scaling laws. I suggest the authors discuss this additional cost in the paper.\n\n-Minor: some typos and grammatical errors: \"H(N,D) models represents on line 70, \"detailed in detailed in\" on line 77. Also, footnote 1 of page 3 appears at page 4.", "was_revised": true, "final_justification": "I thank the authors for the clarifications. The response sufficiently addresses my points. Since the authors said these will be reflected in the revised paper, I increase my score.", "rebuttal": "Thank you for your efforts and contributions in reviewing the paper. Below are our responses to the relevant issues; we hope they will address your concerns and improve the quality of the manuscript.\n\n---\n\n**W1: Clarity on the transition from the formula on line 92 to line 97**\n\nThank you for pointing out this lack of clarity. You are correct that the transition was abrupt and lacked proper explanation. The formula on line 92 is the general, high-level form that we hypothesize. The formula on line 97 is the specific, empirically-derived functional form that resulted from our analysis in Section 3.\nSpecifically, through our data exploration, we found that the general terms could be effectively approximated as follows:\n- The term V(D)+H(N,D) is approximated by $f_B(N;θ_{B∗})D^{f_A(N;θA∗)}$\n- The term E+U(N) is approximated by $f_U(N;θ_{U∗})$\nWe improperly present the final form on line 97 before detailing the derivation and justification. This section will be revised in the new version of the manuscript to ensure a logical flow, in which introduce the general scaling law and then systematically explain how our empirical findings lead to the specific functional forms, defining all terms before they are used.\n\n**W2: Why H(N, D) is not included in model-dependent terms**\n\nThis is an excellent and perceptive question.Our choice to group H(N, D) with the data-dependent term V(D) is motivated by both theoretical precedent and strong empirical evidence from our own analysis.\nOn one hand, our formulation is inspired by the theoretical analysis in the Chinchilla paper(Appendix D.2.) [1], which we use as a foundational starting point. In their work, the total loss is decomposed into: \n\n$$\nL(N, D) = L(f^\\star) + \\left(L(f_N) - L(f^\\star)\\right) + \\left(L(\\bar{f}_{N,D}) - L(f_N)\\right)\n$$\n\n1. A term limited by the model size (N), which is $L(f_N) - L(f^\\star)$\n2. A term, which we denote as H, that is related to the data size (D) and the training dynamics, which is $L(\\bar{f}_{N,D}) - L(f_N)$\n\nOn the other hand, our decision to treat the term $$H(N, D)$$ as a power-law function of the data size, $D$, was fundamentally an empirical one, guided by the results of our analysis.  Our first-order difference analysis (Section 3.1) provided strong evidence for this modeling choice. Specifically, as shown in Figure 3a, the analysis of $$\\Delta_D L(N,D)$$ versus $$D$$ revealed a remarkably stable and clear power-law relationship, evidenced by a high $$R^2$$ value of 0.9807. This empirical observation led us to model $$H(N,D)$$ as a power law with respect to $D$.  Crucially, we recognize that this power-law relationship is not independent of the model size, $N$. Our findings indicate that the parameters of this power law, most notably the exponent, are themselves a function of $N$. Our key contribution, therefore, is demonstrating that the complex behavior of the $$H(N,D)$$ term can be effectively captured by modeling it as a power law in $D$, whose characteristics are modulated by $N$. We will clarify this distinction in the revised manuscript.\n\n\nWe admit that the original manuscript did not explain this motivation clearly. In the revised version, we will explicitly state that we adopt the loss decomposition from Chinchilla  and then clearly articulate that our primary contribution is demonstrating and modeling the previously overlooked dependence of the H term on N.\n\n**W3: Increased degrees of freedom, fitting cost**\n\nRegarding the additional parameter:\nOur parameters is intentionally introduced and is crucial for capturing the interaction effect between model size (N) and data size (D), which, as our results show, is a key factor that existing laws fail to model.\nRegarding the fitting cost: \n1. Farseer achieves a comparable and evne more accurate with  a small number of data points.\nWhile more parameters can sometimes require more data, our Farseer law demonstrates greater efficiency in capturing the true scaling behavior. In fact, our Figure 8 already provides a preliminary illustration of this, showing that Farseer achieves a more accurate fit even with a small number of data points.\n2. The error  of Chinchilla becomes divergent, while Farseer's remains stable. \nThis is a key contribution of our work. It is the empirical demonstration that Chinchilla's formulation, which lacks the higher-order interaction term H(N, D), leads to divergent prediction errors, as shown in Figure 1a and Figure 2. In contrast, Farseer's error remains convergent and stable across the full range of N.\n3. Farseer is closer to the true nature of Scaling Law, not just a simple accuracy-cost trade-off. The primary reason for Chinchilla's divergent error is that it extrapolates using a single, average rate of data scaling derived from its fitting set to predict performance for all other model sizes. This means that as the target model size N for prediction grows larger and deviates from the fitting range, Chinchilla's error is expected to increase and diverge. Farseer, however, is designed to capture this variance by modeling the interaction between N and D, leading to higher predictive confidence for larger models. This is critical because in most industrial and research applications, the goal of using scaling laws is precisely to forecast the performance of these larger, more resource-intensive models.\nFrom another perspective, Chinchilla does not inherently benefit from being fitted on a wider range of model sizes (N). Farseer, on the other hand, leverages the additional data points across N to build a more accurate model of how data scaling dynamics change with model size. Therefore, this is not a simple trade-off between prediction accuracy and fitting cost. Instead, it concerns the fundamental capability of the scaling law to generalize reliably to the large-scale regimes that are of greatest practical interest.\n4 . A scaling law is only valuable if its prediction error is within a reasonable range. Here, we wish to add an observation from our industrial practice. When we need to use large-scale models to determine key conclusions, such as the effectiveness of different Attention structures, experiments are often conducted at the scale of tens of billions of parameters (e.g., >20B) and hundreds of billions of tokens. In this context, a performance difference between two approaches must typically exceed 0.6% for us to consider it a significant gap, while differences below this threshold are often attributed to noise or engineering implementation details. In our experience, the performance gap between two distinct, viable solutions rarely exceeds 3%. However, as shown in Figure 8, Chinchilla's prediction error for a 25B model already surpasses 2.5%. An error of this magnitude severely diminishes its practical value, as it can obscure the true performance differences between models and lead to incorrect conclusions.\n\n**About limitations:**\n\nWe had previously included an extensive two-page discussion of \"Discussion and Future Work\" in Appendix J. However, we completely agree that a more thorough and centralized discussion of limitations is needed. Prompted by your suggestion, we recognize that this topic deserves a more prominent treatment in the main paper. To that end, we have added a dedicated \"Limitations\" section to the revised manuscript. In this new section, we will not only summarize the various aspects we originally mentioned in Appendix J, but we will also condense and integrate our points regarding the \"fitting cost\" of Farseer(as mentioned above).\nWe agree that adding this section to the main body makes the paper stronger, and we appreciate you pushing us to improve its clarity and completeness.\n\n[1] [Training Compute-Optimal Large Language Models](https://arxiv.org/abs/2203.15556)"}, {"review_id": "S3cf4Cdh0X", "reviewer": "Reviewer_gWVt", "summary": "The authors of this paper introduce Farseer, a set of scaling laws designed to describe the performance of language models. The scaling laws proposed by the authors are designed to extrapolate well beyond the setting where the number of parameters and the number of training tokens are comparable to each other. The authors compare to another popular formulation for scaling laws in this setting (Chinchilla) and demonstrate better results in the extreme ends of the scaling of available compute.", "questions": "- I would be grateful if the authors could let me know whether they have looked into applying their ideas in predicting downstream performance, as mentioned above.", "limitations": "The authors have adequately addressed the limitations and the impact of their work.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- Scaling laws are a very important topic and of great interest to the NeurIPS community. As model and dataset sizes increase, experimentation becomes harder. Being able to iterate on smaller scales and extrapolate on larger ones is a very important capability, which allows for easier improvements upon existing techniques.\n\n- The paper is very easy to follow and very clear in writing. The method itself is also presented in a very clear manner.\n\n- As a method, Farseer is a clear improvement upon prior work. While prior scaling laws ignore the interaction terms between the number of training tokens and model size, Farseer explicitly incorporates this term in the scaling law. While this could lead to difficulties in fitting the scaling law itself, the authors have been able to present a clear technique to circumvent this (namely, differential piecewise fitting and iterative improvements over the fit). The proposed method is well motivated, with several experiments to determine the form of each term in the equation (by justifying a power law behavior for the data dependent part $V(D) + H(N,D)$, and using that as a starting point to predict the form of the rest of the terms combined).\n\n- The empirical performance of the proposed scaling law is also very good. The authors demonstrate low relative error of the predicted loss for the model, even on points that were not used to fit the scaling law (which lie above the model sizes used for fitting), thus demonstrating good generalization. Comparisons with Chinchilla also demonstrate a clear improvement over prior forms of scaling laws.\n\nWeaknesses\n\n- While not necessarily a weakness, I believe the paper would be further improved by adding more baselines for comparison if possible, as well as potentially examining the applicability of the approach of Farseer in predicting the downstream performance of models as well (instead of only the loss).\n\n- Some minor comments:\n\n  - Figures 2 and 7 should float to top or bottom of page if possible, for readability.\n\n  - In line 130, there is a LaTeX error for $\\hat{B}_N$.\n\n  - The syntax at lines 146 - 147 is a bit weird.\n\n  - In line 168, the first sentence should be bolded.", "was_revised": true, "final_justification": "As I mentioned in my original review, I believe Farseer to be a very solid work, with clear improvements upon scaling laws proposed in prior works. The responses of the authors to both mine and other Reviewers' comments further reinforce this belief, with additional resutls and clarifications. As such, I am keeping my score of \"Accept\" for this paper.", "rebuttal": "We sincerely thank you for your thorough review and positive assessment of our work. We are very encouraged by your recognition of our paper's clarity, novelty, and significance. Your insightful suggestions are invaluable for improving our manuscript.\nBelow are our responses to your specific points.\n\n---\n**W1 & Q1: More baselines and downstream tasks**\n\nThank you for these excellent suggestions.\n\n**Regarding the inclusion of more baselines,** we focused our comparison on the Chinchilla scaling law because it represents the most widely recognized and dominant formulation in this area. Many other proposed scaling laws are often modifications of the Chinchilla framework or are tailored for very specific, narrow tasks. Therefore, we believe that a direct and thorough comparison with Chinchilla provides the clearest demonstration of Farseer's advancement over the current state-of-the-art.\n\n**Regarding the prediction of downstream task performance,** we agree this is a very important research direction. We did not include it in this work because establishing a precise and stable scaling law for downstream benchmarks is a substantial research challenge in its own right, due to several sources of high variance in their evaluation.\n\nSpecifically, we identify three main issues:\n1. Stochasticity in Decoding: Most benchmarks rely on an LLM decoding process followed by an answer-matching script, a method that is inherently probabilistic. A model's score can fluctuate significantly across identical runs. While this can be partially mitigated by averaging over multiple decoding attempts (i.e., avg@N), it introduces measurement noise that complicates the modeling of a clean scaling signal.\n2. Benchmark Sensitivity and Scope: The performance on any single benchmark can be volatile. This arises because (a) benchmarks often have a limited number of test items, making scores susceptible to high variance, and (b) they are extremely sensitive to the model's output formatting and its ability to follow instructions. A pretrained model's instruction-following capability can change drastically over very few training iterations depending on recent data exposure. This can cause large swings in benchmark scores that reflect shifts in instruction adherence rather than fundamental changes in the model's core capabilities. To obtain a more stable signal, one might need to cluster similar benchmarks and predict an aggregate metric, which adds its own layer of modeling complexity.\n3. Confounding Effects of Fine-Tuning: To fairly compare the underlying capabilities of different pretrained models, one cannot simply test them directly. A rigorous evaluation would require first aligning each model with a standardized, lightweight instruction-following fine-tuning. This would normalize their instruction-following abilities to a similar level before evaluation. However, designing and constructing such a benchmark instruction-following dataset is a highly demanding task in itself.\nFor these reasons, directly modeling downstream performance introduces significant confounding variables that would obscure the scaling dynamics related to pre-training. We believe that rigorously addressing these measurement challenges to isolate a clean scaling law for downstream tasks is a substantial endeavor that warrants a dedicated future study. We have added a discussion of this as a promising direction for future work in our revised manuscript.\n\n**W2: Minor comments on typos and formatting**\n\nThank you very much for your meticulous reading and for pointing out these issues. We appreciate you catching the formatting suggestions for Figures 2 and 7, the LaTeX error in line 130, the syntax in lines 146-147, and the missing bolding in line 168. We have corrected all of these in the updated version of the paper.\n\nOnce again, we thank you for your valuable feedback and strong support for our paper. We hope our responses have satisfactorily addressed your questions."}]

Author rebuttal:

Meta-review: This paper proposes Farseer, a refined scaling law for large language models that explicitly models the interaction between model and data size. The work is supported by an unusually large empirical study (~1,000 trained models) and a comprehensive open-source release.

Reviewers agreed that the contribution is both novel and practically important, showing clear improvements over the widely used Chinchilla law in terms of predictive accuracy and stability. The main concerns were the lack of a strong theoretical foundation for the functional form, the limited maximum scale of validation (25B parameters), and some clarity issues in notation and presentation. The rebuttal addressed these issues satisfactorily, clarifying empirical justification for the functional form, explaining resource constraints, and committing to substantial improvements in clarity and limitations discussion.

Following the rebuttal, reviewer consensus was strongly positive, with multiple score increases. Overall, this is a technically solid and impactful paper that is likely to influence future research and practice in scaling laws, in particular, through its open source release of data.
Decision: Accept (spotlight)