Gemstones: A Model Suite for Multi-Faceted Scaling Laws
Authors: Sean Michael McLeish
John Kirchenbauer
David Yu Miller
Siddharth Singh
Abhinav Bhatele
Micah Goldblum
Ashwinee Panda
Tom Goldstein
Venue: NeurIPS 2025 poster ()
Keywords: neural scaling laws
large language models
Primary area: deep_learning
TL;DR: We fit scaling laws for large language models with varying width-to-depth ratios and parameter counts.

Abstract: Scaling laws are typically fit using a family of models with a narrow range of frozen hyperparameter choices. 
In this work we study scaling laws using multiple architectural shapes and hyperparameter choices, highlighting their impact on resulting prescriptions.
As a primary artifact of our research, we release the Gemstones: an open-source scaling law dataset, consisting of over 4000 checkpoints from transformers with up to 2 billion parameters and diverse architectural shapes; including ablations over learning rate and cooldown.
Our checkpoints enable more complex studies of scaling, such as analyzing the relationship between width and depth.
By examining our model suite, we find that the prescriptions of scaling laws can be highly sensitive to the experimental design process and the specific model checkpoints used during fitting.

Gemstones: A Model Suite for Multi-Faceted Scaling Laws
Sean McLeish ' John Kirchenbauer' David Yu Miller', Siddharth Singh' Abhinav Bhatele', Micah Goldblum?, Ashwinee Panda', Tom Goldstein University of Maryland, Columbia University
Abstract
Scaling laws are typically fit using a family models with a narrow range of frozen hyperparameter choices. In this work we study scaling laws using multiple architectural shapes and hyperparameter choices, highlighting their impact resulting prescriptions. As primary artifact of research, release the Gemstones: an open-source scaling law dataset, consisting of over 4000 checkpoints from transformers with up billion parameters and diverse architectural shapes; including ablations over learning rate and cooldown. Our checkpoints enable more complex studies of scaling, such analyzing the relationship between width and depth. By examining our model suite, we find that the prescriptions of scaling laws can be highly sensitive the experimental design process and the specific model checkpoints used during fitting. Code:
Introduction
Existing works on scaling laws often restrict Transformer architectures to a small range of width- depth ratios [Porian et al., 2024], train on small number of tokens, and fix training hyperpa- rameters such as cooldown schedule across training runs [Hoffmann al., 2022]. These design choices, in turn, can dramatically influence the resulting scaling laws. If a scaling law is sen- sitive to such design choices, then may only be useful for practitioners implementing simi- lar setups to those that produced the scaling law. In practice, practitioners often take guidance from scaling laws that assume completely different design choices than their own implementa- tion, often without understanding to degree to which these choices may impact optimal scaling.
In this work, we produce vast array of model checkpoints for studying how model design and model selection impact scaling laws. Our models, called the Gemstones because they are loosely based on scaled-down variants of the Gemma architecture, vary in their parameter count, width/depth ratio, training tokens, learn ing rates, and cooldown schedules. By fitting scaling laws to these checkpoints, we confirm that scaling law parameters and interpretations indeed depend strongly on the selection of mod- els and fitting procedure used, and we quantify the degree to which these decisions impact pre- dictions. By exploiting the variation among our
Entrediing
Depth
Decode
MLP
Width
Decodar Bock
Residual Connection
Figure 1: The meaning of width and depth. We visualize a standard transformer architecture, high- lighting the "width" as the size of the hidden di- mension and the "depth" as the number of trans- former blocks.
"Correspondence to: smcleish@und edu.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

model checkpoints, we also analyze the impact of architectural shape across loss, benchmark perfor- mance and training time with findings consistent with design choices we see in industry models. Our contributions are summarized as follows:
We open-source more than 4000 checkpoints cumulatively trained on over 10 trillion tokens. The models we provide are diverse across architectural and training hyperparameter axes, enabling more granular studies than previous work (see Figure 2).
We highlight the fragility and common pitfalls of prior scaling laws. There are many decisions to make when choosing points to fit scaling laws that significantly change the slope of the law (see Table 1).
We analyze the impact of model shape on loss, benchmarks and training time. We find that although deep models achieve lower loss and benchmark error when measured using floating point operations, they require significantly more training time when using typical open-source parallelism frameworks (see Figures 4, 7 and 8).
Related Work
Scaling laws address the trade-off between parameter count and number of training tokens, attempting to find the minimum loss possible for a language model with a constrained floating point operation (FLOP) budget. The body of work on scaling laws is vast. Therefore, while we provide brief overview of key prior work here to contextualize our contributions, we also include an extended literature review in Appendix B.
Design Choices for Scaling Laws Scaling laws often treat model design and training as if it has a single dimension (parameter count), while, in practice, training is sensitive to many choices. Notably, Hoffmann et al. [2022] find significantly different fitted laws (Equation (1)) compared to Kaplan et al. [2020]. Pearce and Song [2024] and Porian el al. [2024] attribute most of this discrepancy to the choice to exclude embedding parameters from the parameter count, both showing one law can be transformed into the other via controlled changes. Kaplan al. [2020] justify excluding embedding parameters by showing that non- embeedddndng parameters have a cleaner relationship with test loss. Scaling laws are also commonly included in many large model releases [Hu al., 2024, Bi et al., 2024, Dubey et al., 2024].
Choshen al. [2024] collect both loss and benchmark performance metries for multitude of models and offer practitioner's s guide to fitting scaling laws. Notably, they suggest that 5 models are ample to fit scaling law, and that you should omit the early part of training when fitting. because those early steps don't 't follow the same scaling behavior and can skew the results. In contrast, Li et al. [2024c] demonstrate that varying the pereramter- ratio and relying on limited grid searches when fitting scaling laws can lead to large variations in results. Hägele et al. [2024] suggest that a constant learning rate plus cooldown is preferable a cosine learning rate schedule as all intermediate checkpoints can be used for fitting. The authors also find that stochastic weight averaging should be encouraged in scaling law analysis as tends to lead to better models. Furthermore, Inbar and Sernau [2024] observe that FL.OPs cannot be used to predict wall-clock time nor memory movement, and suggest that fast-training architectures may be preferred over those prescribed by scaling laws.
There are multiple works analyzing whether scaling laws can be used to predict downstream perfor- mance. Bhagia et al. [2024] first predict task-specific loss and then use this to predict performance on the task, using sigmoidal function to map from loss to accuracy. Gadre et al. [2024] predict t error, fitting a power law function on the perplexity of the model to predict error. Gadre et al. [2024] also look at the impact of overtraining, finding scaling laws that extrapolate with the amount of overtraining. Dey et al. [2023] analyze the trade off of inference FLOPs and training FLOPs using scaling laws to prescribe training configurations that balance training and inference. Unfortunately. both Dey et al. [2023] and Biderman et al. [2023] train on the Pile [Gao al., 2020] which has since been taken down due to copyright, leaving gap for a model collection in the open literature.
The Role of Model Shape Another line of research specifically studies the interplay between model width and model depth; for clarity we visualize our working definitions for these dimensions in Figure 1. Levine et al. [2020] find that, for large models, optimal depth grows logarithmically with width. Henighan et al. [2020] find there is an optimal aspect ratio for each modality they study which

---

gives maximum performance: for example, they find 5 to be optimal for math models. Team et al. [2024b] compare two 9 billion parameter models and find the deeper model outperforms the wider one consistently across benchmarks. Unfortunately, the authors are vague about the specific details of this result. Petty et [2024] claim small (<400M) transformers have diminishing benefits from depth. Brown et al. [2022] show that in some cases shallower models can beat their parameter-equivvaeet deep models on tasks for encoder-decoder transformer architectures. These results differ from Kaplan al. [2020] who suggest aspect ratio is not a determining factor for final loss. Tay et al. [2022] show that downstream performance strongly depends on shape when finetuning but pretraining perplexity does not. Alabdulmohsin al. [2024] study the impact of width and depth for encoder-decoder vision transformers, using their laws to create a smaller transformer model which has competitive downstream performance when compared with much larger models. The architecture found in this study has since been used by Beyer et al. [2024] in PaliGemma. Concurrently, Zno et al. [2025] study the impact of width and depth in hybrid architectures, finding that deeper 1.5B model can match or even outperform 3B and 7B models.
As discussed above, the literature how the aspect ratio of LLM affects its performance and scaling characteristics simultaneously extensive but somewhat inconclusive. While we not presume to fully answer every question in this space, the experiments we describe in the rest of this work make progress on how to understand the results of prior studies and the impacts of certain architecture choices in fully open, reproducible, and extensible way.
Designing Our Scaling Laws
We discuss the design of our scaling laws, including model selection, the choice of learning rate, and curve fitting schemes in the subsequent sections and in greater detail in Appendix A.
Architecture. To reduce the search space of all possible models, add some constraints, each of which are either based on precedent from a popular model series like Gemma [Team et al., 2024a,b]. Llama [TTouvron et al., 2023b], Pythia [Biderman et al., 2023], practical considerations such as hardware details (see Appendix A).
Within these constraints, we search the set of feasible models within target parameter count 500M, 1B and 2B with tolcrance of +5%. At smaller scales we train up to models at diverse widths and depths. At large parameter counts we train only models, aiming for one "standard" aspect ratio (simi- lar to existing models), one "wide" model, and one "deep" model. We visualize the models choose to train in Figure 2 overlaid with a se- lection of existing models from prior work. In the Appendix, we plot the entire discrete set of all possible models under our constraints (Fig- ure 10). Our 22 different models range from 50M to 2B parameters, spanning 11 widths from 256 to 3072 and 18 depths from 3 to 80.
3. 3.15b
10*
3,1-706.
groups 50M, 100M,
2-13b
3.1-$b
2:34b
2-2b
-2-276
2b
76
Width
10²
10'
10²
Depth
Chischilla
Llams
Ours
Gemma
Porian al.
Distribution of prior scaling law mod- els, industry models, and our models in terms of width and depth. Prior work (purple and green) and industry models (blue and orange) mostly lie on a fixed width-depth line.
Figure 2:
Polishing the Gemstones. For the main set of training runs, we train each model for 350B tokens of Dolma 1.7 [Soldaini et al., 2024] data. We target a total batch size of 4 million tokens following[Toune Touvron et al., 2023b, Dubey et al., 2024, Bai et al., 2023], with a context length of 2048 and world batch size of 2048 sequences. Following Hägele et al. [2024] and Hu et al. [2024], we use a linear learning rate warm up over 80 million tokens, and then train constant learning rate, which adjust for model size as described in Appendix A.

---

In service of future research based on our model suite, we open source checkpoints for all models billion token intervals, amounting to over 4, 000 checkpoints in total. We also open source the fitingg code and logged metrics for all runs.
We perform ablations over both cooldown and learning rate. For the cooldown ablation, we take the checkpoints saved every 10 billion tokens for the the first 100 billion tokens of training and cool these down, creating a second set of models which have had their learning rate annealed to 0 linearly. Specifically,, we cool each model down by training for a further 10% of the total tokens which it has seen during training, i.e. our cooled down set of models have training budgets ranging from 11 to 110 billion tokens. We also ablate our choice of learning rate by running all models for 100 billion tokens with half of the learning rate we use for our main analysis. The full details of the scalable learning rate and parameter initialization scheme- scheme-designed to enable hyperparameter transfer across model sizes and aspect ratios-are are provided Appendix A.
Training Details We train with AdamW [Loshchilov and Hutter, 2017] with /3 parameters 0.9 and 0.95 and weight decay of 0.1. We do not apply weight decay to the bias or normalization parameters. All models are trained with tensor parallelism [Singh and Bhatele, 2022, Singh et al., 2024] over multiple nodes of AMD MI250X GPUs. To the best of our knowledge, this makes the Gemstone suite of models the largest collection trained on AMD GPUs.
3.1 Fitting Scaling Laws
We fit scaling laws using methods similar to approach 1 and 3 from Chinchilla [Hoffmann et al., 2022]. We fit all laws using the log perplexity of all trained models sample of 100 million tokens from a fixed, held-out validation set from the training distribution. We also collect log perplexity values for a range of open source models [Team et al., 2024a,b, Touvron al., 2023b, Dubey al., 2024, Yang et al., 2024a,b] on the same validation data to allow for a comparison between our predictions and selection of widely used models. We design specialized FLOP counting function we find that simple rules of thumb (e.g., FLOPSs per token= 6 x paramelers [Hoffmann et al., 2022]) do not accurately account for differences in FLOPs between extremely wide and narrow architectures. We discuss this further and present our function in Appendix M.
Following Porian et al. [2024], we plot the Epoch Al Replication [Besiroglu al., 2024] of Chinchilla [Hoffmann et al., 2022] on all plots and use the coefficients for Kaplan plotted by Porian et al. [2024] which were extracted from the original paper [Kaplan et al., 2020].
More Robust Approach to Fitting Compute-Optimal Laws. The first approach in Hoffmann et al. [2022] fits scaling law by plotting the loss against FLOPS for family of models with a range of parameter counts (but relatively consistent aspect ratio, see Figure 2) while varying dataset size, then fitting a line to the pareto-optimal architecture for each FLOP count (see Figure 3). Following Hoffmann et al. [2022], we refer to this as "Approach 1". As we use a constant learning rate, we can use all recorded validation losses to fit our law. Hoffmann et al. [2022] and Kaplan et al. [2020] select model shapes so densely that they have a npar-optimal architecture at each FLOP count. This works when all architectures lie in 1D space (parameterized by parameter count), as each model is optimal in some FLOP regime, and the lower envelope is densely populated. However, in two dimensional exploration (varying width and depth), some models are never optimal, and the ones that are do not densely populate the envelope. We therefore develop a novel fitting method to accommodate sampling strategies like ours that result in regions of lower data density.
The Convex Hull. We fit a lower convex hull to our loss curves. This hull is only supported by sparse set of optimal models. Fitting on only the vertices of this hull naturally excludes sub-optimal models that lie above the convex hull of optimality, and as we show in Section 4, this makes the resulting scaling law far more robust to model selection choices. We provide mathematical definition of our approach in Appendix D.
Our New Method:
Why We Skip Approach 2. Another method to fit scaling laws is to put model runs into isoFLOP bins and choose the best parameter count in each bin. Hoffmann et al. [2022] call this "Approach 2 Our 2-dimensional set of models do not finely cluster into isoFLOP bins, meaning our data is not easily amenable to Approach 2, hence we exclude this approach from our analysis. Hu et al. [2024] and Li et al. [2024c] also eschew this approach.

---

Val Loss
Val Loss
3-
10
10
10
10
10
10
10
10
FLOPs
GPU hours
Tokens/Param PPans/
10
Tokens Tokens/Param Param
10"
10
10
10
10
10
10"
10
10
10
FLOPs
GPU hours
Convex Hull
Ours
Chinchilla Law
Binning
Kaplan Law
Figure 3: Approach 1 prescriptions. Row one: Validation loss over FLOPs (left) and GPU hours (right) for the first 100 billion tokens of training. We use Approach to find the optimal points on the convex hull in each setting, marked with black crosses. Row two: We fit a line the tokens per parameter of empirically optimal models and find a slightly higher, but still constant, tokens per parameter prescription than Hoffmann al. [2022]. Hoffmann al. [2022]'s Approach creates 250 seatttmmiiall-s-spcce FLOPs bins per order of magnitude, and in red we plot the minimizers over these bins, and the scaling law fitted to these minimizers (binning). Clearly, their Approach is not well-suited for our data, and our convex hull approach is better when we select fewer models to fit our law on. Extended plot Figure 20.
Prescribing Optimal Tokens Per Parameter by Fitting Power Laws. The final approach described by Hoffmann al. [2022] is to parametric formula the loss values with the ansatz
B T
L(p, (p,
+
(I)
pa
where p is parameter count and T is tokens. We our models using L-BGFS [Liu and Nocedal, 1989] with Huber loss (ō = 10-2) between the empirical log loss and the model prediction, and use multiple initializations following Besiroglu et al. [2024]. We ablate to check that our fitting procedure robust to the size of the grid of initializations and the choice of delta in Appendix L.4.
4 Experiments
In Section 4.1, we use our new convex hull fitting method to make a scaling law for the compute- optimal pokens-to-paramette
ratio, and compare this to the prescription from our fitted power laws. We show that many design choices such as the learning rate schedule can significantly impact these prescribed scaling laws in Section 4.2. In Section 4.3, analyze the Gemstones loss values multiple datasets and connect our analysis to benchmarks. Finally, we perform an analysis over time taken train instead of over FLOPs in Section 4.4.
4.1 Sizing Up Our Scaling Laws Against Prior Laws and Industry Models
Approach 1. In Figure (row one), we see our validation losses plotted as both a function FLOPs (left) and GPU hours (right) for the first 100 billion tokens of training. We calculate GPU hours from the average recorded optimizer step time for each model.
Our convex hull fits the data better than prior approaches. Hoffmann et al. [2022]'s Approach FLOPs bins per order of magnitude and then uses the models that achieve the best loss in each FL.OPs bin to fit the scaling law (a line). However, for our data, their approach does not work very well because it includes many points that are strictly suboptimal with respect to the minimal loss envelope. Our convex hull method omits these points, and fits the line
creates 250 oggrithmicallyyssaees
5

---

data Table 1:
We demonstrate the variability in fitting scaling laws by resampling many different ways. The slope can be viewed as the exponent in the power law relationship parameters constant compute embeddings, in the final column 'Delta' we show the change in slope produced by the ablations against the corresponding base law fit the full of hot data. Values with an absolute magnitude greater than 0.05 are highlighted in orange, and those exceeding 0.1 are highlighted in red. We see that the reduced sampling has a large impact on the slope of the law and that Approach is more sensitive than Approach 3. We plot these prescriptions in Figure 14 and show this table with embeddings excluded from the parameter count in Table 2.
Grouping by fitting approach and choice include
Tokens
Cooldown
LR Ablation
Embeddings
Slope
Delta
Hoffmann et al. [2022]
0.5126
Approach
(w/ Embeds)
X
0.4579
1005
X
X
0,4994
0.0415
1206
X
X
0.7987
0.3408
all
X
V
0.5131
0.0552
all
X
0.5970
0.1391
Approach 3 (w/ Embeds)
X
X
0.6965
1006
X
x
0,6986
0.0021
1206
X
X
0,7515
0.0550
all
X
V
0,6740
-0.0225
all
X
0.6992
0.0027
Approach 3
Chinchilla Reduced Sampling
all
x
/
0.6328
-0.0636
all
x
X
0.6315
-0.0649
Hoffmann et al. [2022]
X
X
0.6123
0.0997
with far fewer points. The asymptotic flatness of power law curves makes trying to fit scaling law an ill-conditioned optimization problem. Our novel convex hull approach is specifically crafted to reduce this variance and our results suggest that when optimal points are sparse, our approach can be used to obtain a more reliable fit (red vs black crosses Figure 3)
In Figure 3 (row two), we present the prescriptions from our scaling laws for tokens per parameter. We see that the tokens per parameter prescription of our Approach fitting close to constant, as in Hoffmann et al. [2022]. but suggests slightly larger optimal tokens per parameter ratio than their law. We extend this plot showing predicted total parameters, tokens, and over multiple ablations in Appendix L. and we give more detailed plot of each model's 's individual validation loss in Appendix K. In Appendix F, we show leave-one-out analysis over models when fitting both Approach and
4.2 Fragility and Pitfalls of Scaling Laws
To demonstrate the sensitivity of scaling laws to design choices, we fit laws with various assumptions and model selection rules. To provide compute-optimat parameter count prescriptions, tion 4 from Hoffmann et al. [2022], which we restate in Equation (6) for the convenience of the reader.
equa-
In Table we show the optimal predictions of multiple possible laws fitted on different subsets of our data. The "slope" column can be viewed as the exponent in the power law relationship between compute and parameters. In the final column "Delta", we show the change in slope produced by the ablations against the corresponding base law on the full set of hot data, grouping by fitting approach and choice include embeddings. We also plot these prescriptions with a FLOPs x-axis in Figure 14.
One particular dimension of variability we wish to highlight briefly here is the interplay between model selection and the derived law. To do this, we select 5 models from Gemstones that have an analogous model in Hoffmann et al. [2022] (using data extracted by Besiroglu et al. [2024]) with similar parameter count and aspect ratio and then we select Gemstones checkpoints with token counts nearly matching the Hoffmann points. We call this "Chinchilla Reduced Sampling" and

---

fit scaling laws to both of these sub-sampled datasets. We find that fitting Hoffmann's 's data using this reduced sampling results in an increased slope relative to fitting on all data. Meanwhile this subsampling reduces the slope of the line fit on Gemstones." This highlights that scaling law fitting can be quite sensitive to seemingly innocuous changes in model selection for both the variable aspect ratio Gemstones models as well as the simpler model family selected by Hoffman.
Between Table and Table we present the complete results from our series of ablations. Table I shows the results of fitting laws while including embedding parameter count, which both Pearce and Song [2024] and Porian et al. [2024] find to be primary explanation of the discrepancies between the preseriptions found by Kaplan et al. [2020] and Hoffmann et al. [2022]. Then in Table 2 we report results when not including the embedding parameter count. We also show the impact of fitting on our cooldown and learning rate ablation datasets in turn, seeing that both choices have a noticeable impact on the prescription for optimal parameter count. Finally, we remove checkpoints from our data to simulate having only trained for 100 billion tokens or only having data for token counts greater than 120 billion, seeing a greater impact than when fitting on our ablation data.
4.3 Modeling Performance on Different Validation Sets and Downstream Benchmarks
In Figure 4 we plot the loss of the 2 500M Gem- stone models on Dolma, Fine Fine Edu and DCLM-baseline data [Soldaini et al., 2024, Penedo et al., 2024, Li al., 2024b]. We see that when varying the data distribution on which we compute validation loss, although the loss changes for the Gemstones, it is equivalent to y-axis shift from Dolma; the relative order- ing of models remains unchanged. Of particular note is the fact that the deeper models consis- tently provide a lower loss.
2.9
2.8-
Loss
2.7
2.5
Next, following Penedo al. [2024], benchmark our Gemstone models on MMLU (Hendrycks al., 2020], WinoGrande [Sak- aguchi et al., 2021], OpenBook QA [Mihaylov al., 2018 8J, ARC [Clark et al., 2018], Com- monSense QA [Talmor et al., 2018,, PIQA [Bisk et al., 2020], SIQA [Sap et al., 2019] and Hel- laSwag [Zellers et al., 2019]. Specifically, we benchmark the models at 10 billion token inter- vals during training. We show the benchmark accuracy at selected token counts in Figure 8.
x 10
10"
2x 10
xx 10 4x 10"
Total FLOPs (log scale)
FineWeb- Edu
DCLM
FineWeb
Doima
1024x28
-
1536650
2048x27
3072x12
1280x15
1792x18
2560kl
768845
1280x36
1792x7
Figure 4: Loss over multiple webtext datasets. We see that the loss value changes for differ- ent datasets, including Dolma which we train on. DCLM and Wen have higher loss values whereas we measure lower loss values on Webe Edu and Dolma. However, the rank order between models is stable across datasets. This suggests that it may be valid to fit scaling laws on various vali- dation sets without necessarily needing to retrain the underlying models regardless of whether the validation data is i.i.d. with respect to the training distribution.
Predicting Benchmark Error. We follow Gadre al. [2024], predicting downstream av- erage top-1 error (Err) across our benchmarks using the recorded validation loss (L), using a function of the form shown in Equation (2) where e, k y are fit. In Figure 5, we fit a law to benchmark results sampled at every 10 billion training tokens, we see that our fitted curve fits the data well. We observe greater variation around the fit compared to Gadre et al. [2024], which we attribute to the considerable differences in the width and depth of the Gemstones models.
Predicting Benchmark Accuracy. Following Bhagia et al. [2024], we calculate task loss by taking the loss over the correct answer to each benchmark question and averaging over all questions. We then use this task loss to predict average task accuracy across 4 downstream benchmarks. We find the accuracy of ARC, HellaSwag and MMLU to be most predictable at smaller compute scales and use this subset of benchmarks when fitting scaling laws to predict accuracy. Concurrently, Magnusson
We note that there are 5 models in this subset for both Hoffmann et al. [2022] and our data, which meets the rule of thumb given by Choshen et al. [2024] for the minimum number of models that should be used to fit a scaling law.
7

---

et al. [2025] also observe this pattern across the same set of benchmarks. We predict average task accuracy by fitting sigmoidal function of the form shown in Equation (3) where Figure 6, we fit a law to benchmark results sampled at every 10 billion training tokens. We see noisy fit and again suspect this is due to the variation in the Gemstones' width and depth. In Appendix E, we hold out the 2 billion parameter models and show extrapolation for both benchmark scaling laws and Approach 3 loss predictions.
fit. In
0.65
Averuge Top-1 Top-1 Ervo
0.45
0.60
Accurcy
0.40
0.55
0.35
0.50
3.6
3.4
3.2
3.0
2.8
2.6
6.0
5.5
5.0
4.5
4.0
3.5
3.0
Loss
Loss
Data
Fisted Curve
Data
Fitted Curve
Benchmark Scaling Law for Error. We fit a law of the form shown in Equation (2) benchmark results sampled at every 10 billion tokens and observe tight fit.
Figure 5:
Figure 6: Benchmark Scaling Law for Accu-
racy. We fit a law of the form shown in Equa- tion (3) to benchmark results sampled at every 10 billion tokens for ARC, HellaSwag and MMLU,
a
Err(L) = - k- exxp(-7X)
Acc(L) = 1 h-L- +
(3)
4.4 The Width/Depth vs. Compute/Time Continuum
In the previous section show how deep models appear' achieve better final loss and accuracy on benchmarks when measured in terms of FLOPs. However, another crucial axis for practitioners to consider is the amount wall time it takes train model. Figure 7, we visualize in more detail the top of Figure 3, highlighting in color the approximate aspect ratio of the vertices that form our convex hull when fitting. On the left, we see that deep models are able achieve lower loss for given computational budget (FLOPs) and therefore are selected as the vertices of our convex hull when fitting. However, on the right, we see that when the budget is measured in units of computer time (GPU hours), wider models become more pareto optimal. The concept of "overtraining" is an interesting dimension for further analysis, especially while varying width and depth, as one may be
Loss
3.0
10
10
10"
10"
10
10"
10
10"
FLOPs
GPU hours
Width/Depth Ratio
3.2-17.1 (Deep)
85.3-320.0 (Wide)
17.1-36.6
36.6-85.3
Convex Hull
Figure 7: Approach 1 fitting. We enlarge and extend Figure (top), highlighting in color the approximate aspect ratio of the vertices that form our convex hull over all 350 billion tokens of training. We see that wider models achieve lower loss quicker in terms of GPU Hours (right) as the vertices of the convex hull are darker color. However, deeper models (lighter color) achieve lower loss quicker in terms of FLOPs (left).

---

able to overtrain a smaller deep model to reach a low loss value quicker than a larger wider model. We analyze overtraining in greater detail in Appendix J.
We remark that while in Figure 7 the optimal models with respect to time tend to be the wider ones, this is probably due our training scheme. Similar to other open-source efforts such as OLMo et al. [2024], we do not make any use of pipeline parallelism, and only employ tensor parallelism (using hybrid data and tensor parallel algorithm similar the ubiquitous Fully Sharded Data Parallel strategy), In summary, for standard parallelism implementations. wider models are simply easier to scale, but as a result our observations regarding resource overspending may not generalize to other parallelism strategies. As we open source all artifacts, practitioners can efficiently transform our open source results to suit their training setup. By simply running each model shape for only handful of steps, recording the step times, updating the step time column in our fitting data and refitting the laws. This means that practitioners can easily transform our GPU Hours analysis to their specific hardware.
Buried Treasure: Unearthing Value in Depth Finally, we plot the average benchmark accuracy (length normalized) of the Gemstones at 200, 250, 300 and 350 billion tokens. Figure 8 shows that the IB scale models (1280 x 36, 2560 x 8, and 1792 x 18) yield increasing accuracy with depth when constrained to approximately the same FLOP budget (vertically aligned points). We see similar patterns with the 768 x 45, 1280 15, 1792 x 7, and 1024 x 28 models, as well as the larger 2B models. This result is hinted at Table of the Gemma 2 report [TTamm et al., 2024b], where the authors note that for two models the 9B scale the deeper model slightly outperforms the wider model across downstream benchmarks, but details of the exact experiment are sparse. Recent work suggests deeper layers in networks "do less" than shallower ones and can be pruned away [Gromov et al., 2024], but our downstream evaluations suggest that there are also advantages to additional model depth. We see similar patterns in the individual performance each benchmark, and include those charts in Appendix Figure 16.
0.50
Model Shape
Accuracy (Nommllizdd
256x23
256x80
1024x28
0.45
256x27
512x14
1280x36
384x13
512x16
2560x8
500M
512x11
384x36
1792x18
0.40
256x71
768x45
2048x27
512x12
1250x15
1536x50
<50M
512x13
1792K7
3072x12
0.35
768x3
10"
10
FLOPs
Figure 8: Benchmark Performance. We benchmark all models using the 200, 250, 300 and 350 billion token checkpoints. Models show increasing accuracy with depth when constrained to approximately the same FLOP budget (vertically aligned points). This relationship between depth and accuracy can also be observed in many individual benchmarks (Figure 16).
5 Limitations and Conclusions
Altogether, experiments and analysis demonstrate the impact of often overlooked design choices on scaling law outcomes, the importance of measuring the right type of performance metric, and the nuanced relationship between model width, model depth, computational budget, and training time. We hope this work encourages a rich range of future work based on the suite of open source artifacts we release. Potential avenues for extension include exploring hyperparameters that we kept constant such as the expansion factor of the transformer (the ratio by which the dimensionality of the hidden layer in feed-forward network increased relative to its input dimension), the vocabulary size, the learning rate schedule, and the batch size. Although we endeavor to make our laws as generalizable as possible, we still expect that their applicability declines in training set-ups very different from own.

---

Acknowledgments and Disclosure of Funding
We thank (in alphabetical order): Brian Bartoldson, Bhavya Kailkhura, Avi Schwarzschild, Sachin Shah and Abhimanyu Hans for helpful feedback.
An award for computer time was provided by the U.S. Department of Energy's (DOE) Innovative and Novel Computational Impact on Theory and Experiment (INCITE) Program. This research used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. AF--AC00000022222
This work was made possible by the ONR MURI program, DAPRA TIAMAT, the National Science Foundation (11-22122122,, and the NSF TRAILS Institute (2229885). Commercial support was provided by Capital One Bank, the Amazon Research Award program, and Open Philanthropy.
References
Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer. Scaling laws for generative mixed-modal language models. In International Conference on Machine Learning, pages 265-279.
PMLR, 2023.
Lightning AI. Litgpt.
2023.
Joshua Ainslie, James Lee-Thorp. Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.33245, 2023.
Ibrahim M Alabdulmohsin, Xiaohua Zhai, Alexander Kolesnikov, and Lucas Beyer. Getting vit in shape: Scaling laws for compute-optimal model design. Advances in Neural Information Processing Systems, 36, 2024.
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.3, knowledge capacity scaling laws. arXiv preprint arXiv:2404.05405, 2024.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Yamini Bansal, Behrooz Ghorbani, Ankush Garg, Biao Zhang, Colin Cherry, Behnam Neyshabur, and Orhan Firat. Data scaling laws in nmt: The effect of noise and architecture. In International Conference Machine Learning, pages 1466-1482. PMLR, 2022.
Tamay Besiroglu, Ege Erdil, Matthew Barnett, and Josh You. Chinchilla scaling: A replication attempt. arXiv preprint arXiv:2404. 10102, 2024.
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang. Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for transfer. arXiv preprint vrXiv:2407.07726, 2024.
Akshita Bhagia, Jiacheng Liu, Alexander Wettig, David Heineman, Oyvind Tafjord, Ananya Harsh Jha, Luca Soldaini, Noah A Smith, Dirk Groeneveld, Pang Wei Koh, al. Establishing task scaling laws via compute-efficient model ladders. arXiv preprint arXiv:2412.04403 2024.
Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, al. Deepseek I1m: Scaling open-source language models with longternism. arXiv preprini (ale::20000229555 2024.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference Machine Learning, pages 2397-2430. PMLR, 2023.
10

---

Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense natural language. Proceedings of the AAAI conference on artificial intelligence. volume 34, pages 7432-7439, 2020.
Johan Bjorck, Alon Benhaim, Vishrav Chaudhary, Furu Wei, and Xia Song. Scaling optimal Ir across token horizons. arXiv preprint arXiv:2409. 19913, 2024.
Blake Bordelon. Lorenzo Noci, Mufan Bill Li, Boris Hanin, and Cengiz Pehlevan. Depthwise hyper- parameter transfer in residual networks: Dynamics and scaling limit. In The Twelfth International Conference on Learning Representations, 2024. URL https: / http:///ppeeeeeeeeeeeeeeeeeeeeee net/forummid KZJehvRKGD.
Jason Ross Brown, Yiren Zhao, Ili Shumailov, and Robert D Mullins. Wide attention is the way forward for transformers? arXiv preprint arXiv:2210. 00640, 2022.
Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger. Broken neural scaling laws. In The Eleventh International Conference on Learning Representations, 2023. URL https://
openreview.net//oooee v=sck
Leshem Choshen, Yang Zhang. and Jacob Andreas. A hitchhiker's guide to scaling law estimation, 2024. URL
org/ abs//211/ 11840.
Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. In International conference on machine learning, pages 4057-4086. PMLR, 2022.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 1003:v/800005555 2018.
Nolan Dey, Gurpreet Gosal, Hemant Khachane, William Marshall, Ribhu Pathria, Marvin Tom, Joel Hestmess, et al. Cerebras-gpt. gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster. arXiv preprint 23t.::20300032200 2023.
Nolan Dey, Quentin Anthony, and Joel Hestness. mal update parameterization, September 2024. URL https://www. www.tttiiiionrsuuieee-eeeeeee thtttttttteeeeeeee-eeeee tte-maaiimaa www.maaaaaaaateaeeeaaaaaaaaeeeea -pw.ate-pparmmttriiaattttt
The practitioner's guide to the maxi-
ai/blog/
Nolan Dey, Bin Claire Zhang, Lorenzo Noci, Mufan Li, Blake Bordelon, Shane Bergsma, Cengiz
Pehlevan, Boris Hanin, and Joel Hestness. Don't 't be lazy: Completep enables compute-efficient deep transformers, 2025. URL https: https: //ar/iv////ii////aa www./rgg//////eo/////..//.eeeee/ abs/ 201...0............111111.111111
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21780 2024.
Katie E Everett, Lechao Xiao, Mitchell Wortsman, Alexander A Alemi, Roman Novak, Peter J Liu, Izzeddin Gur, Jascha Sohl-Dickstein, Leslie Pack Kaelbling, Jaehoon Lee, and Jeffrey Pennington. Scaling exponents across parameterizations and optimizers. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 12666-12700. PMLR, 21-27 Jul 2024. URL https: //proceedings. mlr. prwss/ v23ps//23//////// everett24a. html.
Elias Frantar, Carlos Riquelme, Neil Houlsby, Dan Alistarh, and Utku Evci. Scaling laws for sparsely-connected foundation models. arXiv preprint arXiv:2309.0853 2023.
Samir Yitzhak Gadre, Georgios Smyriis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, et al. Language models scale reliably with over-training and on downstream tasks. arXiv preprint arXiv: 403.v22403.085400 2024.
11

---

Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020. URL https: //arxiv...vvv org/ abs/ 2101.00027
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. URL htlpttts:
wttgrecorrs//eaooooooooooooorrrt 12600602..
Behrooz Ghorbani, Orhan Firat, Markus Freitag. Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. Scaling laws for neural machine translation. arXiv preprint (09::200 2021.
Mitchell A Gordon, Kevin Duh, and Jared Kaplan. Data and parameter scaling laws for neural machine translation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5915 15225,92,21 2021.
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Hannaneh Hajishirzi. Olmo: Accelerating the science of language models. Preprint, 2024.
Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A Roberts. The unreasonable ineffectiveness of the deeper layers. arXiv preprint arXiv:2403. 178. 2024.
Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna Ben Allal, Leandro Von Werra, and Martin Jaggi. Scaling laws and compute-optimal training beyond fixed training durations. arXiv preprint arXiv:2405. 18392, 2024.
Soufiane Hayou and Greg Yang. Width and depth limits commute in residual networks. In Interna- tional Conference on Machine Learning, pages 12700-12723. PMLR, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint 0rXiv:2009.03300, 2020.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.1477 2020.
Danny Hermandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint arXiv: 102.0/22........ 2021.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. An empirical analysis of compute-optimal large language model training. Advances in Neural Information Processing Systems, 05:3006130000,, 2022.
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06399 2024.
Itay Inbar and Luke Sernau. Time matters: Scaling laws for any budget, June 2024. URL http: //arxiv. org/ abs/ 18922. arXiv:2406.198 18922 [cs].
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv: arXh::2001.0836/ 2020.
12

---

Jakub Krajewski, Jan Ludziejewski, Kamil Adamczewski, Maciej Pióro, Michat Krutul, Szymon Antoniak, Kamil Ciebiera, Krystian Król, Tomasz Odrzygóždž, Piotr Sankowski, et al. Scaling laws for fine-grained mixture of experts. arXiv preprint arXiv:2402.07871, 2024.
Yoav Levine, Noam Wies, Or Sharir, Hofit Bata, and Amnon Shashua. The depth-wiig width interplay in self-attention. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 22640-22651. Curran As- sociates, Inc., 2020. URL StKLLttttp//pprcceeddiggg neurips cc/paper f fww.fflllpppeeeeee pttple//ppppee/22/// .df.
Bozhou Li, Hao Liang, Zimo Meng, and Wentao Zhang. Are bigger encoders always better in vision large models? arXiv preprint arXiv:2408. 00620, 2024a.
Jeffrey Li, Alex Fang. Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng-Yu Yu Hsieh, Dhruba Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang, Rulin Shao, Sarah Pratt, Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang. Khyathi Chandu, Thao Nguyen, Igor Vasiljevic, Sham Kakade, Shuran Song, Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke Zettlemoyer, Kyle Lo, Alaaeldin El-Nouby, Hadi Pouransari, Alexander Toshev, Stephanie Wang. Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alexandros G. Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar. Datacomp-Im: In search of the next generation of training sets for language models, 2024b.
Margaret Li, Sneha Kudugunta, and Luke Zettlemoyer. (mis)fitting scaling laws: survey of scaling law fitting techniques in deep learning. In The Thirteenth International Conference on Learning Representations, 2024c. URL https://openreviest wwwp://ttttotoonoooo/oo/oaaoea http:://iir.coviiiaa///////t///f
Zhengyang Liang, Hao He, Ceyuan Yang, and Bo Dai. Scaling laws for diffusion transformers. arXiv preprint arXhv:2410.08184 2024.
Dong C. Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical Programming, 45(1):503-528, 1989. doi: 10. 1007/BF0158911 16.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv: 1711 2017.
Ian Magmusson, Nguyen Tai, Ben Bogin, David Heineman, Jena D Hwang, Luca Soldaini, Akshita Bhagia, Jiacheng Liu, Dirk Groeneveld, Oyvind Tafjord, et al. Datadecide: How to predict best pretraining data with small experiments. arXiv preprint arXiv:2504.113: 2025.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02778 2018.
Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36:50358-500 50378, 2023.
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo. Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 olmo 2 furious. arXiv preprint arXiv:2501.0065 000656, 2024.
Tim Pearce and Jinyeop Song. Reconciling kaplan and chinchilla scaling laws, 2024. URL https: //arxiv. ceaziivorgg//eeee aabs/2466. 12907.
Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, al. The fineweh datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37:308 1008811-330 1-30849, 2024.
13

---

Jackson Petty, Sjoerd van Steenkiste, Ishita Dasgupta, Fei Sha, Dan Garrette and Tal Linzen. The impact of depth on compositional generalization in transformer language models, 2024. URL https://aant http://axxvv.rrrgg.. http//bs/2300 /2300. 19956.
Tomer Porian, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt, and Yair Carmon. Resolving discrepancies in compute-optimal scaling of language models. arXiv preprint orXiv:2406.19146 2024.
Yangjun Ruan, Chris J. Maddison, and Tatsunori Hashimoto. Observational scaling laws and the predictability of language model performance, 2024. URL https://arxiv.org: wrt.///////.........//// 10938.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9)999-106. 2021.
Maarten Sap. Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.09728. 2019.
Siddharth Singh and Abhinav Bhatele. Axonn: An asynchronous, sessage-driven parallel framework for extreme-scale deep learning. 2022 IEEE International Parallel and Distributed Processing Symposium (IPDPS), pages 606-616, 2022. doi: 10.1109/1PPS553621. 2022.00005...
Siddharth Singh, Prajwal Singhania, Aditya K. Ranjan, Zack Sating, and Abhinav Bhatele. A 4d hybrid algorithm to scale parallel training to thousands of gpus, 2024. URL https://arxiv. org/abaa /2305. 13525.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A. Smith, Hannaneh Hajishirzi, Iz Beltagy. Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research, arXiv preprini, 2024.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsensegat: A question answering challenge targeting commonsense knowledge. arXiv preprint
2018.
Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler. Scale efficiently: Insights from pretraining and finetuning transformers. In International Conference on Learning Representations, 2022. URL https: https:/ //tp://///////eeee/////
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifrre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024a.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at practical size. arXiv preprint 24xx:2408.00111 2024b.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini arXiv:2302.1 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b. 2023b.
14

---

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison. Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's 's transformers: State-of-the-ar natural language processing, 2020. URL https: : /tt////axiiv org/ abs/ 1910. .07711.
An Yang. Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang. Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024a. URL attps://arxivy org/ abs/ 2407 10671.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, al. Qwen2.5 technical report. arXiv preprint 151::2412.151155. 2024b.
Ge Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tuning large neural networks via zero-shot hyperpa- rameter transfer. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 17084-17097. Cur- Associates, Inc., 2021. URL https: https::/ neereeeeeediiinnenniissese.eee neurips wwtrriis.ccpappeee files/ paapees/ppppr//// 200/://f.4.8ftraaacc000000//2222 Pdde
Greg Yang and Edward J. Hu. Tensor programs iv: Feature learning in infinite-width neural networks. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 1 11727-11737. PMLR, 18-24 Jul 2021.
Greg Yang, Dingli Yu, Chen Zhu, and Soufiane Hayou. Tensor programs vi: Feature learning in infinite-depth neural networks. arXiv preprini arXiv:2310.02244, 2023..
Longfei Yun, Yonghao Zhuang, Yao Fu, Eric P Xing, and Hao Zhang. Toward inference-optimal mixture-of-expert large language models. arXiv preprint arXiv:2404.02852 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv: 1905.0780, 2019.
Biao Zhang, Behrooz Ghorbani, Ankur Bapna, Yong Cheng, Xavier Garcia, Jonathan Shen, and Orhan Firat. Examining scaling and transfer of language model architectures for machine translation. In International Conference on Machine Learning, pages 26176-26192. PMLR, 2022.
Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. When scaling meets finetuning: The effect of data, model and finetuning method. arXiv preprint arXiv:2402.17193 2024.
Jingwei Zuo, Maksim Velikanov, Ilyas Chahed, Younes Belkada, Dhia Eddine Rhayem, Guillaume Kunsch, Hakim Hacid, Hamza Yous, Brahim Farhat, Ibrahim Khadraoui, et al. Falcon-h1: A family of hybrid-hear language models redefining efficiency and performance. arXiv preprint arXiv:2507.22444 22448. 2025.
15

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In Section I, we claim three contributions of our work and link to the according figures which support these claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: In Section 5, we highlight some general limitations of our work. We give a comprehensive list of design decisions taken for the Gemstones Appendix A.1 and Section 3 for the reader.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
16

---

Answer:
[NA]
Justification:
There are no theorems.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We give comprehensive list of design decisions taken for the Gemstones in Appendix 1, Appendix K and Section 3 for the reader. In the supplementary material include the fitting code for all laws shown. We will also open source all training code, model checkpoints and metries logged during training, but these are too large for the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
17

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
We give access to all code required fit scaling laws in the supplementary material. Model checkpoints, training code and all logged metrics will also be open sourced in the future.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.em cc/ lttpi//ggiidd//odeeeeeeeee
wom.iisiiiiiiiyy.. for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: /Cooeeubdeeeeeeeeeeeee
http:puulllgguudee / issi onPoiccy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We give comprehensive list design decisions taken for the Gemstones in Appendix A. and Section 3 for the reader. We also deseribe the decisions take around fitting scaling laws in Section 3.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We include error bars for all test benchmark results, these are the standard егror as reported by the Eval-Harness [Gao et al., 2024].
Guidelines:
The answer NA means that the paper does not include experiments.
18

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
detail that MI250x GPUs to train the Gemstones. As with any model release this size, significant computational resources were consumed. We estimate small 350b token runs took approximately 250 node hours and large runs took approximately 2500 to 3500 node hours. Including ablations and hyperparameter transfer experiments give conservative estimate of 40 thousand node hours. Unfortunately due to system limitations experienced we were unable to fully utilize the GPU capacity leading to longer training times than theoretically expected. We also used small amount of compute to debug our code at large scale. We open source all artifacts so future practitioners do not have reuse resources for this purpose. Our fitting code (included in the supplementary material) can be highly parallelized and takes approximately 4 minutes in the worst case (approach 3) fit scaling law on 96 CPU threads. Guidelines:
Justification: In Section
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: / /tt.rpeuuu.ppeee http//pppbbt/tceccciiiieieeeeeea
Answer: [Yes]
Justification: We submit double blind paper in good faith. All data used open source and license agreements are strictly followed.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
19

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In Section 5, we discuss potential societal impacts of our work. By building upon prior research, our work contributes deeper understanding of these topics, which we believe adds value to the community. This understanding enables better assessment of both the beneficial applications and the possible risks associated with our methods.
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: In Section 5, we discuss potential societal impacts of our work. As there are already language models trained on Dolma data fully open sourced, we believe that this work does not carry any significant risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
20

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: train our models using Dolma [Soldaini et al., 2024] data. Our models are based on Llama [Dubey et al., 2024], Gemma [Team et al., 2024a] and Pythia [Biderman et al., 2023] models. All datasets used for evaluation are also open source.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We release code for fitting with documentation. The model checkpoints and raw metrics are too large for supplementary material but will be released with full documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots. if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Research was done using existing public datasets and benchmarks, no human subjects or crowdsourcing.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
21

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: No such research was conducted.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLMs were not used in such manner at most, to help format plots.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our L.M policy (https: : http::/p.....oooeeeeeeeee/////// for what should or should not be described.
22

---

A Model Implementation Details
All models have a head size of 128 because 256 is the maximum head dimension supported by the AMD implementation of Flash Attention we utilize and we constrain our search models with > 1 attention heads. We assume the simple convention of the Llama series where the head dimension always the embedding dimension divided by the number of heads, implying that the embedding dimension (width) must be divisible by 128. Following conventions from the Gemma suite, we constrain the head count be even to enable Grouped Query Attention [Ainslie al., 2023] with query to key ratio of 2 and we fix the intermediate size to be 4x the width of the model. We choose our vocabulary size to match the 50, 304 tokens in the Pythia tokenizer. While many of the architecture choices mirror those from Gemma, for simplicity we do not use logit softcapping nor do we tie the embedding and language modeling head weight matrices.
A.1 Optimal Learning Rates for Gemstones
Training models across diverse shapes and scales requires learning rates that ensure both stability and near-optimal performance. Suboptimal learning rates risk misrepresenting scaling laws, as they could conflate architectural preferences with hyperparameter sensitivity. For the Gemstone models-varying in width, depth, and size-we address this challenge through unified learning rate scaling rule and a parameter initialization scheme tailored for stability.
Unified Learning Rate Scaling Rule Existing scaling rules prescribe learning rates (Ir) as Iroee/ /width for width scaling or traase/ hoowe//deptt /dpphh for depth scaling. Since Gemstone models vary both dimensions, we propose hybrid rule: Irei = Irasee/ /wwi//hwdth (width x vdepth) This accounts for the compounding effect of gradient dynamics across width and depth, balancing update magnitudes during optimization.
Empirical Validation To validate Traase we stress-test four extreme model shapes: wide (64 layers, 768 width) and deep (128 layers, 512 width) at 100M and 2B parameter scales. Each is trained for 1A steps with batch size of 2048 and context length of 2048 (4.2B tokens). We sweep (reff from 10-4 to 5 x 10-2. As shown in Figure 9 (left), optimal frai varies widely across architectural shape. However, rescaling the x-axis by width /depth collapses all curves onto a shared trend, revealing Trase = 5 as the consistent optimum (right panel). This confirms our rule's 's efficacy for width-depth transfer.
Flaws in the Gemstones. While Uroase = achieves stable training for most models under the scheme described above, wider architectures (e.g.. 256 width-depth ratio) occasionally exhibit loss spikes nonetheless. Despite these instabilities, via rollbacks and minor modifications to the learning rates for the most extreme models, all models in the suite are trained to 350B tokens without divergence. We discuss these issues and our solutions further in Appendix K.2.
Ablation Study To assess sensitivity to Iraase replicate training for subset of models with Iroae = 2.5 (e.g. dividing Irai by 2). While losses are marginally higher, scaling law fits remain robust, suggesting our conclusions not artifacts of aggressive learning rates.
Scalable Parameter Initialization Rules. Finally, stable training across model shapes and scales
also requires model specific tweaks to parameter initialization Yang et al. [2021]. Following OLMo( I) [Groeneveld et al., 2024], we apply parameter initialization strategy intended to enable stable training and learning rate transfer across scales. We initialize all parameters truncated normal (μI = 0, 1/V 1/v2 width (1 + 1) and the MLP projections as a = 1/v2- (4 x width) (7+ 1) where in each case the layer index (not the total model depth) and the 4x factor comes from the relation of width to MLP intermediate dimension.
-3 o. b = 3- o)) with modified variances dependent the parameter type. We use except for the attention projections which are initialized
23

---

15
10
Loss
10
10
10
10"
10*
Irbase
Iraff
Width Depth
256 80
768
1536 50
3072 12
Figure 9: Learning rate scaling is necessary for width-depth transfer. Left: Preliminary training runs with initialization rules active, but no learning rate scaling. Right: Same data, but with x-axis rescaled to simulate the application of learning rate scaling with Traas lreft (width x /depth).
A.2 Software and Data
We train all models using a fork of litgpt [AL, 2023] enhanced with AxoNN [Singh and Bhatele, 2022, Singh et al., 2024] tensor parallelism. We open source all models used in our analysis to Hugging Face [Wolf et al., 2020] and the logging from training on Weights and Biases in json format.
B Extended Related Works
Scaling laws have been constructed in many different areas of machine learning since their original proposal in [Kaplan et al., 2020]. Early work on scaling laws for machine translation by Ghorbani al. [2021] splits the parameters term into two, one for each of encoder and decoder components. and similarly to Gordon et al. [202]] analyzes the relationship between BLEU scores and scaling laws. Subsequently, Zhang al. [2022] and Bansal et al. [2022] studied the impact of architecture choice on the scaling law, finding increasing data or parameters can compensate for worse architectural decisions. However, the advent of performant open source pipelines for language model development following the release of the Llama series [Touvron et al., 2023a] spurred renewed flurry of interest the topic in academic settings.
Architecture Allen-Zhi and Li [2024] builds scaling laws to model how specific architectural choices impact measures of knowledge acquisition bbits-per-param) in highly controlled settings; they run parallel experiments across dimensions like architecture, quantization, and sparsity to derive insights about which design choices affect acquisition and storage capacity the most. However the more general study of scaling laws for sparse architectures is quite extensive. Clark et al. [2022] analyze how the number of experts can be used in the law, studying both linear and quadratic interactions for many types of routing models. Frantar et al. [2023] focus on weight sparsity within foundation models, adding a multiplicative parameter on the parameters term in the law. Yun et al.
[2024] analyzes the trade offs between optimal training and optimal inference and Krajewski et al. [2024] find that with optimal settings, a Mixture of Experts model always outperforms a transformer model at any computational budget. Model shape has also been analyzed for sparse mixture of expert models and the context of finetuning. Krajewski et al. [2024] use the ratio between the standard feed-forward hidden dimension and the hidden dimension of an individual expert to allow their law for mixture of expert models to predict the width the experts.
Downstream Benchmarks Beyond modeling just training loss-the canonical prediction target for scaling laws-there are multiple works analyzing whether scaling laws can be used to predict performance downstream tasks. Ruan et al. [2024] show that scaling laws can be predictive of benchmark performance. Caballero et al. [2023] observe that traditional scaling laws cannot capture complex behaviors like non-monotonic trends nor inflection points. They propose broken scaling laws-a piecewise or smoothly broken power that better predicts performance of both upstream and downstream tasks.
Finetuning and Data-constrained Regimes analyzing finetuning and data limited scaling. Hernandez et al. [2021] find that finetuning is much
Further analyses using scaling laws have extended to
24

---

more compute efficient when the pretraining ignored. Zhang et al. [2024] study parameter efficient finetuning regimes find a multiplicative law better for the finetuning setting than the classical additive law used by others. Muennighoff et al. [2023] analyze the data constrained training regimes, finding epoching data up to four times is as good as training on deduplicated data in terms of reducing loss.
Multi-modality These techniques are not limited to generative text modeling only; they have also been applied to multi-model models. Henighan et al. [2020] find optimal model size can be described as power law for model modeling including images and video. The authors also find that model size does not help 'strong generalization' for problem solving. Aghajanyan et al. [2023] analyze text, images, code and speech, presenting scaling law describe the competition between these modalities and describe a regime for optimal hyperparameter transfer from the unimodal to multimodal regimes. Liang et al. [2024] look at scaling laws for diffusion transformer models. Li et al. [2024a] analyze scaling laws for vision encoder commonly used to encode image inputs for transformer model backbones, finding increasing the size of the encoder alone can lead to performance degradation in some cases.
Zero-shot Hyperparameter Transfer The ability to train a series of models with extremely different parameter counts is an implicit requirement of any scaling law analysis. Bjorck et al. [2024] find that optimal learning rates change with length. Work on zero-shot hyperparameter transfer across transformer model widths is mature [Yang et al., 2021, Everett et al., 2024, Hayou and Yang, 2023, Dey et al., 2024]. Achieving transfer across diverse model depths less well studied, especially in transformer language models [Bordelon et al., 2024, Yang and Hu, 2021, Yang et al., 2023].
While Yang et al. [2023] argue that depth transfer requires scaling in 1/vL because this is the unique regime with maximum feature diversity, more recently Dey et al. [2025] argue that instead the scaling should be in 1/L.
B.1 Chinchilla Equation 4
Hoffmann et al. [2022] use the variable names N and D for the number of parameters and number of tokens respectively, defining their parameterized form as:
A
B
i(NN, D) - F+
(4)
No
D3
Equation of Hoffmann et al. [2022] is defined as:
C
Nope(C) = G
Dope = G-1
(5)
aA
B
where
G
and h
(6)
BB
8
C Data Sampling
We plot the entire space of all possible models subject to our design constraints discussed in Figure 10. While exploring the impact finer grained depth differences during our experiments, we decided to add two additional models slightly outside the +5% tolerance band at the 100M scale; for width 512, in addition to the originally chosen depths of 12 and 13, we added 11 and 14; these appear as a dense collection of points at the same width.
D Mathematical Definition of Our Convex Hull Method
We give a mathematical definition for our convex hull method, loosely based on the wikipedia entry for reconstructing functions from epigraphs:
We define the set of points have to fit on as FLOPs/GPU hours (æ), loss value (L) pairs.
D ((z, M. C R²
25

---

Width
10'
10"
Depth
50M A 2M
1B 50M
7B a: 350M
100M 5M
2B 100M
Ours
500M 25M
4B 200M
Figure 10: All possible model shapes we could have chosen based on our architecture within 15% shown circles. The points selected highlighted as stars, including the two extra points we select to have four models of width 512.
Let cDnv(D) denote the convex hull, the linear interpolation of any two points
A 2 0. Σ A = }
conv(D) = Ar(o,, Li)
im
Define
i(x) = min(tvl(z, y) E D})
The lower convex hull is the graph of this new function:
{z, [x))]{z e Dom(L))
We think the easiest visualization of this is Figure where the red line is the convex hull, the colored crosses are the vertices and the gray lines are all possible points in the dataset.
E Extrapolation to Larger Models
To quantify the robustness of our scaling laws to changes in model scale, we perform two types of extrapolation analyses. the first experiment, we hold out the "2B" parameter models, fit scaling laws to the samller models, and then test the extrapolation performance of the fitted law. Due to the 10% tolerance margin in our parameter count strata, actuality we fit benchmark scaling laws using models with less than 1.8B parameters and extrapolate to models with more than 1.8B parameters. In Figure I we see that extrapolating to predict error offers a much better than we see in Figure 12 when predicting accuracy.
In second type of experiment, we again analyze extrapolation function of model size but this time quantifying prescription robustness in terms of estimated versus actual validation loss (using approach 3). Following Choshen al. [2024], we report the mean absolute relative error (ARE) over the last 30% of training tokens (2506 to 350b tokens for Gemstone models). We find the ARE when fitting on all checkpoints and then testing on models with more than 1.8/ parameters is 0.68%. When only fitting models with less than 1.8B parameters and extrapolating to models with more
26

---

0.65
Average Top-1 Ervor
0.45
0.60
Accuracy
0.40
0.55
0.35
0.50
3.6
3.4
3.2
3.0
2.8
2.6
6.0
5.5
5.0
4.5
4,0
3.5
3.0
Loss
1.8B Parameters
Fitted Curve
1.8B Porameters
Fitted Curve
1.8B Parameters
2 1.8B Parameters
Figure 11: Benchmark Scaling Law for Error. We fit a law of the form shown in Equation (2) benchmark results sampled at every 10 billion tokens using models with less than 1.8B param- eters and observe tight when extrapolating to models with more than 1.88 parameters.
Figure 12: Benchmark Scaling Law for Accu-
racy. We fit law of the form shown in Equa- tion (3) to benchmark results sampled at every 10 billion tokens for ARC, HellaSwag and MMLU using models with less than 1.8B parameters and observe poor fit when extrapolating to models with more than 1.8B parameters.
than 1.8B parameters, the ARE is 0.63%. Choshen et al. [2024] find ARE's up to 4% are typically used distinguish between modeling choices, implying that extrapolation error is well within the acceptable range.
F Leave-One-modelOuut Analysis
To evaluate how robust our scaling laws are to a different aspect our experimental design, we estimate the variability in our fitting process caused by our model selection using a leave-one-out analysis. In Figure we re-fit the same scaling law multiple times leaving each one of the model shapes out in turn using both Approach and Approach 3. We visualize these results by plotting the minimum and maximum tokens per parameter ratios yielded across all leave-one-out trials along with the prescription based on all data (bounds of the gray shaded region the green line). While the implications of the precise min/max values are somewhat up to interpretation, compared the difference between our fit, Kaplan's 's, and Chinchilla's s. the relative narrowness of the gray region tokens per parameter prescription we find versus prior work unlikely to simply be an artifact of (our) specific model selection. Finally, comparing the left and right sides of Figure 13, the smaller gray region the former suggests that Approach 1 is less sensitive to this model re- sampling process than Approach 3. We hypothesize that least some of this increased robustness fit can be attributed to our novel variance- variance-reducing convex hull method applied in Approach 1.
suggests that any disagreement
G Variability in fitting
In Table 2, we show a similar table Table but exclude embeddings from the parameter count in the fitting process. We also visualize all fitted lines in Figure 14.
G.1 Variability from sampling
In this section, we visualize the variability in the fitting process due to the frequency of sampling data. We do this by sampling fitting data every billion tokens training and every billion tokens of training and comparing the laws found. In Figure 15, comparing the "Ours" lines, we see that for Approach the difference minimal where as for Approach there a change in the law. This can be intuitively explained as the data the hull is much sparser the points the law is fitted to
27

---

Tokens per Parameter ratio
10"
Tokens per Parameter ratio
10'
10
109
10
10
10
10
10
10
10"
10" FLOPs
10
10"
FLOPs
Our
Kaplan Law
Ouas
Kaplan Law
LO0 variance
Chinchilla Law
LO0 variance
Chinchilla Law
Figure 13: Leave-One-Out Validation. We leave each model out and refit the law to obtain the gray shaded region (min/max ratio across all trials) and compare it with the law fit to all data in green. Left) shows the result when Approach used, and right) shows result using Approach 3.
Table 2:
We demonstrate the variability in fitting scaling laws by resampling many different ways. The slope can be viewed as the exponent in the power law relationship parameters constant compute-aponent Grouping by fitting approach and choice include embeddings, the final column 'Delta' we show the change in slope produced by the ablations against the corresponding base law fit the full set of hot data. Values with an absolute magnitude greater than 0.05 are highlighted in orange, and those exceeding 0. are highlighted in red. We that the reduced sampling has large impact the slope of the law and that Approach is more sensitive than Approach 3. We plot these prescriptions in Figure 14 and show this table with embeddings included in the parameter count Table 1.
data
Tokens
Cooldown
LR Ablation
Embeddings
Slope
Delta
Kaplan et al. [2020]
0.7300
Approach (no Embeds)
all
X
X
0.5689
1006
X
0.6269
0,0579
1206
X
X
0.9666
0.3977
all
x
0.6224
0.0535
all
X
0.7242 X
0.1552
Approach (no Embeds)
X
X
0.7141
100b
X
X
0.7030
-0.0111
1206
X
X
X
0.7350
0.0209
XXX
all
X
/
0.6929
-0.0211
0.7104
-0.0037
changed, hence for Approach more fitting data gives a more reliable fit. Hence, in all other plots in this paper for Approach I we fit with data recorded every 2 billion tokens for accuracy and for Approach we fit with data every billion tokens for speed.
Further, in Figure 15, we also present laws fitted on the DCLM and Febe Ediebb-Euudde data in Figure 4. For the DCLM and Feb- Web-Edd Edu data, we record data every 10 billion tokens of training. In Figure 15, we see that difference between the laws found by fitting on different data sets compared to our main analysis on Dolma minimal.
H Individual Benchmark Results
In Figure 16, we see zero-shot MMLUU scores our larger models are quite non-trivial at 28% 33%, despite being trained open dataset, without cooldown period, or any sort of post-training.
28

---

All Tokan, widB--11 Only
Apprench I Cooldown No Embels
Apprasc Tokens
Coollown
Approach < Token
o-1000 Tokets No Embedd
10
Approach Ablation
Cooldowa Embeds
Approach All Tokens No Embbela
<-0 Tokets
10
Approsc Carldown
Approsch 1-120h Tokens Only
Params
Chaschlia Reduced Sumpling Data)
Tokens
10
Approsch LR Ablation No Embeeds
All Tokens No Eithedss
Approsch oo Toker Embeds
Tokons Only Embedh
10"
LR Ablation Chischilla
>120b Tokens Only
Reduced Surplied (Our Data)
Agerosch >120b Tokens Only
10
Chinchilla Reduced Sampling
Embels
(Hallnnn al. dats)
Kaplan Law
LR Ablation
Chinhillaa Low
10
10"
10
10"
10"
No Embeds
FLOPs
Figure 14: We demonstrate the variability in fitting scaling laws by resampling our data many different ways. We label prescriptions found using Approach 1 with "Approach 1 in the legend, otherwise approach is used. All tokens counts available are used to fit the laws unless stated otherwise in the legend, for example I 100B means that only token counts less than or equal to 100B are used in fitting.
No Embeds: Embedding parameters are not counted when fitting these laws.
Cooldown: Only data from the cooldown ablation is used to fit this law. LR Ablation: Only data from the learning rate ablation training runs, where the learning rate is halved, is used to fit these laws.
512 Only: Only models with width 512 are used to fit these laws.
Chinchilla Reduced Sampling: We subsample our data to be as close as possible to the token counts and model sizes that Hoffmann et al. [2022] use fit their scaling laws and also fit new scaling laws on this subset of Hoffmann et al. [2022] data. Details in Section 4.2.
Ours 10b (Approach 3)
Tokens per Purameter
DCLM frog-10b (Approach 3)
10"
FineWeb-Fdu freg 10h (Approach 3)
Ours 2b (Appreach
DCLM lioq-2b (Approadh 1)
10-1
FineWeb-Edu freq (Approach 1)
Ours froq (Approach 1)
Ours fb (Appreach I)
10
Kaplan Law
Chinchilla Low
10
10
10
10 25
FLOPs
Figure 15: We fit laws when data is sampled every 2 and 10 billion tokens of training. We also compare to laws fit on DCLM and Web-FuuFddu data. We see sampling frequency important for Approach and that the difference laws fitted on FineWeb-Edu, DCLM, and Dolma is small.
I The Price of Stepping Off the Scaling Law
By analyzing the cost of stepping off of the scaling law, we find that some kinds of design errors are more damaging than others. We also find that training on more tokens than is strictly recommended (aka "overtraining") is typically quite efficient in terms of pushing down loss.
If You Value Your Time, Train Wide Models. We first show that in our training setup, training wider models far more efficient than training deep models. In Figure 17, we reflect on the consequences of suboptimal architectural choices, by considering how much of a given resource-FLOPs or GPU hours-would be "overspent" to reach any target loss value with the plotted architecture rather than the prescribed width and depth. We find that choosing to train "skinny" models (top left) wastes many FLOPs and GPU hours. The scale of overspend is quite different however, with the least efficient models only overspending about 50% on FLOPs but wasting more than 200% of the GPU hours spent by the best configuration. In other words, the time taken to train a single (very) suboptimal model to the desired loss value, one could train three optimal-width-aee while the time-optimal models tend to be the wider ones, this is probably due to our training scheme. Similar to other open-source efforts such OLMo et al. [2024], we do not make any use of pipeline
models. We note that
29

---

arc:challenge
arc:casy
commonsenscc
0.325
nomn
0.500
0.5
500 hom
Model Shape Random
256x23
0.250
256x27
512x11
0.200
256x71
10
10
10"
10
212x12
FLOPs
FLOPs
FLOPs
512x13
hellaswag
mmlu
openhookqu
768x3
256x30
0.6
0.350
512x14
512x16
0.325
800 500t
0.5
060 166ns
0.35
0.500
1250x15
1792x7
1024x29
0.250
0.3
0.25
2560s8
10
10
10
10
10"
1792x18
FLOPs
FLOPs
FLOUS
2048x27
1536x50
piqa
siga
winogrande
3072x12
0.75
0.12
0.56
0.40
0.54
0.65
0..38
520
0.52
0.50
0.55
0.34
10"
10
10
10
10"
FLOPs
FLOPs
FLOPS
Figure 16: Individual benchmark performance. We benchmark all models using the 200, 250, 300, and 350B token checkpoints, converting this to FLOPs using the formula shown in Appendix M. Models show increasing accuracy with depth when constrained to approximately the same FLOP budget (vertically aligned points) on many benchmarks. We plot the average benchmark accuracy in Figure 8
parallelism. In summary, for standard parallelism implementations, wider models are simply easier scale, but result our observations regarding resource overspending may not generalize to other parallelism strategies.
J Scaling Laws Predict That Overtraining Is Efficient.
Similarly to Gadre et al. [2024], we can shift optimal points to simulate overtraining. To do this, we fix FLOP budget and trace out path of model sizes and corresponding token counts to remain within that budget. For each model size and token count, we record the "overtraining factor," which the selected number of training tokens divided by the optimal number of tokens for that model shape. An overtraining factor of less than one corresponds to undertraining the model, and factor greater than one represents overtraining. We show the results of this process in Figure 18. We see that overtraining does increase predicted loss at a given FLOP count but that these curves are actually quite flat. We include the loss values open source models on our own validation set to allow readers to contextualize the y-axis values. Especially at high FL.OP counts, our laws predict overtraining becomes quite efficient that results fairly small elevations in loss for relatively large reduction in model size.
30

---

FL.OPs Overspend by Architecture
GPU Hours Overspend by Architecture
0m5
10
GPU Houns %/ Oracspen
8
FLOP Dvarppen
Depths
6
Depils
40
10
4
10
Point Fronticr
10" Width
10"
Width
Figure 17: The inefficiency of training models with suboptimal widths and depths. We plot the FLOPs (left) and GPU Hours (right) overspend after training our Gemstones for 300 billion tokens. define the overspend as how many resources (FLOPs or GPU hours) are required for a model with given width-depth configuration to reach some target loss, relative to the models that achieve that target loss the fastest (the "points on (pareto)-frooee frontterr" We can see that the skinny models (top-left, dark points) use many more FLOPs or GPU hours to reach target loss than the wide models. We note that these inefficiencies exist in our training setup because we only use tensor parallelism and not pipeline parallelism but highlight how easily transfer these results other environments in Section 4.4.
Industry models often use fewer parameters and train on more tokens than prescribed in prior work. We find the impact of overtraining smaller model on predicted loss to be small. Combining this with Figure 17, where wider models are predicted to optimal in terms GPU hours, reinforces the message that FLOPs optimality is not the end of the story for training models. Trading some FLOPs optimality for time optimality necessarily means overtraining, but Figure 18 suggests the difference is marginal. We believe this combined evidence makes significant progress towards explaining the differences between the prescriptions found in prior work and training choices observed industry.
3.05
3 3.00
Our Prokeed Low Qva1244.B
9215ke11
3
Qem2 5:B
ELOP: 1.42301+144
207-
3 3.66
Prodeeed
FLOPs - 99
131
Out Paddedddd
1330
19"
16
Oustniiing (lg)
Figure 18: Quantifying the cost of overtraining. We simulate deviations from our prescriptions to assess their impact model performance by increasing the optimal token count prescribed by overtraining factor. We then optimize the model shape to achieve the lowest loss possible at each FLOP budget and overtraining factor. Note that 10"h 1x, the prescribed optimal point. We take four FLOP budgets (title of each plot) and plot the loss as a function of overtraining factor and see that under or overtraining increases predicted loss but by only a small amount. We plot the losses of selected open source models on our validation set to help ground the y-axis ranges.
31

---

K
Training
Despite our best efforts to sufficiently mix the training data, still see slight jumps in the global training loss when the training switches between chunks of data, hence we use validation loss to fit all laws this is smooth.
K.1
Loss Curves
4.5
4.0
3.5
3.0
2.5
10
10
Tokens (log)
Width x Depth
1024x28
1792x7
256x71
384x36
512x14
1280x15
2048x27
256x80
512x11
512x16
1280x36
2560x8
3072x12
512x12
768x3
1536x50
256x23
384x13
512x13
768x45
1792x18
256x27
Figure 19: Loss curves for the main 22 training runs.
K.2 Additional Training Complications
Any gemstone naturally contains small number of inclusions fractures. We discuss few of the minor imperfections in our model collection below.
Dealing with Training Instabilities After some of the widest models were trained beyond 50B tokens we began to observe unrecoverable loss spikes that were proceeded by small wobbles in the loss trajectory. Under the general intuition that the culprit considered were simply too extreme for existing initialization and learning rate scaling approaches to handle,
most likely that the width/ /dephh ratios
some of the models with "patch" in place.
We modified the initialization rules and learning rate scaling factors to rescale the depth and layer indices of the model such that if width depth > 256 scale variances and learning rates as if the depth of the model was actually depth' = initialize and scale learning rates more conservatively, if the aspect ratio were only 100 while keeping the original width of the model. We found this allowed us to complete training for a full set of 22 models out to 350B tokens for even our most extreme models.
Srridth110 [wiithh1100], The overall effect of the patch is to
However, after 350B tokens, despite these efforts we observed that most extreme models which were patched still diverged anyway. While partial cause of this could be the constant learning rate scheduler employed during training, concurrent work, from the authors of the original OLMo paper and codebase [Groeneveld et al., 2024] from which we derived some of our choices, reported that the initialization scheme dubbed the "Mitchell-init" indeed systematically prone to instabilities later on in training [OLMo et al., 2024]. While unfortunate finding, were unable to rerun all of our experiments due to the consumption of significant non-fungible compute resources in the original experiments.
Models Lacking Ablations Our cooldown ablation is from initial experiments below 100B tokens of training which do not use the patched learning rates scaling rules. This means there are minor
32

---

Val Loss
Val Loss
10
10
10
10
10
10
10
FLOPss
GPU hours
Parameturs
Paramelers
10
10
10a
10
10
10
10
10"
10
10
10
10"
FLOPs
GPU hours
10"
10a
Tokems
Tokens
10
10
10
10
10"
10"
10
10
10
10"
FLOPs
Tokeno TokemoParram
10
Tokems ParmmPaaam
10-
10a
10a
10
10"
10
10
10
10"
10
Convex Hull
Kaplan Law
Chinchilla
Figure 20: Extended Approach I plot from Figure 3, including tokens and parameters axes. As in Figure 3, we present an analysis over FLOPs on the left and over GPU hours take to train on the right.
discrepancies between the cooldown ablation and main set of training runs for the widest models from the three largest parameter count groups (1792 x 2560 x 8, 3072 12). We also do not cool down the 100B token checkpoint for the 3072 x 12 model, as it was experiencing loss spike at that final point. Finally, we do not include ablations for the two width 512 models which do not fall into the +5% boundary of the 100M parameter count (512 11, 512 x 14), they were only added to the collection in later experiments.
L Ablations for Approach 1
L.1 Extended Paper Figures
In Figure 20, we plot extended version of the Approach plot present in Figure 3.
L.2
Alternative Learning Rates
In Figure 21, we present the Approach 1 prescription when fitting on the learning rate ablation data.
L.3
Cooldown
In Figure 22, we present the Approach prescription when fitting on the cooldown ablation data.
L.4
Varying Delta in the Huber loss
In Figure 23, where we plot the exponents found by optimizing the Huber loss versus the size of the grid search used for optimization. We see that delta of 10-5 is unstable for smaller grid sizes and
33

---

Val Loss
Val Loss
10"
10"
PL.0Ps
GPU hours
Parumetens
10
Parumetens
10
10
10
10
10
10"
FLOPs
GPU hours
10"
10
Toketts
Takens
10
10
10
PL.0Ps
GPU hours
Thkens ThkeenPParam Param
Tokens Param
10-
10
To
10
toa
10"
10"
FLOPs
GPU hours
Convex Hull
Ours
Kaplan Law
Chinchilla Law
Figure 21: Approach 1 fitted on the learning rate ablation dataset. As in Figure 3, we present an analysis over FLOPSs on the left and over GPU hours take to train on the right.
3.5
3.5
2.5
FLOPs
GPU bours
10
10
Purumeters
Purumeters
10'
10
10a
10a
10
FLOPs
GPU bours
10"
10
Tokens
Tokens
10
10
FLOPs
GPU bouns
TokensParam
10
PoramsParam
10a
to
10'
1oa
10"
10
PL.PPs
GPU hours
Convex Hull
Ours
Kaplan Law
Chinchilla Law
Figure 22: Approach fitted on the cooldown ablation dataset. As in Figure 3, we present an analysis over FLOPs on the left and over GPU hours take to train on the right.
34

---

Scaling Law vs Fitted Tokens Grid Size
Scaling Law Fitted Tokens Grid Size
Exp (Paass Eap- Exp)
Pamine Exp (P (Panns Eap TTkees txpp
0.34
0.32
0.30
0.28
20000
40000
60000
80000
20000
40000
60000
80000
Grd
0.001
0.9001
Le-05
0.001
0.0001
Le-05
Figure 23: We plot the size of the grid search as the axis and the gradient of the prescribed tokens as the axis. We vary delta and see that a delta of 10 10-5pp is highly unstable when fitting on smaller grid sizes. On the left, plot only fitting data less than 100 billion tokens. On the right, we plot fitting on all data up to 350 billion tokens. We see that including more data increases the stability of the exponents found for smaller grid sizes for deltas 10 40-4,1 10
including more tokens in the fitting data generally increases stability of the exponents found during optimization.
M FLOP counting matters
In Figure 24 we show that the common approximation of FLOPs per token= 6 x parameters, miscounts the true FLOPS by a significant amount.
80
1.5
60
1.3
Depth
40
1.1
1.0
20
Our
-0.8
1000
2000
3000
Width
Figure 24: We color the points based on the ratio of our calculated FLOPs per token which is shown in the code below and using 6 x parameters. We see counting the FLOPs properly becomes more important for aspect ratios off outside of the standard regime.
VOCAB_OUR 50304
SEQ_LEN = 2048
WORLD WORLD_BATCH. B____ACC___II__ 2048.0
HEAD_ SIZE 128
EXPAND_FACTON 4.0
def Zlops_per_token_ee token_ggal
width: NDArray [number] depth: NDArray [number] vocab_stt Beabb_siee-VDABBORRE
number,
number,
35

---

queries_por-group 2, seq_len-SEq_LE.
):
aun
Some details (negligible even for extremely wide models) omi omitted, including: numerically stable softmar softmaz addition only being rows
dot products being only n-1 additions (fused multiply by add exists anyway)
queads width / HEAD_SIZE
num num_ kvhadd
2 num_ qheads /
embeddings 0 0 if sparse lookup, backward FLOPs negligible
attention 2.0 seq_len (num_qheads num_kvheads) width HEAD_SIZE attention += (
3.5 seq_len (num qumm_hheads num komeas
READ_SIZE
) # RoPE, as imp lementeed here/GPT-Neod
score FLOPs are halved because causal triangular mask usable sparsity
kg loglogits = 1.0 * seq_len * seq_len * HEAD_SIZE * num_qheads
softmax 3.0 seq_len seq_len num_qheads
softmax_q_red = 2.0 * seq_len seq_len HEAD_St SIZE * num qheads
final_linear 2.0 seq_len width HEAD_SIZE num_qheade
attn_bwd =
2.0 attention
+ 2.5 * (kq_logite + softmax + softmax_q_red)
2.0 Linal_lines
) depth attention += kqglog + softmax + softmax_q-red final linear
ffw_size EXPAND_FACTI
width
dense_block block
6.0 seq_len width * ffw_size
three matmuls instead of usual two because of GEGLU dense block +=
10 seq_len ffw_size
) # 7 for other ops: for cubic, two additions, two scaler mults
block 2.0 width seq_len both/sanduich residual additions
rmsnorm 2 7.0 width seq_len
final_rms norm 7.0 width seq_len # one last RMSNorm
final_logits 2.0 seq_len width vocab_size
nonattn_bud 2.0
embeddings depth blense_bock rmenorm) sinal_rms_nen final_logits
forward_pass embeddings
+ depth (attention dense_block + rmsnorm)
final_rms
final_logits
)
backward_pass attn_bud nonattn_bod # flash attention
return (forward_pass backward_pass) / seq_len
36


Reviews:
[{"review_id": "vj08L1tlnj", "reviewer": "Reviewer_shdg", "summary": "This paper studies scaling laws for pre-training while going beyond narrow hyper-parameter choices and varying both depth and width. They futhur adapt the typical formulation of scaling laws to be more robust to biases and sparse data regions in the fitting data. The authors Intermidairy checkpoints are also released at frequent intervals to aid in further open-science study of these laws.", "questions": "- Since intermidiary checkpoints are also used in the fitting, does this bias the laws? how sensitive are the laws with respect to the frequency of the included checkpoints? \n- If you were able given resource and time-constraints, etc. to ablate another axis, which of the ones listed in limitations would you priotize and why? I think it'd would be an appreciated piece of information if included in the limitations section with more detail.", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n- The paper is genrally well-written \n- The experimental depth of the work requires singificant compute and resources which is not widely accessible to most researchers. The open-sourcing of the many model checkpoints will be greatly valuable to the community. \n- The paper covers both different optimization, and architectural choices in deriving the laws. \n\nWeaknesses: \n- The work would have benefited from having a larger scale of models, exclusively reserved for testing laws derived at a smaller scale. My understanding is that all datapoints up to 2B scale are used in fitting the loss, correct? It would be good to test the laws on a held out model size using the optimal ratios dervied from the fittings.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful comments and their acknowledgement of the multi-faceted contributions that the Gemstone model suite will make to the scientific community.\n\n## Larger Scale Models\n\nWe too would like to scale to larger models but unfortunately the compute required increases greatly with model size. Training models larger than 2 billion parameters was simply not possible within our compute budget. Kaplan et al. [1], one of the most widely known scaling laws, also only scale to 1.5 billion non-embedding parameter models, so we feel our work can still have a significant impact on the scaling laws community at this scale. Moreover, Bhagia et al. [2] only use models of up to 2b parameters to predict the performance of 6b and 14b parameter models. Finally, in the Gemma 2 report [3], Table 9, already corroborates our prediction that depth improves benchmark performance more than width; our work demonstrates this trend in an open, scientific and reproducible manner for the NeurIPS community. Hence, we believe our paper will be very impactful to the community.\n\nDuring the rebuttal period, we have updated our manuscript to include benchmark scaling laws where we fit to models less than 2b parameters and check the predictions against the actual data for our 2b parameter models observing high agreement. However, the rebuttal process constraints mean that we cannot share the new plots directly, so we will simply summarize the results here and then include them in the camera ready copy. We find that the formula proposed by Gadre et al. (fitted on all data in Figure 5) to be more robust to extrapolation than the formula proposed by Bhagia et al. (fitted on all data in Figure 6). The Gadre et al. formula yields near perfect extrapolation to the 2 billion parameter models when fitted on all models with less than 2b parameters.\n\n## Using Intermediate Checkpoints\n\nHägele at al. [4] suggest practitioners should use a warmup-stable-decay scheduler when fitting scaling laws as it allows intermediate checkpoints to be used for fitting scaling laws and offers similar performance to a cosine decay scheduler. The laws we present are fit based on checkpoints taken every 10b training tokens, but we also fit laws on checkpoints every 2b tokens and found these to be the same; hence for efficiency we fit on checkpoints taken every 10b tokens. We have included this ablation fitting on checkpoints every 2b tokens in our camera ready appendix, showing little to no difference to the laws fitted on checkpoints every 10b tokens.\n\n## What would we do next?\n\nGiven additional time and computational resources, we would first look at the impact of the expansion factor of the transformer, as this is most closely related to the aspect ratio and it would be interesting to compare and contrast to our current findings. \n\nWe thank the reviewer again for their effort in engaging with our work so far. We believe that your feedback has been valuable in preparing a more camera-ready version of our manuscript. If you find our responses satisfactory and there are no further questions we can address, please kindly consider moving your score in the direction of an even stronger recommendation of acceptance.\n\n[1] Kaplan J, McCandlish S, Henighan T, Brown TB, Chess B, Child R, Gray S, Radford A, Wu J, Amodei D. Scaling laws for neural language models. (arXiv)\n\n[2] Bhagia A, Liu J, Wettig A, Heineman D, Tafjord O, Jha AH, Soldaini L, Smith NA, Groeneveld D, Koh PW, Dodge J. Establishing task scaling laws via compute-efficient model ladders. (arXiv) \n\n[3] Team G. Gemma 2: Improving open language models at a practical size. (arXiv)\n\n[4] Hägele A, Bakouch E, Kosson A, Von Werra L, Jaggi M. Scaling laws and compute-optimal training beyond fixed training durations. (Neurips 2024)"}, {"review_id": "G6mVN1jkvg", "reviewer": "Reviewer_2Aaa", "summary": "This paper is an empirical study investigating the fragility of neural scaling laws. The authors argue that existing prescriptions are often unreliable as they are highly sensitive to the experimental design process. \n\nTo enable a more robust analysis, the authors introduce \"Gemstones,\" an open-source suite of over 4,000 Transformer checkpoints (up to 2B parameters) with diverse width/depth ratios and training configurations. Using this dataset, they demonstrate that:\n\n- Scaling laws are highly sensitive to the specific models and hyperparameters used for fitting. In particular, I really like this point \"There are many decisions to make when choosing points to fit scaling laws that significantly change the slope of the law\" \n\n- A critical width-vs-depth trade-off exists: deeper models are more FLOP-efficient, while wider models are more time-efficient in terms of wall-clock hours (note that this is workflow dependent) \n\nThe main contribution is the model suite itself, which provides a valuable resource for the community to study scaling phenomena without prohibitive compute costs.\n\nI thank the authors for doing a great service to the open research community. To further increase the paper's impact, I would encourage them to also release a Colab notebook to aggregate and visualize the experimental data (e.g., FLOPs vs. loss, downstream performance), which would make it easier for users to access and build upon their findings.", "questions": "see above.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- Clarity and High-Quality Presentation: The paper is well-written and easy to follow. The authors should be commended for explaining their design choices in detail.\n\n- Insightful Ablation Studies: The width/depth ratio analysis is a particularly interesting and useful contribution. The authors could strengthen this by comparing their findings against Kaplan's original scaling law paper and referencing the discussions in Allen-Zhu's \"Physics of LLMs.\"\n\n- Effective Demonstration of Fragility: Table 1 provides a very clear and compelling demonstration of the variability in fitting scaling laws, successfully highlighting the fragility of current methods.\n\nWeaknesses\n\n- Limited Model Scale: The primary weakness is that the study is limited to models up to 2B parameters. While an analysis at a larger scale (e.g., 10B) would be more impactful, it is completely understandable that this is constrained by the significant computational cost.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and insightful comments on our draft. We appreciate their acknowledgement of the fact that the Gemstone model suite itself represents a significant contribution, and are also encouraged by the fact that they find our detailed analysis of the fragility of scaling laws and our findings on the trade off of width and depth when training transformer language models interesting.\n\n\n## Larger Scale Models\n\nWe too would like to scale to larger models but unfortunately the compute required increases greatly with model size and so training models larger than 2 billion parameters was simply not possible within our compute budget. We still feel that our work will have a significant impact on the scaling laws community as others have shown laws fitted on models with up to 2b parameters can be highly informative scaling laws [1,2,3]. Moreover, in the Gemma 2 report [3], Table 9 already corroborates our prediction that depth improves benchmark performance more than width; our work demonstrates this trend in an open, scientific and reproducible manner for the NeurIPS community. Hence, we believe our work can still be very impactful for the NeurIPS community despite the fact that the models we release are smaller than those that industry labs often train.\n\n\n## Colab Notebook\n\nWe agree that in addition to the model suite and fitting datasets, an interactive colab notebook could greatly increase the accessibility of our data for the community and thereby elevate the impact of the work. We are happy to invest effort into this while preparing the camera ready manuscript for publication. \n\n## More Nuanced Discussion Points\n\nWe appreciate the reviewer pointing out a potential connection between our work and Allen-Zhu's \"Physics of LLMs.\" We have updated our camera ready draft to include a discussion these nuanced links focusing on how design decisions in “clean room” scientific settings can transfer to larger scale and more realistic experiments.\n\nAs we have made significant efforts to incorporate all feedback into our camera-ready version, if you do not have any other questions we can address, please kindly consider raising your score as an even stronger recommendation of acceptance.\n\n[1] Li M, Kudugunta S, Zettlemoyer L. (Mis) Fitting: A Survey of Scaling Laws. (ICLR 2025)\n\n[2] Kaplan J, McCandlish S, Henighan T, Brown TB, Chess B, Child R, Gray S, Radford A, Wu J, Amodei D. Scaling laws for neural language models. (arXiv)\n\n[3] Bhagia A, Liu J, Wettig A, Heineman D, Tafjord O, Jha AH, Soldaini L, Smith NA, Groeneveld D, Koh PW, Dodge J. Establishing task scaling laws via compute-efficient model ladders. (arXiv) \n\n[4] Team G. Gemma 2: Improving open language models at a practical size. (arXiv)"}, {"review_id": "MspGT53QC0", "reviewer": "Reviewer_bWFc", "summary": "This paper empirically studies the impact of various design choices on the outcome of scaling laws and highlights the sensitivity of scaling laws to these choices (in particular, the model depth, width, and the performance metrics being used). Alongside the empirical study, the paper also releases their checkpoints for more granular scaling analyses involving architectural and training hyperparameter variations.", "questions": "I have detailed most major questions in the weakness part in the previous section, and here are some remaining ones:\n* In Table 2, which examines the variability of scaling laws without embedding parameters, Approach 3 appears to be quite reliable across different configurations. This raises the question of whether the observed \"scaling law variability\" is actually an artifact of the fitting variance, rather than a result of underlying design choices. Is that correct?\n* How does the convex hull approach perform in Table 1 experiments?", "limitations": "Yes.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 1, "strengths_and_weaknesses": "Strengths\n* The paper studies the impact of some nuanced design choices on scaling laws that are sometimes understudied or overlooked in prior scaling laws, and highlights the sensitivity of scaling law outcomes to these choices. This study by itself could be interesting for practitioners, encouraging more careful studies of these nuanced choices.\n* The paper releases their checkpoints for future scaling law analyses, which could be useful for practitioners with limited compute.\n\n\nWeaknesses\n* The conceptual contributions of this work seems limited:\n  * The main conceptual takeaway of the empirical study – many decision choices could affect the outcome of scaling laws – have been extensively studied in previous works, and practitioners of scaling laws have been studying many of such nuances. It is not clear to me how this paper may change the way practitioners conduct scaling law analyses. I would encourage the authors to emphasize the key, discint contributions in the introduction.\n  * The specific study of the impact of model aspect ratio on loss, downstream metrics, and training time could be useful, but seems constrained in scope and has been somewhat studied in prior works (as mentioned in the paper).\n* The presentation of the paper could be improved:\n  * In particular, the experiment section is composed of a stack of loosely connected experiments, which makes it a bit hard to understand the goal of each experiment and interpret their overall significance. I would encourage the author to restructure this section so that the motivation behind each experiment is clearly articulated in advance.\n  * For the method section, the convex hull approach is only very briefly described without any formulations or details.\n  * Some of the visualizations showing different model shapes (e.g., Figs. 4 and 8) are overloaded and difficult to interpret. I recommend grouping the annotations by marker and line style for each line group (e.g., by model depth and width) to improve clarity.\n* Some empirical design or conclusion may not be rigorous:\n  * There are not many unique training confirmations (e.g., 22 model shape variations) being used in the scaling law analyses, which make some analyses questionable:\n    * In Figure 3, the comparison between different approaches may be confounded by the potential overfitting effect with limited data points. It would be much more convincing to include more data points and split them into train/test sets for robustness check.\n    * In Table 1, the conclusion that the scaling law exponent is sensitive to various design choices is also confounded by the variability due to limited data points. (Pairwise) statistical testing needs to be included here to demonstrate significance.  \n    * The statement that “open-source more than 4000 checkpoints cumulatively trained on over 10 trillion tokens.” feels a overclaim since there are only a small set of unique training runs with 350B training tokens.\n  * In the cooldown phase, the model is trained on an additional 10% of the total tokens it has already seen during training. This data repetition seems unusual and may affect the analyses. Could you clarify the motivation behind this choice?", "was_revised": true, "final_justification": "While the author responses addresses some of my questions and I have raised my score to a 3, I still have several major concerns regarding the conceptual contributions and the rigor of the paper, and I think the paper needs major revision before it can be accepted:", "rebuttal": "We thank the reviewer for their thoughtful review of our work and appreciate that they have noted the impact that open sourcing the Gemstone model suite will have on the scientific community we are a part of. We also hope that our release will enable more researchers to participate in nuanced discussions on width and depth, which the reviewer notes have previously been understudied.\n\n\n## Conceptual Contributions\n\nWe acknowledge we are not the first to see variability in scaling law analyses, and discuss this in our related work section. However, we do believe that showcasing this variability along both previously studied axes as well as novel ones in a single controlled setting is of value to the community as scaling laws are widely used for planning model training runs. In particular, we think that a large-scale study of width and depth provided by this work along paired with scaling law analyses represents a non-trivial addition to the open source literature. For example, in the Gemma 2 report [1], Table 9 presents a single result on how depth increases benchmark performance more than width, but few details about the experiment are included. In contrast, our work corroborates this in a scientific setting and open source all results for the community to build upon. We also emphasize our other novel, impactful contributions that the reviewer already notes, such as finding width and depth to affect loss differently over time vs. FLOPs, analyzing transfer of scaling laws over different validation datasets, and introducing the convex hull methodology.\n\n\n## The Convex Hull Approach\n\nWe provide a mathematical definition below, loosely based on the wikipedia entry for reconstructing functions from epigraphs, and have added this to our camera ready draft:\n\nWe can define the set of points we have to fit on as FLOPs/GPU hours $(x)$, loss value $(L)$ pairs as $\\mathcal{D} = \\\\{ (x_i, L_i) \\\\}_{i=1}^n \\subset \\mathbb{R}^2$\n\nLet $ \\text{conv}(\\mathcal{D}) $ denote the convex hull, the linear interpolation of any two points in $\\mathcal{D}$:\n$$\n\\text{conv}(\\mathcal{D}) = \\left\\\\{ \\sum_{i=1}^{n} \\lambda_i (x_i, L_i) \\ \\middle|\\ \\lambda_i \\geq 0,\\ \\sum_{i=1}^{n} \\lambda_i = 1 \\right\\\\}\n$$\n\nDefine $\\hat{L}(x) = \\min(\\\\{y | (x,y) \\in \\mathcal{D}\\\\})$ as the lower convex hull is the graph of this new function:\n$ \\\\{(x, \\hat{L}(x)) | x \\in \\text{Dom}(\\hat{L})\\\\} $\n\nWe think the easiest visualisation of this is in Figure 7 where the red line is the convex hull, the colored crosses are the vertices and the grey lines are all possible points in the dataset. The exact code to generate the convex hull is in our supplementary material, in the function `plotters.figure_2_correct_data.get_resource_hull`\n\n## Rigour\n\nThe reviewer has noticed a key merit of our paper in that we present two distinct sets of contributions. One part of our work looks at the established set of design choices one must make when fitting scaling laws and the other considers varying the shapes of the models we train along dimensions of width and depth.\n\nWe remark that many impactful scaling law studies train fewer distinct model shapes than we do, such as Porian et al. [4] who train 16 distinct model shapes and even one of the largest studies, Chinchilla [2], only considers 50 distinct model shapes. To fit a scaling law, we need data points which consist of a compute budget (often made up of a model parameter count and number of training tokens) and loss values. Some prior work such as Chinchilla [2], Kaplan [3] and Porian [4] choose to do this by running many independent training runs as they use a cosine learning rate scheduler. To reduce cost and increase efficiency, we follow Hägele et al. [5] who propose practitioners should use a warmup-stable-decay learning rate scheduler so that all intermediate checkpoints in the stable regime can be used during fitting, this means we can utilise all 4000 of our checkpoints in different ways when fitting scaling laws much more effectively and train for much larger token counts. Our 4000 checkpoints can be split into 3 categories {main, cooldown, lr ablation}; each can use all points to fit a scaling law. We also highlight that in Table 1 we see fragility over each of these subsets, inferring this is a repeatable pattern. We also refer to our related works section to highlight some fragility in scaling laws has been seen in prior work as it is inherent to all scaling laws due to the asymptotic flatness of power law curves; we corroborate and extend these results leveraging controlled experimental training of the Gemstone suite.\n\n\n## Cooldown\n\n10% here is referring to token counts—_there is no data repetition during training run in any of our experiments_. For example, a model trained for 40b tokens and then cooled down, would be cooled down on 4b _new_ tokens. However, it is worth noting that the 4b additional tokens the model would see during cooldown are actually the next 4b tokens that the model would have trained on had the learning rate remained constant. This design allows for a controlled comparison where two “hot” and “cooled down” checkpoints can be compared such the _only_ difference is whether or not the most recent 10% of tokens were trained on at a constant lr, or while the lr was linearly decaying.\n\n\n## Reducing Variance During Fitting\n\nWe would like to highlight the reduced variance and better fit found by our convex hull approach as demonstrated in Figure 3 and utilized throughout the paper including the experiments behind Tables 1 and 2. In addition, as shown in Figure 19, we also check that our grid search size and delta in the Huber loss are in a low variance regime when fitting Approach 3 laws to limit the impact of fitting noise on our results. During the rebuttal period, we have also conducted a leave-one-out analysis over the 22 models and added this to the camera ready draft appendix so that the reader can visually contextualize fitting variability caused by model selection.\n\nAs we have made significant efforts to incorporate all feedback into our camera-ready version, including addressing concerns over readability of figures, if you feel that your questions have been sufficiently addressed, please kindly consider raising your score.\n\n[1] Team G. Gemma 2: Improving open language models at a practical size. (arXiv)\n\n[2] Hoffmann J, Borgeaud S, Mensch A, Buchatskaya E, Cai T, Rutherford E, Casas DD, Hendricks LA, Welbl J, Clark A, Hennigan T. Training compute-optimal large language models. (arXiv)\n\n[3] Kaplan J, McCandlish S, Henighan T, Brown TB, Chess B, Child R, Gray S, Radford A, Wu J, Amodei D. Scaling laws for neural language models. (arXiv)\n\n[4] Porian T, Wortsman M, Jitsev J, Schmidt L, Carmon Y. Resolving discrepancies in compute-optimal scaling of language models. (NeurIPS 2024)\n\n[5] Hägele A, Bakouch E, Kosson A, Von Werra L, Jaggi M. Scaling laws and compute-optimal training beyond fixed training durations. (Neurips 2024)"}, {"review_id": "EoacgkR7i2", "reviewer": "Reviewer_by1a", "summary": "In this paper, the authors design new scaling law approach, including model selection, the choice of learning rate, and\n107 curve fitting schemes. As a byproduct, the authors release the Gemstone, an open-source scaling law dataset, consisting of over 4000 checkpoints from transformers with up to 2 billion parameters and diverse architectural shapes.", "questions": "Please see Strengths And Weaknesses. I would expect to see answers for: \n\n- if the model size exceeds 2B, will observations in this paper still hold? And at what size, the observations could deviate?\n\n- If the model architecture changes to non-Gemma models, will observations in this paper still hold? If not, why?\n\n- Please make the general discussion of the fragility and common pitfalls of prior scaling laws more detailed and clearly marked. It is also better to have some sort of side-by-side comparison with the proposed one. (for example, using a table/figure)", "limitations": "Yes.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "(+) The authors open-sourced more than 4000 checkpoints cumulatively trained on over 10 trillion tokens.\n\n(+) The paper is clearly written and easy to follow.\n\n(-) The paper title is confusing. If Gemstone is a model suite, then it is more suitable for the DB track. Instead, it seems the authors propose a new scaling law approach. However, to this reviewer, it is unclear what is the takeaway from this new scaling law curve.\n\n(-) The model size cut-off is 2 billion parameters, but is limited. Only the lightweight LLMs are of this size, and their standard version are usually much larger at example sizes of 7B, 8B, 13B or 70B. Such upper bound of 2 billion parameters also limit the study of scale up and \"hitting the wall\".\n\n(-) Gemstone is loosely based on scaled-down variants of the Gemma architecture. It overlooks the massive LLM architectures used in other SOTA models such as LLAMA family, GPT family, and DeepSeek Family. This limits the generalizability of the observations.\n\n(-) Scaling is usually discussed in the context of optimal performance of a given model size. It is expected that different architectures and hyperparameters may lead to a model of a given size that performs worse than the best known models at the same size. Therefore, this reviewer is not convinced by the motivation.\n\n(-) The fragility and common pitfalls of prior scaling laws was not clearly spell out. It's better to make a more explicit comparision with the proposed method.", "was_revised": true, "final_justification": "Thanks for your rebuttal and partially addressing my concerns. I have raised my score accordingly. However, the inherent problem of 2B model size and limitation of Gemma structure remains.", "rebuttal": "We thank the reviewer for taking time to read our draft and their acknowledgement of the large contribution that our Gemstone model suite will be to the community.\n\n\n## Our Choice of Track\n\nWe believe our scientific contributions, including our analysis of the pitfalls of scaling laws and the scientific insights gained from varying the width and depth of the models, make this paper more appropriate for the main track. In addition, we introduce a convex hull fitting approach and analyze the impact of fitting scaling laws across different validation sets. Our results include several novel findings related to width and depth: increasing width decreases loss quicker than depth over time but slower over FLOPs, and increasing depth improves benchmark scores more than width under a fixed compute budget.\n\n\n## Model Sizes Greater Than 2B\n\nWe agree with the reviewer that studying larger models would be an interesting extension of our work, but as the compute required scales greatly with model size, we chose to train for longer token horizons rather than increasing model scale. We highlight that several scaling law papers have fitted on models with less than 2 billion parameters and produced results that generalise to much larger parameter counts [1,2,3]. Moreover, scaling laws fitted on small models have become commonplace in large SoTA model releases, predicting the behaviour of much larger models with extraordinary accuracy [4,5,6]. Hence, we believe our paper will be impactful for the NeurIPS community even thought it does not include models beyond 2b parameters.\n\n\n## Non-Gemma Architectures\n\nIn order to conduct a rigorous scientific study, we restrict some architectural considerations in order to analyze width and depth with high fidelity. Although many of our architectural choices are based upon Gemma models, we also take inspiration from Llama and Pythia models. Specifically, we take the convention of Llama models where the head dimension is always the embedding dimension divided by the number of heads and use the Pythia tokenizer in all experiments.\n\nWe would also like to emphasize how our scaling law fragility results relate to these fixed architectural parameters. Since we find that many factors impact the fit of a scaling law we caution against transferring scaling laws without careful attention to the individual design decisions implicit in that scaling law. That said, we believe that in our setting our scaling laws would generalise as well as the ones shown in [2] and [3] who also fit on models of less than 2b parameters and generalise to much larger parameter counts with high accuracy. We also believe that our key methodological contributions such as the convex hull will transfer to other settings. Moreover, our findings such as the choice of aspect ratio to optimize loss with respect to time and to step count are widely impactful for all practitioners as we will open source all of our data. Finally, we hope this work drives extra consideration in other areas of architecture optimization such as expansion factor in transformer models and number of experts in a mixture of expert models as these are highly related to aspect ratio.\n\n\n## Scaling\n\nWe have to politely disagree with the reviewer and think scaling should be discussed in terms of optimal performance for a given compute budget (FLOPs), not model size. We believe the primary motivation of scaling laws is to optimize performance of larger models by training many smaller ones and fitting laws to the data those runs produce. We aim to offer the community a new angle to optimize performance at a given parameter count by leveraging our findings on how width and depth can be varied to achieve better performance. We note that it is common to overtrain models past any prescribed “compute optimal” point (Gadre et al.) and note how practitioners could also use our work to further understand how width and depth affects their FLOPs per training step and optimize over this dimension as well. \n\nWe are aware that training models with varying widths and depths does lead to some models being suboptimal for a given FLOP budget and highlight how our convex hull approach is designed to stop this impacting our fitted laws. We also note the impactful additional information gained for our community by identifying which models are slightly suboptimal and our analysis of them throughout our paper. Finally, we highlight how the convex hull approach can also be used by the community in the future to fit high quality scaling laws with less data as shown in Figure 3.\n\n\n## Common Pitfalls of Scaling Laws\n\nIn the fragility and common pitfalls section, we are highlighting inherent fragility and pitfalls of all scaling laws. The asymptotic flatness of power law curves makes estimation of their parameters an ill-conditioned problem. Moreover, this is compounded when fitting (interpolating) what are already inherently noisy obervations from real training runs and we believe that our results show how this ill-conditioning can quickly impact predictions in many situations. We want to emphasize that all scaling laws in general should be considered in as close to their original setting and context as possible and practitioners should not expect them to transfer to significantly different settings without non-trivial effort. However, this is not a flaw of our study or methodology and to our knowledge, there is no specific proposed method to compare to that solves this variability problem entirely. With that said, in Figure 3, we do demonstrate how our convex hull method for the approach 1 fitting method (black crosses, blue line) compares to prior fitting approaches (red crosses, red line), and think that the reduction in the noise it yields is extremely promising.\n\nIn response to your feedback, we have made significant efforts to improve our camera-ready version. If your primary concerns are at least somewhat addressed, and you have no further questions, please kindly consider raising your score.\n\n[1] Li M, Kudugunta S, Zettlemoyer L. (Mis) Fitting: A Survey of Scaling Laws. (ICLR 2025)\n\n[2] Kaplan J, McCandlish S, Henighan T, Brown TB, Chess B, Child R, Gray S, Radford A, Wu J, Amodei D. Scaling laws for neural language models. (arXiv)\n\n[3] Bhagia A, Liu J, Wettig A, Heineman D, Tafjord O, Jha AH, Soldaini L, Smith NA, Groeneveld D, Koh PW, Dodge J. Establishing task scaling laws via compute-efficient model ladders. (arXiv)\n\n[4] Grattafiori A, Dubey A, Jauhri A, Pandey A, Kadian A, Al-Dahle A, Letman A, Mathur A, Schelten A, Vaughan A, Yang A. The llama 3 herd of models. (arXiv)\n\n[5] Hu S, Tu Y, Han X, He C, Cui G, Long X, Zheng Z, Fang Y, Huang Y, Zhao W, Zhang X. Minicpm: Unveiling the potential of small language models with scalable training strategies. (arXiv)\n\n[6] Bi X, Chen D, Chen G, Chen S, Dai D, Deng C, Ding H, Dong K, Du Q, Fu Z, Gao H. Deepseek llm: Scaling open-source language models with longtermism. (arXiv)\n\n[7] Gadre SY, Smyrnis G, Shankar V, Gururangan S, Wortsman M, Shao R, Mercat J, Fang A, Li J, Keh S, Xin R. Language models scale reliably with over-training and on downstream tasks. (ICLR 2025)"}, {"review_id": "DzFiTIb2z6", "reviewer": "Reviewer_Kwid", "summary": "The paper presents an empirical study on scaling laws. The setup is using the Dolma 1.7 dataset and training standard decoder-only LLMs with variable depth/width ratios. The contributions include:\n\n1. Replicating Chinchilla's finding of keeping tokens/params constant when scaling (fig 1)\n2. A new proposed fitting method using convex hulls.\n3. A demonstration on the variability of fitting laws (table 1)\n4. A demonstration that depth is better than width with constant flops (Figure 7,8)", "questions": "1. Are figures 4,5,6 novel results? It's not clear what is novel about them to me.\n2. Could you try your convex hull method on other datasets? Does it consistently demonstrate better stability? If so, that's a good contribution that should be highlighted more.\n3. Could you add a section with \"recommendations for practitioners\"? Currently there are lots of experiments, but the take-away message is less clear. \n4. “then train at a constant learning rate, which we adjust for model size as described 137 in Appendix A.1”. Could you comment on why you are training with constant LR? Typically LLms are trained with cosine delay.", "limitations": "na", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Pros:\n- The results on depth vs width are interesting and impactful.\n- The demonstration of variability in fitting laws is useful.\n- The proposed convex hull method could be impactful.\n- The paper is well written.\n- The authors give lots of details regarding experimental setup -- training data, model implementation etc.\n\n\nCons:\n- Many of the results have limited novelty. \n- The take-away message is not clear.\n\n\n\n\nMinor nitpicks:\n- Figure 1 doesn't convey any useful information. I'd recommend removing it. Everyone knows what width/depth is.\n- Figure 3 has too bad aspect ratio, it needs more vertical space.\n- Some references are missing, e.g. \"Scaling Optimal LR Across Token Horizons\".", "was_revised": true, "final_justification": "Nothing material has changed from my initial review that compels me to change the score.", "rebuttal": "We thank the reviewer for their time and effort, and appreciate that they acknowledge the impact that our new convex hull approach and results on width and depth will have on the NeurIPS community.\n\n\n## Figures 4,5 and 6\n\nTo the best of our knowledge, Figure 4 is a novel result in a scientific setting. Figures 5 and 6 are demonstrating that our suite of models also extends to benchmark scaling laws. During the rebuttal period we have also conducted experiments fitting scaling laws over the datasets shown in Figure 4 to more effectively demonstrate the impact of using laws fitted on different datasets. Moreover, we extend our current benchmark scaling laws in Figures 4 and 5 by detailing their extrapolation when the 2b parameter models are held out. All of these experiments will be included in the camera ready version even though we cannot directly share them during the rebuttal process. In summary, we find that the formula proposed by Gadre et al. (fitted on all data in Figure 5) to be more robust to extrapolation than the formula proposed by Bhagia et al. (fitted on all data in Figure 6) with the Gadre et al. formula giving near perfect extrapolation to the 2 billion parameter models when fitted on all smaller models. With regards to fitting laws over multiple validation sets, we find that the fitted laws only vary slightly as the current Figure 4 implies.\n\n\n## Convex Hull Method\n\nWe agree the convex hull methodology has the potential to be highly impactful and widely used within the community. We hope that future research considers this method alongside the classical approach 1 fitting procedure and highlights how it can be used by practitioners to fit stable scaling laws with fewer data points.\n\n\n## Recommendations\n\nWe caution making sweeping recommendations for future practitioners due to the fragility we find to be inherent to scaling laws. Hence, our main recommendation is to be very careful of the pitfalls we highlight, and to make sure that the final configuration for training a larger “production” model mirrors the set of assumptions made when building the smaller models used to fit the scaling law. We also think our other contributions will also be useful to future practitioners: how width and depth impact training speed, how width and depth impact of benchmark accuracy versus validation loss, and how using the convex hull method for approach 1 can drastically reduce variability during fitting.\n\n\n## Learning Rate Schedule\n\nHägele at al. [1] suggest practitioners should use a warmup-stable-decay scheduler when fitting scaling laws as it allows intermediate checkpoints to be used for fitting and offers similar performance to a cosine decay scheduler. Appendix 1 details how we use a scalable initialisation and learning rate transfer to models of different shapes to be compared fairly in our study.\n\nIn response to the reviewer’s valuable feedback, we have added extra references listed to our camera ready draft, and have made significant efforts to incorporate all other the feedback into the manuscript. If your primary concerns have been adequately addressed, and you do not have any other questions, please kindly consider raising your score as a stronger recommendation of acceptance.\n\n[1] Hägele A, Bakouch E, Kosson A, Von Werra L, Jaggi M. Scaling laws and compute-optimal training beyond fixed training durations. (Neurips 2024)"}]

Author rebuttal:

Meta-review: This is a good empirical scaling study, which covers Gemma-LM scaling.

The main contributions (and positives) are the following:

1) Depth vs Width alternatives. Most previous studies stay within a very tight range of depth/width pattern (Figure 2). This paper shows 2 interesting features on a depth-width tradeoff. Moving off this line tend to produce flop-optimal models. However wider models are more wallclock-time optimal.
2) They provide a large family of opensourced checkpoints for exploration. Extensive details on setup: initialization, LR choices, etc. are appreciated for full reproducibility (which is famously lacking in some scaling law papers).

They have acknowledged limitations (model size <2B), restricted to Gemma-family models.

I believe on balance, this is a good paper; I believe the referees are generally positive about the content and presentation, with only minor suggestions within the scope of paper (for example, scaling this work is obviously desirable but introduces resource demands which may not feasible; further, at current scales, it demonstrates a trend already).

(Accept)
Decision: Accept (poster)