Title: Gemstones: A Model Suite for Multi-Faceted Scaling Laws
Venue: NeurIPS 2025
Accepted: yes
Keywords: neural scaling laws

large language models
Abstract: Scaling laws are typically fit using a family of models with a narrow range of frozen hyperparameter choices. 
In this work we study scaling laws using multiple architectural shapes and hyperparameter choices, highlighting their impact on resulting prescriptions.
As a primary artifact of our research, we release the Gemstones: an open-source scaling law dataset, consisting of over 4000 checkpoints from transformers with up to 2 billion parameters and diverse architectural shapes; including ablations over learning rate and cooldown.
Our checkpoints enable more complex studies of scaling, such as analyzing the relationship between width and depth.
By examining our model suite, we find that the prescriptions of scaling laws can be highly sensitive to the experimental design process and the specific model checkpoints used during fitting.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an empirical study on scaling laws. The setup is using the Dolma 1.7 dataset and training standard decoder-only LLMs with variable depth/width ratios. The contributions include:\n\n1. Replicating Chinchilla's finding of keeping tokens/params constant when scaling (fig 1)\n2. A new proposed fitting method using convex hulls.\n3. A demonstration on the variability of fitting laws (table 1)\n4. A demonstration that depth is better than width with constant flops (Figure 7,8)\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n- The results on depth vs width are interesting and impactful.\n- The demonstration of variability in fitting laws is useful.\n- The proposed convex hull method could be impactful.\n- The paper is well written.\n- The authors give lots of details regarding experimental setup -- training data, model implementation etc.\n\n\nCons:\n- Many of the results have limited novelty. \n- The take-away message is not clear.\n\n\n\n\nMinor nitpicks:\n- Figure 1 doesn't convey any useful information. I'd recommend removing it. Everyone knows what width/depth is.\n- Figure 3 has too bad aspect ratio, it needs more vertical space.\n- Some references are missing, e.g. \"Scaling Optimal LR Across Token Horizons\".\n\nQUESTIONS:\n1. Are figures 4,5,6 novel results? It's not clear what is novel about them to me.\n2. Could you try your convex hull method on other datasets? Does it consistently demonstrate better stability? If so, that's a good contribution that should be highlighted more.\n3. Could you add a section with \"recommendations for practitioners\"? Currently there are lots of experiments, but the take-away message is less clear. \n4. “then train at a constant learning rate, which we adjust for model size as described 137 in Appendix A.1”. Could you comment on why you are training with constant LR? Typically LLms are trained with cosine delay.\n\nLIMITATIONS:\nna", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors design new scaling law approach, including model selection, the choice of learning rate, and\n107 curve fitting schemes. As a byproduct, the authors release the Gemstone, an open-source scaling law dataset, consisting of over 4000 checkpoints from transformers with up to 2 billion parameters and diverse architectural shapes.\n\nSTRENGTHS AND WEAKNESSES:\n(+) The authors open-sourced more than 4000 checkpoints cumulatively trained on over 10 trillion tokens.\n\n(+) The paper is clearly written and easy to follow.\n\n(-) The paper title is confusing. If Gemstone is a model suite, then it is more suitable for the DB track. Instead, it seems the authors propose a new scaling law approach. However, to this reviewer, it is unclear what is the takeaway from this new scaling law curve.\n\n(-) The model size cut-off is 2 billion parameters, but is limited. Only the lightweight LLMs are of this size, and their standard version are usually much larger at example sizes of 7B, 8B, 13B or 70B. Such upper bound of 2 billion parameters also limit the study of scale up and \"hitting the wall\".\n\n(-) Gemstone is loosely based on scaled-down variants of the Gemma architecture. It overlooks the massive LLM architectures used in other SOTA models such as LLAMA family, GPT family, and DeepSeek Family. This limits the generalizability of the observations.\n\n(-) Scaling is usually discussed in the context of optimal performance of a given model size. It is expected that different architectures and hyperparameters may lead to a model of a given size that performs worse than the best known models at the same size. Therefore, this reviewer is not convinced by the motivation.\n\n(-) The fragility and common pitfalls of prior scaling laws was not clearly spell out. It's better to make a more explicit comparision with the proposed method.\n\nQUESTIONS:\nPlease see Strengths And Weaknesses. I would expect to see answers for: \n\n- if the model size exceeds 2B, will observations in this paper still hold? And at what size, the observations could deviate?\n\n- If the model architecture changes to non-Gemma models, will observations in this paper still hold? If not, why?\n\n- Please make the general discussion of the fragility and common pitfalls of prior scaling laws more detailed and clearly marked. It is also better to have some sort of side-by-side comparison with the proposed one. (for example, using a table/figure)\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper empirically studies the impact of various design choices on the outcome of scaling laws and highlights the sensitivity of scaling laws to these choices (in particular, the model depth, width, and the performance metrics being used). Alongside the empirical study, the paper also releases their checkpoints for more granular scaling analyses involving architectural and training hyperparameter variations.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n* The paper studies the impact of some nuanced design choices on scaling laws that are sometimes understudied or overlooked in prior scaling laws, and highlights the sensitivity of scaling law outcomes to these choices. This study by itself could be interesting for practitioners, encouraging more careful studies of these nuanced choices.\n* The paper releases their checkpoints for future scaling law analyses, which could be useful for practitioners with limited compute.\n\n\nWeaknesses\n* The conceptual contributions of this work seems limited:\n  * The main conceptual takeaway of the empirical study – many decision choices could affect the outcome of scaling laws – have been extensively studied in previous works, and practitioners of scaling laws have been studying many of such nuances. It is not clear to me how this paper may change the way practitioners conduct scaling law analyses. I would encourage the authors to emphasize the key, discint contributions in the introduction.\n  * The specific study of the impact of model aspect ratio on loss, downstream metrics, and training time could be useful, but seems constrained in scope and has been somewhat studied in prior works (as mentioned in the paper).\n* The presentation of the paper could be improved:\n  * In particular, the experiment section is composed of a stack of loosely connected experiments, which makes it a bit hard to understand the goal of each experiment and interpret their overall significance. I would encourage the author to restructure this section so that the motivation behind each experiment is clearly articulated in advance.\n  * For the method section, the convex hull approach is only very briefly described without any formulations or details.\n  * Some of the visualizations showing different model shapes (e.g., Figs. 4 and 8) are overloaded and difficult to interpret. I recommend grouping the annotations by marker and line style for each line group (e.g., by model depth and width) to improve clarity.\n* Some empirical design or conclusion may not be rigorous:\n  * There are not many unique training confirmations (e.g., 22 model shape variations) being used in the scaling law analyses, which make some analyses questionable:\n    * In Figure 3, the comparison between different approaches may be confounded by the potential overfitting effect with limited data points. It would be much more convincing to include more data points and split them into train/test sets for robustness check.\n    * In Table 1, the conclusion that the scaling law exponent is sensitive to various design choices is also confounded by the variability due to limited data points. (Pairwise) statistical testing needs to be included here to demonstrate significance.  \n    * The statement that “open-source more than 4000 checkpoints cumulatively trained on over 10 trillion tokens.” feels a overclaim since there are only a small set of unique training runs with 350B training tokens.\n  * In the cooldown phase, the model is trained on an additional 10% of the total tokens it has already seen during training. This data repetition seems unusual and may affect the analyses. Could you clarify the motivation behind this choice?\n\nQUESTIONS:\nI have detailed most major questions in the weakness part in the previous section, and here are some remaining ones:\n* In Table 2, which examines the variability of scaling laws without embedding parameters, Approach 3 appears to be quite reliable across different configurations. This raises the question of whether the observed \"scaling law variability\" is actually an artifact of the fitting variance, rather than a result of underlying design choices. Is that correct?\n* How does the convex hull approach perform in Table 1 experiments?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is an empirical study investigating the fragility of neural scaling laws. The authors argue that existing prescriptions are often unreliable as they are highly sensitive to the experimental design process. \n\nTo enable a more robust analysis, the authors introduce \"Gemstones,\" an open-source suite of over 4,000 Transformer checkpoints (up to 2B parameters) with diverse width/depth ratios and training configurations. Using this dataset, they demonstrate that:\n\n- Scaling laws are highly sensitive to the specific models and hyperparameters used for fitting. In particular, I really like this point \"There are many decisions to make when choosing points to fit scaling laws that significantly change the slope of the law\" \n\n- A critical width-vs-depth trade-off exists: deeper models are more FLOP-efficient, while wider models are more time-efficient in terms of wall-clock hours (note that this is workflow dependent) \n\nThe main contribution is the model suite itself, which provides a valuable resource for the community to study scaling phenomena without prohibitive compute costs.\n\nI thank the authors for doing a great service to the open research community. To further increase the paper's impact, I would encourage them to also release a Colab notebook to aggregate and visualize the experimental data (e.g., FLOPs vs. loss, downstream performance), which would make it easier for users to access and build upon their findings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- Clarity and High-Quality Presentation: The paper is well-written and easy to follow. The authors should be commended for explaining their design choices in detail.\n\n- Insightful Ablation Studies: The width/depth ratio analysis is a particularly interesting and useful contribution. The authors could strengthen this by comparing their findings against Kaplan's original scaling law paper and referencing the discussions in Allen-Zhu's \"Physics of LLMs.\"\n\n- Effective Demonstration of Fragility: Table 1 provides a very clear and compelling demonstration of the variability in fitting scaling laws, successfully highlighting the fragility of current methods.\n\nWeaknesses\n\n- Limited Model Scale: The primary weakness is that the study is limited to models up to 2B parameters. While an analysis at a larger scale (e.g., 10B) would be more impactful, it is completely understandable that this is constrained by the significant computational cost.\n\nQUESTIONS:\nsee above.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper studies scaling laws for pre-training while going beyond narrow hyper-parameter choices and varying both depth and width. They futhur adapt the typical formulation of scaling laws to be more robust to biases and sparse data regions in the fitting data. The authors Intermidairy checkpoints are also released at frequent intervals to aid in further open-science study of these laws.

STRENGTHS AND WEAKNESSES:
Strengths: 
- The paper is genrally well-written 
- The experimental depth of the work requires singificant compute and resources which is not widely accessible to most researchers. The open-sourcing of the many model checkpoints will be greatly valuable to the community. 
- The paper covers both different optimization, and architectural choices in deriving the laws. 

Weaknesses: 
- The work would have benefited from having a larger scale of models, exclusively reserved for testing laws derived at a smaller scale. My understanding is that all datapoints up to 2B scale are used in fitting the loss, correct? It would be good to test the laws on a held out model size using the optimal ratios dervied from the fittings.

QUESTIONS:
- Since intermidiary checkpoints are also used in the fitting, does this bias the laws? how sensitive are the laws with respect to the frequency of the included checkpoints? 
- If you were able given resource and time-constraints, etc. to ablate another axis, which of the ones listed in limitations would you priotize and why? I think it'd would be an appreciated piece of information if included in the limitations section with more detail.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-05