Title: Zero-Shot Performance Prediction for Probabilistic Scaling Laws
Venue: NeurIPS 2025
Accepted: yes
Keywords: Gaussian Processes

Active Learning

Scaling Laws
Abstract: The prediction of learning curves for Natural Language Processing (NLP) models enables informed decision-making to meet specific performance objectives, while reducing computational overhead and lowering the costs associated with dataset acquisition and curation. In this work, we formulate the prediction task as a multitask learning problem, where each task’s data is modelled as being organized within a two-layer hierarchy. To model the shared information and dependencies across tasks and hierarchical levels, we employ latent variable multi-output Gaussian Processes, enabling to account for task correlations and supporting zero-shot prediction of learning curves (LCs). We demonstrate that this approach facilitates the development of probabilistic scaling laws at lower costs. Applying an active learning strategy, LCs can be queried to reduce predictive uncertainty and provide predictions close to ground truth scaling laws. We validate our framework on three small-scale NLP datasets with up to $30$ LCs. These are obtained from nanoGPT models, from bilingual translation using mBART and Transformer models, and from multilingual translation using M2M100 models of varying sizes.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes using latent variable multi-output Gaussian processes (specifically the MaGP model) to predict training and learning curves in a zero-shot manner. The key insight is modeling these prediction tasks as hierarchical multitask learning problems, where correlations across tasks and within hierarchies can be exploited. They demonstrate this on three NLP datasets: nanoGPT training curves, bilingual translation learning curves, and multilingual translation performance. An active learning component helps reduce uncertainty in predictions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- The motivation is solid - developing scaling laws is indeed computationally expensive, and any principled method to reduce these costs would be valuable.  I think the probabilistic nature of the approach is a clear advantage over deterministic scaling law methods. Getting uncertainty estimates on scaling law parameters could be quite useful for decision-making about model development. The two-layer hierarchy matches intuitive structure in all three tasks and lets the GP share signal efficiently.\n- The experimental evaluation is reasonably comprehensive given the scope. Testing on three different types of NLP tasks (language modeling, bilingual, and multilingual translation) shows some generality. The comparison against multiple baselines, including BNNs with different basis functions, helps establish the method's effectiveness. I also liked the active learning analysis showing how uncertainty can be reduced through strategic querying.\n- Table 2 and Figure 7 don’t just report accuracy; they tag each train-test split or query strategy with the PetaFLOPs burned to obtain it. The active-learning policy slices AbC by roughly one-half relative to a Largest-First schedule while staying a good 2× cheaper than the Random schedule .\n\n\n\nWeaknesses\n\n- I'm curious about the computational costs of the MaGP method itself. While the goal is to reduce the cost of obtaining scaling laws, how expensive is the GP inference, especially with the hierarchical structure and active learning?\n- The active learning results are promising, I'd also like to see more analysis of which queries are actually selected. Maybe the patterns might provide insights into what information is most valuable for scaling law prediction.\n- The writing is quite dense in places, particularly sections 3 and 4. I found myself re-reading several passages to understand the model setup. Given that this is targeting a NeurIPS audience who might not all be familiar with hierarchical GPs, clearer exposition would help.\n\nQUESTIONS:\nNA\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper frames learning-curve and training-curve extrapolation in NLP as multi-task, two-level hierarchical regression problem and instantiates it with the latent-variable multi-output Gaussian process of Ma et al. (2023) (“MaGP”). Main claims are as follows :\n\n1. Zero-shot prediction of unseen learning curves is feasible by leveraging task-level correlations and hierarchical priors .\n2. The GP posterior can be Monte-Carlo–sampled to yield probabilistic scaling laws, giving uncertainty bands on fitted γ-laws .\n3. By employing an active learning approach, the uncertainty of the predictions can be significantly reduced   \n4. The approach generalises beyond scaling-law data (nanoGPT) to bilingual and multilingual translation curves\n\nThe approach outperforms DHGP, two BNN variants, and a naïve regression baseline across MSE/MNLPD metrics. The empirical evidence relies on three small datasets (≤ 30 curves per domain) with only few points per curve, yet MaGP consistently beats baselines.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- Casting curve extrapolation as hierarchical multitask GP clearly articulated.\n- Folding model uncertainty into γ-law fits via MC integration is principled and valuable for budgeting decisions in LLM training.\n- Active-learning cost analysis a realistic compute/accuracy trade-off, absent from most scaling-law studies .  \n- Comprehensive baselines, Includes DHGP, two BNN variants, and task-averaging heuristics, isolating the gain from hierarchy + latent correlation.\n\n### Weaknesses\n\n- All experiments use very few curves; scalability to realistic hyper-parameter spaces (≥ 10²−10³ curves) is unclear. LLM training would often depend on multiple hyperparameters and this is a combinatorially complex space\n- Does not discuss vertical offset and irreducible loss, how would that impact these predictions?\n- Variance-only ignores compute-cost heterogeneity across model sizes. \n- Query budget is ~25; realistic large-model experiments may allow far fewer queries.\n- Hierarchical grouping is hand-crafted. How sensitive is the approach to mis-specified hierarchies?\n\nQUESTIONS:\n- GP cubic complexity is O(N³); the paper omits runtime or inducing-point strategies. Please report wall-clock time and memory for nanoGPT-large fits.\n- Would cost-aware fits (e.g., loss/FLOP) further improve results?\n- Authors should try to incorporate the irreducible loss in these predictions\n- For BLEU/ChrF tasks only MSE/MNLPD are shown; confidence intervals overlap heavily. Maybe break it down further and improve the plotting?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a framework for predicting learning and training curves of NLP models that leverages latent variable multi-output Gaussian process models. The key idea behind the approach is to model correlations between tasks and hierarchies to facilitate predictions for new configurations of models (ie with different numbers of parameters or different languages). Experiments validate the framework in three settings: scaling law prediction and predicting zero-shot performance/few-shot performance of translation models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The idea of using active learning for model prediction is creative, and results show that this method performs comparably to using the largest first strategy while reducing computational costs.\n- Experiments show the broad applicability of the proposed method for developing scaling laws, selecting model architectures, and predicting performance for bilingual translation models.\n\nWeaknesses:\n- There are no comparisons to more recent scaling law estimation methods, despite them being cited in the introduction.\n- The experiments are with limited models and datasets, and it is therefore unclear how robust the results are. For example, for the scaling laws experiments, the paper subsamples from the NanoGPT dataset and only evaluates language models in this previous data, which were not trained for many steps (and not larger model families). \n- The datasets used in the paper are never properly introduced; authors should describe upfront what the datasets they use are and where they come from. \n\nMinor Notes:\n- Line 263: Figure has \"??\"\n- Line 372: \"proof that\" --> \"prove that\"\n\nQUESTIONS:\n- Line 232: Why do authors subsample to 11 datapoints? How much more computational demand does it require to use the full dataset?\n- Why is there no direct comparison to prior scaling law estimation methods (e.g., those used by Kaplan et al., Hoffmann et al., Henighan et al.)?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper applies hierarchical Gaussian processes, often used in other types of hierarchical datasets, to the task of predicting learning curves (LCs) for NLP models and estimating frontier scaling laws for these LCs.\n\nHierarchical Gaussian processes encode outside knowledge/belief about relationships/correlations between tasks which permit stronger few-shot modeling or zero-shot modeling in LCs from tasks without much data.\n\nThe study is predominantly empirical, examining LC prediction on different small models while tweaking two model or dataset parameters (e.g. size of embedding AND number of hidden layers, or target language AND source language of translation). Hierarchical GPs permit structuring the modification of one parameter as a subtask of the supertask of modification of the other parameter.\n\nUsing a GP from Ma et al. [1], the authors explore LC prediction across a number of tasks and zero-shot few-shot settings. They also explore the error in scaling laws (linear frontier functions in log-log space) estimated from LCs. The authors compare to baselines including other GPs, Bayesian Neural Networks, and naive baselines such as averaging across tasks that are known to exist in the same subgroup in the hierarchy.\n\nResults indicate that good performance prediction and scaling law estimation can be made with Hierarchical GPs, and furthermore, that using uncertainty in probabilistic prediction can be used to select candidates for querying LCs in active learning.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n**S1**\n\nNovel application of this model to my knowledge.\n\n**S2**\n\nWide applicability, and selection of hierarchical modeling for LC prediction is an insightful way to apply belief about relationships in LCs.\n\nWeaknesses\n\n**W1**\n\nThe paper suffers a lack of clarity in communicating its assumptions and how they map to modeling decisions. A one-off statement in Section 5 that \"If correlations between data samples of different tasks are not relevant, or even a hindrance, this baseline is supposed to outperform the MaGP model\" but provides no further explanation of this distinction or the theoretical basis for the claim. I think extra attention should be paid to explaining a) the hypothesis about the specific types of hierarchical relationships that DHGP and MaGP can each express and b) why all of the data relationships in the paper more closely mimic MaGP and not DHGP.\n\nAs another example, another one-off statement in 5.3 mentions that \"hierarchies are indeed exchangeable\", which I interpret to mean that modeling one parameter axis as the higher level and the other as the lower, or vice versa, leads to no difference in effect. Yet what results indicate this? How does this interact with our chief assumptions? Are hierarchies exchangeable for all examined problems?\n\nIn essence I am requesting additional focus on the core hypothesis, a categorization of what evidence would contribute toward or against it, a clear set of research questions that are being examined, and clearly formatted evidence to address each research question.\n\n**W2**\n\nThe scope of experiments is relatively wide, but the depth in analysis, explanation, and experimentation granted each experimental setting leaves some to be desired. For example, why pursue a different set of baselines in each approach? Is there a justifiable reason, is it convenience, etc.? Each experimental setting is introduced technically but not in enough detail to describe these sorts of decisions.\n\nThe haphazard use of baselines can harm my assessment of the significance. I have no reason to believe right now that the paper captures a sharp enough view of the SoTA in performance prediction for each task type that the integration of the MaGP method actually means something substantial for those practitioners interested in predicting LCs.\n\nAs another example of some missing depth of analysis, the process of using Monte Carlo estimation and then fitting a scaling law to the estimates in addition to the real data is interesting but worth giving a full treatment or exploration. Why does bootstrapping a GP and then fitting a linear law tend to outperform merely fitting the linear law to the existing data? By how much does it do so? Is there any justification as to why this would be better?\n\n**W3**\n\nSeveral figures suffer from clarity issues. For example:\n\nFigure 4 Left -- how is the main diagonal modeled as test data if the test set is actually the next diagonal over to the right?\n\nFigure 4 Left -- No legend, no axes labels, no sense of what conclusions to take away.\n\nFigure 4 Right -- Font is too small to read, unclear what red vs blue numbers are, legends poorly formatted.\n\nFigure 8 -- no legend.\n\nFigure B.5 -- hard to read.\n\n**W4**\n\nTables suffer from clarity issues. For example:\n\nConvention states that a bold value is used to indicate the winner of a head-to-head competition in tables. Yet sometimes the bold values are inappropriately applied. For example:\n\nTable E.2 -- no bolding when MaGP does not win?\n\nTable H.3 -- bolding of non-winning values?\n\nUse of arrows on column headers to indicate direction of optimal metric values inconsistent. Present, for example, in Table 2 but not Table 1.\n\n**Grammatical and textual nitpicks**\n\n82 -- no space before “Recently”\n\n178 -- (512-8 two times?)\n\n263 -- Figure ??\n\n267 -- \"weather\" -> whether\n\n281 -- Flops exponents formatting off\n\n294 --\"while the compute costs\" -- incomplete phrase\n\n296 -- \"weather\" -> whether\n\n336 -- \"hense\"\n\n364 -- sentence has no following punctuation\n\n372 -- proof -> prove\n\n373 -- \"baselines\"\n\nAppendix A Line 33 -- grammar needs a revision\n\nAppendix B Line 72 -- \"increaseing\"\n\nAppendix D Line 122 -- \"commitee\" & 123 \"leanring\"\n\nQUESTIONS:\nMost of my questions are actually covered in my W1 and W2. I am very curious about a deeper analysis of many of the experiments and arguments made.\n\nI have only one more question to write here that has not yet been covered.\n\n**Q1**\n\nIn Figure 7, can you include a curve with cumulative compute cost as the x axis? I wonder if Smallest First is still competitive if you consider the fact that Smallest First is much cheaper to get to a critical mass of queries as compared to any other method.\n\nLIMITATIONS:\nLimitations section is a bit short. It mostly discusses limitations on the scope and breadth of analysis. I am curious about the limitations of this methodology. For example, which relationships will Gaussian processes struggle to handle? What are the failure modes of MaGP that DHGP or other models don't suffer from?\n\nAs far as societal impacts go, I do not think there are major adversary societal impacts to be concerned with. Many techniques have been proposed or already are in use for scaling law estimation. Generally, being aware of scaling laws helps us plan to improve efficiency and reduce waste while training models.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-09"}