Unveiling Downstream Performance Scaling of LLMs: A Clustering-Based Perspective
Authors: Chengyin Xu
Kaiyuan Chen
Xiao Li
Ke Shen
Chenggang Li
Venue: ICLR 2026 Poster (2026)
Keywords: Performance Prediction
Scaling Law
Large Language Models
Pretraining
Primary area: foundation or frontier models, including LLMs
TL;DR: We developed a Clustering-On-Difficulty (COD) framework that accurately predicts LLM downstream performance, achieving a 1.55% average prediction error on a 70B parameter model.

Abstract: The escalating scale and cost of Large Language Models (LLMs) training necessitate accurate pre-training prediction of downstream task performance for comprehensive understanding of scaling properties. This is challenged by: 1) the emergence phenomenon, where unpredictable capabilities appearing suddenly at critical model scales; and 2) uneven task difficulty and inconsistent performance scaling patterns, leading to high metric variability. Current prediction methods lack accuracy and reliability. We propose a Clustering-On-Difficulty (COD) framework for downstream performance prediction. The COD framework clusters tasks by their difficulty scaling features, thereby constructing a more stable and predictable task subset that exhibits well-behaved scaling characteristics with the increase of compute budget. We adopt a performance scaling law to predict cluster-wise performance with theoretical support. Predictable subset performance acts as an intermediate predictor for the full evaluation set. We further derive a mapping function to accurately extrapolate the performance of the subset to the full set. Applied to an LLM with 70B parameters, COD achieved a 1.55\% average prediction error across eight key LLM benchmarks, thus providing actionable insights for scaling properties and training monitoring during LLM pre-training.

Published as a conference paper at ICLR 2026
UNVEILING DOWNSTREAM PERFORMANCE SCALING OF LLMS: A CLUSTERING-BASE PERSPECTIVE
Chengyin Xu; Kaiyuan Chen, Xiao Li, Ke Shen, Chenggang Li Bytedance Seed
(xuchengy n. 98, chenkai yuan. 99, lixiao 20) @bytedance.com (shenke, 1i cichenggang @bytedance, com
ABSTRACT
The escalating scale and cost of Large Language Models (LLMs) training ne- cessitate accurate pre-training prediction of downstream task performance for comprchensive understanding of scaling properties. This is challenged by: 1) the emergence phenomenon, where unpredictable capabilities appearing suddenly critical model scales; and 2) uneven task difficulty and inconsistent performance scaling patterns, leading high metric variability. Current prediction methods lack accuracy and reliability. We propose a Muustringg-n-Diffiiuu for downstream performance prediction. The COD framework clusters tasks by their difficulty scaling features, thereby constructing a more stable and predictable task subset that exhibits well-behaved scaling characteristics with the increase of compute budget. We adopt performance scaling law to predict cluster- wise performance with theoretical support. Predictable subset performance acts as an intermediate predictor for the full evaluation set. We further derive mapping func- tion to accurately extrapolate the performance of the subset to the full set. Applied to an LLM with 70B parameters, COD achieved 1.55% average prediction error across eight key LLM benchmarks, thus providing actionable insights for scaling properties and training monitoring during LLM pre-training.
(COD) framework
INTRODUCTION
Large Language Models (LLMs) have emerged as transformative technologies in natural language understanding, generation, and reasoning (Achiam et al., 2023) Guo et al.) 2025 Bubeck et al.) 2023). Their impressive success heavily relics on scaling model parameters and pre- pre-training data, with training loss empirically following power-law relationship with compute (Hoffmann et al. 2022 Kaplan et al. 2020). However, this reduction in training loss primarily reflects an in-domain compression effect and does not necessarily indicate improved out-of-domain generalization or downstream performance-the factor of primary concern in practice. Specifically, performance scaling of downstream tasks aims to predict the accuracy of the target LLM on downstream tasks using metrics from smaller models. Our objective is to develop a prediction method that works reliably on diverse range of downstream tasks. optimizing the worst-case prediction error.
Despite extensive efforts, a reliable scaling law for downstream tasks remains elusive. One line of work attempts to extrapolate the performance
large model by modeling the performance-
relationship (Chen et al. 2024] Gadre et al.) 2024; Du et al. 2024 Xiao et al., (0224) Owen] [024)). but this often fails to capture the emergent behaviors of LLMs and the mismatch between the in- domain loss and downstream metrics (Zhang al. 2021). Another line of research focuses on direct
extrapolation of the performance-compute relationship (Achiam et al., 2023) Hu et al.) (024), yet single family of curves usually fails to capture the performance evaluation benchmarks with complex difficulty distributions across samples.
A key limitation of existing methods is their unreasonable assumption that all evaluation samples follow a uniform performance scaling pattern. We observe that different evaluation samples actually follow distinct performance scaling patterns, and thus applying single extrapolation formula the entire evaluation set is suboptimal. We give a detailed analysis in Sec.
Equul contribution.

---

Published as a conference paper at ICLR 2026
To address these challenges, we propose new performance scaling law, derived from the existing loss scaling law (Kaplan et al., (2220, specifically applicable to evaluation subsets that exhibit consistent performance scaling patterns. Building the performance scaling law, develop Clustering- On-Difficulty (COD) multi-stage framework for predicting downstream performance. Specifically, first create predictable subset by filtering improved MeanShift clustering algorithm. Next, we fit the performance-compuie relationships in the predictable subset under our performance scaling law, extrapolate the performance of large models within each clusters, and finally map the aggregated predictions to the complete task set.
clusters that lack scaling properties using an
Crucially, the COD framework effectively resolves the challenges posed by emergent and hetero- geneous behaviors. Regarding non-emergent behaviors, performance metries for small models often fluctuate around random guessing or exhibit severe volatility, causing existing single-stage fitring methods to fail. Our method circumvents this by identifying a strong correlation between the predictable subset metries and the full set metrics. This allows us effectively estimate the full set performance using the predictable subset, where the relationship can be fitted with a smooth curve. Regarding heterogeneous behaviors, we observe that even within the predictable subset, different task clusters exhibit distinct scaling laws. By first performing cluster-wise extrapolation and then aggregating the results, COD can accurately capture the intrinsic heterogeneous scaling patterns within the evaluation set.
We validate our COD approach on eight popular evaluation sets, including MATH (Hendrycks et al. 2021), BBH (Suzgun et al. 2023), and MMLU pro (Wang al. 2024) datasets. COD achieves average prediction error of 1.55% on an LLM with 70B parameters. Our results demonstrate that this difficulty-aware framework substantially outperforms existing methods, establishing a promising paradigm for accurate downstream performance scaling of LLMs.
Our contributions can be summarized as follows.
We propose the COD framework to address high variance and emergent phenomena in LLM performance scaling by effectively modeling the difficulty distribution within the evaluation sets.
We introduce a downstream performance scaling law for cluster-wise performance prediction, with theoretical support and experimental validation.
Extensive experiments conducted in eight different evaluation sets demonstrate that COD provides reliable predictions with an average prediction error of 1.55% on an LLM with 70B parameters.
RELATED WORK
2.1 LOSS SCALING LAWS
Loss scaling laws provide systematic framework for understanding the relationship between computational resources, data, model size, and the LLM performance. Early work by Kaplan et al. (2000) demonstrates that the pre-training loss of LLMs follows a power-law relationship with the compute (the number of floating-pott operations) used training. Subsequent studies extend these findings to other domains, such computer vision (Zhai al. 2022), graph learning (Ma al.) 2024), and vision-language models (Alabdulmohsin et al. [0022] Henighan et al. (020). Recent research has also explored scaling laws in specific contexts, such fine-tuning (Hernandez al. 2021: Tay et al. 2022),, vocabulary size optimization (Tao et al. 2024). retrieval- retrieval-augmenned models
(Shao et al. 22024), and hyperparameter tuning (Lingle) 20224] Yang et al.] (2022). These studies
highlight the broad applicability of scaling laws and their potential to guide the efficient allocation of computational resources.
2.2 DOWNSTREAM TASK PERFORMANCE SCALING
Predicting downstream task performance remains a critical challenge due to emergent abilities in LLMs that some capabilities manifest only after exceeding task-specific thresholds (Wei et al. 2022 Schaeffer et al. 2023). Recent works, such as using loss (Chen et al. 2024) or principal capability proxy, have demonstrated potential, but encounter challenges in aligning surrogate metrics with original task objectives. Other approaches manage to improve prediction
(Ruan al.] 2024
accuracy by increasing the metric resolution (Hu al. 2024 incorporating experimental data from
2

---

Published as a conference paper at ICLR 2026
CEval
GSMBK
BBH
7B
1.0
Custer 1
673M
0.3
0.4
238M
122M
Accuracy
Accuracy
0.0
0.2
0.4
15 Cons: Coene
0.3
Corat
0.0
Soune
2.5 Training Loss
2.6
23
10'
10t
101
Training Loss
Compute
Figure 1: Performance-loss relationship across different model sizes (left) and learning rate schedules (middle). Performance computt relationship for different clusters of the BBH samples(right)
other models (Ye et al. 2023). Here, we briefly review the two main types of methods for predicting downstream performance:
Lossiintermediate prediction. These methods predict the final training loss (or in-domain validation loss) of LLMs with loss scaling laws first, and then predict downstream performance through loss- performance relationships (Chen et al., 2024: Gadre et al., 2024 Du et al., 2024; Bhagia et al.. 2024). While these methods leverage established scaling laws for loss predictions, they encounter fundamental limitation: the inconsistent mapping between loss and performance metrics. addition, intermediate variable for estimation. Although answer loss correlates with the final performance metrics, its predictability remains low as predicting answer loss shares the challenges with predicting performance, including emergence phenomenon and high variance in task difficulty.
Xiao al. (2024) employ the evaluation answer loss
End-to-end performance-compute prediction. These methods (Hu al. 2024 Owen 2024
Achiam et al. 2023 Caballero et al. 2022) directly model the relationship between performance and the compute budget (or the number model parameters). They are classified into exponential and piecewise types based on different formula formulations:
Exponential methods: Achiam al. 2023) estimate and fit this relationship using a subset of the evaluation set, while still failing to predict the full set. Hu et al. (2024) address the challenge of non-emergent capabilities smaller models by employing multiple non-greedy decoding evaluations, thereby enabling accurate extrapolation of performance predictions for models with up to 2.4B parameters. However, it suffers from prohibitively high overhead during evaluation and can only predict non-greedy decoding metrics.
Piecewise method: Caballero et al. (2022) propose a smooth broken power-law that models LLM scaling by decomposing into multi-segment power laws. However, when predicting metrics for large-scale models (e.g., 70B parameters), performance trends often exhibit unexpected inflection points due to emergent capabilities or saturation effects, making piecewise functions inadequate for capturing these novel scaling regimes.
PILOT STUDY
In this section, we present the pilot experiments to illustrate the shortcomings of existing approaches.
Training loss may mismatch downstream task performance. Predicting downstream performance from training loss assumes LLMs achieve identical downstream results at the same loss value, which does not hold universally. In practice, training loss primarily serves as an indicator of in-domain fitting, whereas downstream tasks typically represent out-of-domain evaluations. Moreover, training configurations, such as model size and learning rate, can significantly affect not only the final loss but also the model's generalization capabilities.
Fig. leftt) illustrates the performance -los relationships for LLMs of different sizes on the CEval benchmark (Seifert et al. (024).. At the same training loss level, smaller models can outperform larger ones in terms of test accuracy. Because smaller models initially exhibit weaker in-domain fitting capacity, they typically require training steps reach the same loss value, which can lead
3

---

Published as a conference paper at ICLR 2026
Target EvalSet
Custerable Subset
Predictaite Subset
ACC. Compute
F.ttting
c. Exploration
Clustering Oifiicalty
Mapping
No Predi ctable Subset
Predistable
Kon Emargeni Subsen
Task-uise Poosrate Compute
Subset
The pipeline of Cluster-On- Diffiicutyu downstream task performance scaling, including stages: Represent task difficulty feature with task-wise passrate vector. Cluster on the difficulty feature and filter outliers. b. Fit cluster-wise iefiormance-complln curve. Classify clusters into extrapolatable clusters, con-extrapolattble clusters, and non-emergent clusters. Predict accuracy on extrapolatable clusters. d. Map subset accuracy prediction to full evaluation set performance.
Figure 2:
better in-domain generalization once they do. Fig. Mmiddll) compares the performance of LLMs trained under different learning rate schedules on the GSM8K dataset (Cobbe et al. (021). At the same loss level, the performance under the cosine schedule is always worse than that under the constant schedule. indicating that a lower learning rate may prioritize memorization over generalization, thereby diminishing downstream performance.
Diverse scaling patterns within the evaluation set. Different task samples exhibit unique com- putational thresholds, learning slopes, and upper bounds, making it challenging find a single fitting function (or function group) that generalizes well across diverse task samples. Fig. rright) illustrates the performance-comput relationships on three clusters randomly selected from those formed by clustering tasks based on their difficulty in the BBH benchmark. (Suzgun et al., 2023), with each cluster containing samples with similar difficulty. Even within a single evaluation set, these scaling curves vary significantly, indicating that mne-size-iits-le performance-compee curve is insufficient for capturing the full spectrum of a downstream evaluation set.
Taken together, these observations highlight the importance of modeling the heterogeneous scaling properties within an evaluation set and identifying a robust intermediate metric to serve as a reliable indicator of the downstream performance of LLMs.
4 METHOD
In this section, we first formulate the problem, then present COD in four stages (see Fig. 2 (1)
We construct sample-level difficulty scaling features and apply an improved MeanShift clustering algorithm (Sec. 4.1 (2) We derive performance scaling law with respect to task difficulty variance, enabling extrapolation performance-comput relationships for clusters with similar difficulty features. Cluster-wise curves fitted on small models identify extrapolatable clusters (Sec. 4.2)
(3) We extrapolate performance for these clusters to predict the target large model's 's accuracy on the predictable subset (Sec. 4.3). (4) Finally, we map subset accuracy full evaluation results (Sec. (.4).
Problem Formulation. Consider a language model Mc trained with a compute budget of C measured in FLOPPs Let P be set of downstream tasks that aim evaluate the model on. Fach sample T E P is defined by question-answw answer pair (g, ame)). Given a question q, the model Mc outputs probability distribution p(alq; Mc) over the space of all possible answers.
Our goal is to predict the downstream task performance of a large language model Mc using only evaluation results from smaller models {Mc,, .c...... Mc. } where C, & Curget for all i. Formally, we aim find the prediction method e minimize the absolute prediction error over
4

---

Published as a conference paper at ICLR 2026
DBSCAN
MeanShif
Improved-MeanShhit
Figure 3: t-SNE visualization of different clustering methods: DBSCAN(left), McanShif((Midddle) Improved-Ment MeanShift(Right) Each point represents an evaluation sample.
group of tasks sets (Pr hm:
arg min
Σ
AAce(Cuaga: T) Acce -Acc(Curgel))
JP
TEP,
ACogeet T) = & ((Aco(C;; T)HE (Ci) Crarget)
where Ac, T) denotes the accuracy of model Mc on task T, and Acc(Cuarget: T) is the predicted accuracy for the target model.
4.1 CLUSTERING ON DIFFICULTY
Although downstream tasks in the same evaluation set share similar themes, they exhibit significant differences in difficulty, resulting in distinct performance scaling patterns that make universal fitting function inapplicable. We propose clustering tasks by similar performance scaling behaviors to minimize intra-cluster heterogeneity maintaining a minimum cluster size.
Specifically, we train a group of language models with increasing parameter counts. These models trained with the same ratio of training tokens to compute per token. We use the same set of small models for prediction to evaluate the difficulty characteristics the task, and will not introduce the target large model evaluation results to avoid feature leakage.
For each task, we generate 100 samples using top_p=0.7 and temperature=1.0 for each model, and compute the pass rate by averaging the results. This pass rate serves as an estimate of the model's 's expected accuracy on the task. The resulting values are concatenated into a difficulty vector, ordered by increasing model size. For most tasks, this difficulty vector exhibits monotonic increase, reflecting the gradual improvement of model capability with scale.
After obtaining the difficulty feature vector for each task, we use the improved clustering algorithm that incorporates the following features: (1) Minimizing intra-class variance to ensure similar extrapolation properties within each cluster; (2) Automatic determination of cluster numbers, as the optimal number varies across evaluation sets and is difficult to pre-specify.
To further reduce intra-class variance, we propose an improved MeanShift algorithm to constrain the cluster diameter. At the same time, we maintain minimum number of tasks in each cluster to reduce metric fluctuations. We provide the t-SNE visualization of clustering results evaluation tasks on BBH (Suzgun et al.) (023) to compare the proposed method and classic clustering algorithms including DBSCAN (Ester et al.) (996) and MeanShift (Fukunaga and Hostetler, 1975). Each point represents evaluation sample, and its color denotes the cluster type. As shown in Fig. improved MeanShift effectively splits dense areas whereas DBSCAN and the original MeanShift produce connected clusters with large within-cluster distances.
We provide numerical comparison of clustering algorithms Sec. 5.3.2 and explain implementation details of improved MeanShift in Appendix A.T smoothing techniques in Appendix A.2
5

---

Published as a conference paper at ICLR 2026
4.2 FITTING
After clustering, we compute metrics for small models within each cluster. We then fit the accuracy- compute curves for each cluster using theoretically derived novel performance scaling law, focusing on clustered samples after excluding outliers. We derive the following fitting formula for the downstream task scaling law based on the training loss scaling law.
Theorem 1 (Scaling Law for Downstream Task Performance). Consider a language model Mc trained with compute budgel C and a set of downstream tasks P. Under the following assumptions: Assumption I (Power-law scaling of answer loss): the expected answer loss follows:
Lp(C) == (( Lq. Ornee: C)] = +7/
(I)
where oz, 3,, > 0 task-specific constants, with ? representing the irreducible loss.
Assumption (Unique deterministic answers): Each question has a unique deterministic answer. The model receives score 1 and only if Mc oulputs dues and otherwise.
Assumption 3 (Accuracy decomposition): The expected accuracy decomposes as:
[T-]]Acc(C]] = =g (1 (1-1) 9) 18 (gg /up asroee Qq, Mc),,
(2)
where E [0, is the random guessing baseline.
Then, the expected accuracy task P can be modeled as:
Ep Ep[Acc(C]]] - (ep))xxpperll (-ap--a---- -7) 2HL(C) (
(3)
where 1z (C) Var(g. [L atruess C)) the loss variance across the task set.
p Orrec: C)] the mean loss and (c(
Proof Sketch. By definition of the language model loss, p(ouee 44, Mc) Under Assumption I, if the answer loss follows power law L ~ aC- + then the task passrate should approximately scale as exp(-aC-8 7).
Qmee C)).
The key subtlety lies in the averaging: accuracy computes E[exp(-LL) (arithmetic mean of passrates) while the loss scaling law gives us exp(-E (geometric mean). Using Taylor expansion:
E[exp(-L)) s exp(-1z) + of.
2μz²'
where yr and of. are the mean and variance of the loss distribution.
This approximation accurate when tasks have similar difficulty feature (oī//ī. < 1), motivating our clustering approach to reduce intra-cluster variance. Assumption 3 adds the parameter for random guessing. The complete proofs are provided in Appendix
Theorem D demonstrates that metric of an evaluation set with similar difficulty features can be effectively modeled using the following formula:
(C) = g + (1
-0-
where a and b jointly influence how accuracy varies with C, while constrains the upper bound of the fitting curve, and represents the expected random guess metric for a task cluster. a, b, and trainable parameters. Note that these assumptions may not perfectly hold in practice, we provide additional discussions on assumption 3 in Appendix
4.3 EXTRAPOLATION
To ensure reliable extrapolation, we identify clusters exhibiting robust scaling patterns, as some clusters may show saturated non-emergent performance smaller models, making them unsuitable for prediction. We aim to find an extrapolation subsets that represent the full set performance, and the subset metric intermediate indicator for the prediction of the full accuracy.

---

Published as a conference paper at ICLR 2026
A cluster is deemed extrapolatable if it meets two criteria: (1) its expected accuracy increases monotonically with model size, and (2) its performance converges to at least a predefined threshold P
(where P I accounts for practical limits like ambiguous questions or finite training coverage).
We filter out non-extrapolatable clusters using two rules based on the parameters from Eq.
1. Negligible accuracy growth, indicated by minimal or values.
Poor extrapolation reliability, indicated by an excessive e value.
In practice, for extrapolatable clusters, we set a > 1, 0.1, and 0 S c < 1. Further ablation experiments are provided in Appendix C.T
The clusters satisfying these conditions form the predictable subser. The final performance prediction for target model on this subset is the weighted average of the extrapolated predictions from these individual clusters, with weights proportional to cluster sizes.
4.4 MAPPING FROM PREDICTABLE SUBSET TO TARGET EVALUATION SET
We map predictions from the predictable subset Pp' to the complete evaluation set P using smooth function. This mapping strategy motivated by the observation that extrapolatable and non- extrapolatable samples, despite their difficulty differences, usually belong to the same question types, which implies consistent relative metric ordering between the predictable subset and full evaluation set. The mapping function Acc(P") - Acc(P) is continuous, smooth over [O, 1], monotonically increasing, and constrained to pass through (0, 0) and (1, Empirical validation indicates that smoothing spline optimally captures this relationship. Specifically, smoothing spline to model the mapping. where æ represents the average accuracy of the predictable subset P In practical implementation, under the premise of fixing the curve pass through [o, 0] and [1, 1], we determine the number of piecewise cubic segments (knots) by setting a Root Mean Square Error (RMSE) fitting threshold of 0.005. The number of segments dynamically adjusted until the fitting RMSE meets this threshold. We list the implementation details and visualization of mapping Sec. C.2
employ a cubic
To ensure reliability, we calibrate ff using evaluation results from existing models as anchors. This subset-to-full mapping generally demonstrates robustness across diverse model architectures and training data, often permitting the use of external models (e.g., Qwen2-72B (Yang et al. (022b)) as anchors for many tasks (see Appendix C.3 for experiments). The final metric prediction for target LLM with estimated training computation Co is then p f y(C)), combining the cluster-wise extrapolation y(C)) from Eq. (4) with the mapping f.
5
EXPERIMENTS
5.1
EXPERIMENTAL SETUPS
In our experimental setup, we train nine language models ranging from 122M to 70B parameters in total, which share the same data distribution and architecture, with the training data scaled proportionally to their sizes. We show the detailed training configurations and recipe in Appendix D
For evaluation, we adopt the following widely used benchmarks, including GSM8K Cobbe et al. 2021). MATH (Hendrycks et al. (021). BBH (Suzgun et al.) 2023). TriviaQA (Joshi et al.. 2017).
MBPP (Austin et al.) 2021), AGlEval (Zhong et al. 2024). DROP (Dua al. 2019), MMLUUprr (Wang et al. 2024). All models are evaluated in a few-shot in-context learning manner, and we aligned our evaluation setups with LLaMa3 (Dubey et al. 2024). We evaluate the proposed COD performance scaling for LLMs against existing approaches multiple public benchmarks. Using eight smaller language models as known information, we estimate the downstream task performance of pretrained LLM with 70B parameters.
5.2 PREDICTION EXPERIMENTS
We compare COD against four representative prediction methods:
7

---

Published as a conference paper at ICLR 2026
Table Absolute prediction error (%) on evaluation sets for predicting the performance of the 70B model. Errors < 2% are considered accurate (green), while errors > 5% are considered invalid (red). 4 indicates lower is better.
Overall Metrics
Individual Task Sets
Method
M
Max.
GSM8k MATH BBH
TriviaQA MBPP
AGIEval DROP MMI.Upre
I Loss-intermediate
5.29
9.39
9.39
6.95
2.33
5.81
5.52
1.41
5.37
5.55
End-to-end
3.10
6.00
4.00
3.86
0.64
0.68
1.75
6.00
4.11
3.72
cond-to-nn(passrattt
5.02
8.80
6.71
8.80
3.51
4.00
7.34
6.78
0.26
2.74
coddttoenndBNSSL
5.17
13.05
4.23
5.88
13.05
5.86
2.55
0.82
1.53
7.42
COD (w/o mapping)
2.24
5.26
4.70
0.50
2.91
1.98
0.89
5.26
1.08
0.57
COD (Complete)
1.55
2.68
2.68
0.79
0.47
1.97
2.42
1.64
1.05
1.39
1. Loss-intermediate (Chen et al. 2024):
First predicts the target LLM's final training or validation loss, then estimates downstream task metrics based on the relationship between smaller models' evaluation metrics and their losses.
2. End-to-ende End-to-end(exp) (Xiao et al. 2024): Directly extrapolates large model metrics from smaller model evaluation set metries using exponential-bae
performance scaling laws.
3. End-to-ene cadd-to-end(passaatt (Achiam et al. 2023 Hu et al., 2024): A variant of end-to-end method, which estimates large model passrates from smaller model passrates. We conduct 100 trials per evaluation set for smaller models enhance reliability and report absolute prediction error the passrate metric.
4. End-to-end BNSL) (Caballero et al. 2022): Decomposes the end-to-end mapping into multi- segment power-law framework.
also evaluate two variants our COD approach to validate the benefits of its components:
1. COD (w/o mapping): Performs difficulty- based KMeans clustering, extrapolates per cluster, and aggregates metrics without subset-to-fie mapping.
2. COD (Complete): Our full proposed multi- approach, including clustering, predictable cluster filtering, subset extrapolation, and subset-to-fin mapping.
Comparative results are shown Tab. Prediction accuracy is measured by the absolute error between predicted and actual performance. We report mean and max prediction enrors across all evaluation sets, as well as errors for individual sets. Our complete COD approach significantly outperforms existing methods in both mean (1.55%) and maximum (2.68%) prediction errors, offering reliable guidance for large model training. Although baseline methods achieve acceptable performance partial datasets, their large prediction errors other datasets severely compromise their overall reliability.
Fig. 4 visualizes the performance-complue relationships. The COD method does not merely extend the existing scaling trend; instead, it effectively predicts whether growth slowdown will occur subsequently and enables better estimation of the magnitude of curve bending. On the BBH dataset, while End-to-end(exp) and bos--intermediate approaches perform comparably to COD, they show poor fitting on small-model data. COD reveals more complex, better-fitted multi-phase trajectory. On MATH and MMLU-pro, where predicting accelerated growth versus plateaus is crucial, the doss-nntermediate method underestimates model ceilings, and two end-to-end methods exceeds 3% error. COD's s superior performance stems from its nuanced analysis of difficulty distributions and scaling laws, allowing it to predict growth challenging sets and capture diminishing returns in saturated sets.
5.3 ABLATION STUDY
To further validate the generalizability of the COD method, conduct ablation experiments different architecture, clustering method and extrapolation function, while the Appendix C we discuss ablation studies on the selection criteria for predictable subsets, interpolation mapping methods, and the influence anchor point on predictions. We also examine its predictive capabilities after continual training with results documented in Appendix
8

---

Published as a conference paper at ICLR 2026
GSMSK
MATH
BBH
TriviQA
Beaa hode 040 1091
light hode
Hight hote
0.1
0.2
15" TF"
7 7V" "gl"
"00"""""""""" "gg "g""""""
Compurett8)
CompureeBB)
CompurerB))
CompureBB)
MEBPP
ACtEval
DROP
MMLLl-pro
Small
hose
Bager hakel
Rigas Pestl
Pogad hotel
0.0
200 corrd
520 Duil)
0.2
Comsutet8e)
Computcr85)
Computo188)
Computoree)
Figure 4: Performance-commpute relationship for different prediction methods on eight evaluation sets. Table Absolute prediction error (%) on evaluation sets for predicting the performance of the 32B MoE model. Errors 2% are considered accurate (green), while errors 5% are considered invalid (red). & indicates lower is better.
Overall Metries
Individual Task Sets
Method
Mcan]
Max.
GSM8k MATH BBH TriviaQA MBPP AGIEval DROP
Loss-intermediate
3.65
7.24
0.45
0.62
4.48
0.36
4.92
7.24
4.55
End-to-end(exp)
3.95
7.86
2.75
1.43
3.88
1.72
7.86
7.79
2.21
COD (Complete)
8.11
0.54
1.72
5.29
0.27
8.11
0.57
5.24
5.3. PREDICTION ON MOE MODELS
COD relies on repeated sample-level evaluation and clustering, introducing additional computational overhead. However, difficulty characteristics are task-inherent and largely model-agnostic, suggesting that the resulting clusters can generalize across model families.
To test this transferability, conduct one-shot evaluations 32B activated--prrmmeter parameter MoE model using clusters derived from pre-trained dense models. Results Tab. show that COD achieves the lowest average prediction етror in crosstarchitecture extrapolation, indicating that its difficulty features and clusters transfer effectively across model families. Nevertheless, prediction accuracy is lower than dense-to-dense extrapolation. We hypothesize that aligning the model used for difficulty estimation with the target model for extrapolation reduces intra-cluster scaling discrepancies and improves prediction accuracy.
5.3.2 COMPARISON OF CLUSTERING METHOOS
We assess the impact of different clustering algorithms on prediction accuracy. The goal is achieve tight intra-clusted cluster difficulty similarity (low average distance to center) while maintaining cluster stability (min. 10 samples/cluster) We compare our Improved-Me Mernosed with DBScan, MeanShift, and K-Means. For K-Means, we adjust it to approximate our goals: (1) search for & yielding min. cluster sizes 10: (2) treat samples outside a radius (e.g., 2x average intra-cluster distance) from any cluster center as outliers, and ensure clusters don't 't drop below 10 samples. We term this "Improved- -Meedd for this comparison. Clustering quality measured by Intra-cluster Average Distance (LAD) and Outlier Rate (OR). Prediction benefits measured by Extrapolation Error (EE) on the predictable subset and Final prediction Error (FE) on the full evalset. (Tab.
Tab. shows Improved- Beeovved-MMaaan KMeans and Improved-MeanSSe to their intra-cluster distance constraints. The results also confirm these methods lead to lower EE and FE. Although Improved-KNN Meanns has the best IAD, performs poorly GSM8k, AGIEval, and
yield better clustering (lower IAD) due

---

Published as a conference paper at ICLR 2026
Table Clustering performance across different algorithms on metries including IAD (Intra-cluster Average Distance(), OR (Outlier Rate %), EE (Extrapolation Error]), FE (Final Prediction Error,).
The bottom lines show the mean and max EE and FE across evalsets.
KMeans
DBScun
McanShift
Improved-KMenn Improved-MeanShhit
Dataset
IAD
OR
EE
FE
IAD
OR
EE
FE
IAD
OR
EE
FE
IAD
OR
EE
FE
[AAD
OR
EE
FE
GSMSK
0.22
0.00
0.51
0.53
4.08
4.12
0.29
0.61
0.67
0.74
0.13
2.73
3,92
4.08
0.16
7.05
2.68
2.34
4.38
4.16
0.21
144
2.55
0.09
2.22
0.11
0.79
3.53
4.36
0.27
20.72
2.12
0.20
37.23
0.02
2.17
0.47
TriviaQA
0.44
2.97
2.46
0.70
6.38
1.11
0.81
0.25
6.77
3.64
4.90
0.12
11.97
1.18 1.12
0.19
11.54
1.56 1,97
MBPP
0.34
2.53
2.67
0.51
12.80
1,57
1.41
0.22
15.60
2.40
1.22
0.17
19.40
2.39
3.25
0.17
21.60
1.61
2.42
AGIEval
0.46
2.61
2.68
0.56
3.67
6.43
6.27
0.29
2.99
2.63
3.23
0.15
7.60
5.96
5.56
0.21
11.50
1.11 1.64
DROP
0.56
1.66
1.64
0.67
11.08
3.03
2.66
0.25
11.81
4.18
4.00
0.14
21.42
3.99
5.24
0.20
19.88
1.44
1.05
MMLU-pro
0.32
3.69
3.69
0.42
0.56
3.72
3.69
0.29
0.39
3,15
3.08
0.16
2.85
0.56
0.61
0.22
4.40
1.26
1.39
Mean
3.03
3.06
3.48
6.43
2.67
2.51
2.35
2.82
1.08 1.55
Max
8.16 8.99
4.38 4.36
4.18 4.90
5,96
5.56
1.61 2.68
Table 4: Ablation study on extrapolation formulas. EE, TR, FE shown for BBH, MATH, MMLUUpro.
BBH
MATH
MMLU-pro
Method
EE|
TR(%)
FEL
TR(%)
FE|
EE|
TR(%)
FE
w/o Random Guess (f1)
10.40
48.29
11.65
3.96
76.82
3.22
4.40
95.05
4.32
w/o Constant с (f2)
2.15
57.21
4.10
1.50
76.82
1.36
95.60
3.96
Direct Power Law (fa)
8.90
49.05
8.85
3.33
76.82
2.70
4.30
95.15
4.20
Ours (f)
0.54
53.39
0.47
0.84
76.82
0.79
1.26
94.38
1.39
DROP. This is likely because K-Means requires pre-specifying kt and our search for & can be unstable, leading to large errors some sets. In contrast, -ManoSdf- which automatically determines based on distance constraints, offers more stable clustering and the lowest maximum prediction error.
5.3.3 DIFFERENT EXTRAPOLATION FORMULAS
We ablate our proposed fitting formula f(C) = = - -a0 (Ours) by removing or
modifying components: (1) (C) -0 bcc (w/o random guess); (2) (2(C)
(1 g)
eaC (w/o constant c); (3) fs(C)= - (Direct Power Law (Hu et al. 2024 Tab. shows Extrapolation Error (EE), Task Ratio of predictable subset (TR), and Final prediction Error (FE). Our proposed formula consistently achieves the lowest EE and FE. fi struggles with finite-answer tasks where small models have non-zero scores. f2 inaccurately assumes perfect scores are attainable, ignoring data limitations and task ambiguities. The direct power law (/a) fails to model the 0-1 metric range and the varying difficulty of improvement near random guess and saturation. The weak correlation between TR and prediction eиror demonstrates the robustness our COD framework: when the proportion of the predictable subset is low due non-emergent tasks, the performance of looapottapol tasks can still accurately inferred through the proposed mapping function.
CONCLUSION AND DISCUSSION
In this work, we propose novel framework for predicting LLM downstream performance scaling, which consists three key contributions: (1) the COD framework that effectively models the intrinsic diverse scaling patterns of tasks in the evaluation set; (2) scaling law for downstream extrapolation; and (3)
task performance that provides a fitting formula for performance-compun systematic methodology for identifying and leveraging predictable subset that provides robust intermediate metric for accurate full-set performance predictions. We discuss the limitations and future works in Appendix H
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
We have read and adhered to the ICLR Code of Ethics. This work proposes a computational framework to enable more efficient resource allocation in the training of LLMs. The research is methodological nature and aims to support more sustainable and responsible practices within the field of Al development.
We provide a detailed account of our methods, theoretical proofs, and experimental settings. We openly diseuss the limitations of our framework in Appendix H. This study does not involve human subjects or the use of sensitive personal data.
We utilized language models the writing level, including checking for grammatical errors in the article and modifying expressions. The use of language models had no impact on the article's innovative contributions, experiments, or analytical perspectives.
REPRODUCIBILITY STATEMENT
We have made extensive efforts to ensure the reproducibility of our work. The core methodology, the
Clustering- On- In-Difficulty (COD) framework, is detailed in Sec. | The improved MeanShift clustering
algorithm is described in Sec. 4.1, with full pseudocode provided in Appendix A.. (Algorithm 1).
Our performance scaling law (Theorem 1) is presented in Sec. 4.2, with a complete proof available
in Sec.
B We discuss the additional computational cost of the COD method in Appendix G
Our experimental setup, including model architectures, training data philosophy. and hyperparameters. thoroughly documented in Sec. [.] and Appendix D). with specific model configurations listed in Tab. A4 The evaluation benchmarks, protocols, and few-shot settings are described in Sec. and summarized in Tab. A5 Extensive ablation studies validating our component choices-iiclluiigg extrapolation formulas (Sec. 5.3.3), clustering algorithms (Appendix 5.3.2). interpolation methods (Appendix C.2), and criteria for filtering clusters (Appendix C.T) are provided to support our findings. We visualize the task difficulty distribution for each evalset in Appendix F
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXivv2303.08771 2023.
Joshua Ainslie, James Lee-Thorp. Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprini arXiv::305. 13245, 2023.
Ibrahim M Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. Revisiting neural scaling laws in language and vision. Adv. Neural Inform. Process. Syst. (NeurIPS), 35:22300-2233 22312, 2022.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, al. Program synthesis with large language models. arXiv preprint wwXXi:2108.007333 2021.
Hritik Bansal, Arian Hosseini, Rishabh Agarwal, Vinh Q Tran, and Mehran Kazemi. Smaller, weaker, yet better: Training Ilm reasoners via compute-optimal sampling. In The 4th Workshop on Mathematical Reasoning and Al at NeurIPS, 2024.
Akshita Bhagia, Jiacheng Liu, Alexander Wettig, David Heineman, Oyvind Tafjord, Ananya Harsh Jha, Luca Soldaini, Noah A Smith, Dirk Groeneveld, Pang Wei Koh, al. Establishing task scaling laws via compute-efficient model ladders. arXiv preprint arXiv:2412.044400 2024.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint 2303 2023.
Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger. Broken neural scaling laws. arXiv preprint arXiv:2210.1444 14891, 2022.
11

---

Published as a conference paper at ICLR 2026
Yangyi Chen, Binxuan Huang, Yifan Gao, Zhengyang Wang, Jingfeng Yang, and Heng Ji. Scaling laws for predicting downstream performance llms. arXiv preprint arXiv:2410.08527, 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110 20::22100.1100 14168, 2021.
DeepSeek-AL et al. Deepseek-v3 technical report, 2025. URL https://arxiv..rg/////a 2412. 19377
Zhengxiao Du, Aohan Zeng, Yuxiao Dong, and Jie Tang. Understanding emergent abilities of language models from the loss perspective. arXiv preprint arXiv:2403. 15796, 2024.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In NAACI-HLT; pages 2368-2377 2019.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXhv:2407.21785, 2024.
Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. KDD, volume 96, pages 226-231, 1996.
Keinosuke Fukunaga and Larry Hostetler. The estimation of the gradient of a density function, with applications in pattern recognition. IEEE Transactions on information theory, 21(1):32-40, 1975.
Samir Yitzhak Gadre, Georgios Smymiis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, et al. Language models scale reliably with over-training and on downstream tasks. arXiv preprint arXiv:2403..08 085500 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et The llama herd of models. arXiv preprint arXiv:2407.2178. 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning. arXiv preprins arXiv:2501. 2025.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Adv. Neural Inform. Process. Syst. (NeurIPS), 2021.
Tom Henighan, Jared Kaplan, Maxwell Katz, Anselm Levskaya, Sam McCandlish, Andreas Stuhlmuller, Scott Gray, and Dario Amodei. Scaling laws for autoregressive generative mod- eling. arXiv preprint arXiv:2010. 14701, 2020.
Danny Hemandeez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint 1rx.v:2102.01129, 2021.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optiman large language models. arXiv preprint arXiv: 2022.
Shengding Hu, Xin Liu, Xu Han, Xinrong Zhang, Chaoqun He, Weilin Zhao Yankai Lin, Ning Ding, Zebin Ou, Guoyang Zeng, et al. Predicting emergent abilities with infinite resolution evaluation. In Int. Conf. Learn. Rep. (ICLR), 2024.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Anmual Meeting of the Association for Computational Linguistics, pages 1601-1611, 2017.
12

---

Published as a conference paper at ICLR 2026
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint wrx.f:20001.8861,,, 2020.
Lucas Lingle. large-scale exploration of j-transfer. arXiv preprini arXiv:2404.057228 2024.
Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang. Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of- experts language model. arXiv preprint arXiv:2405.0445% 044454 2024.
Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Fineweb-edu: the finest collection of educational content, 2024.
Qian Ma, Haitao Mao, Jingzhe Liu, Zhchua Zhang, Chunlin Feng. Yu Song, Yihan Shao, and Yao Ma.
Do neural scaling laws exist on graph self-supervised learning? arXiv preprint IrXi:2448.112433 2024.
David Owen. How predictable is language model benchmark performance? arXiv preprint arXiv:2401.04757 2024.
Yangjun Ruan, Chris J Maddison, and Tatsunori B Hashimoto. Observational scaling laws and the predictability of langauge model performance. Advances in Neural Information Processing Systems, 37:15841-15892. 2024.
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models mirage? In Adv. Neural Inform. Process. Syst. (NeurIPS), 2023.
Christin Seifert, Jörg Schlötterer, et al. Ceval: A benchmark for evaluating counterfactual text generation. In International Natural Language Generation Conference, pages 55 55-69, 2024.
Rulin Shao, Jacqueline He, Akari Asai, Weijia Shi, Tim Dettmers, Sewon Min, Luke Zettlemoyer, and Pang Wei Koh. Scaling retrieval-based language models with trillion-token datastore. arXiv preprint arXiv:2407. 2024.
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202. 2020.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling 11m test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408. 03314, 2024.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputiig, 568: 127063, 2024.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe Le, Ed Chi, Denny Zhou, et al. Challenging big-bench tasks and whether srain-of--thougee can solve them. In Findings of the Association for Computational Linguistics, pages 18003-13051 2023.
Chaofan Tao, Qian Liu, Longxu Dou, Niklas Muennighoff, Zhongwei Wan, Ping Luo, Min Lin, and Ngai Wong. Scaling laws with vocabulary: Larger models deserve larger vocabularies. arXiv preprint arXiv:2407.13623, 2024.
Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler. Scale efficiently: Insights from pretraining and finetuning transformers. lnt. Conf. Learn. Rep. (ICLR), 2022.
Yubo Wang, Xueguang Ma, Ge Zhang. Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint 2406:2406001574 2024.
Jason Wei, Yi Tay, Rishi Bommasani, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.072 07682, 2022.
Chaojun Xiao, Jie Cai, Weilin Zhao, Guoyang Zeng. Xu Han, Zhiyuan Liu, and Maosong Sun. Densing law of lms.. arXiv preprint 15X1v:24/2.04315, 2024.
13

---

Published as a conference paper at ICLR 2026
An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui. Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint 1rX1v:2412.15115, 2024a.
Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024b.
Greg Yang. Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2203.03466, 2022.
Qinyuan Ye, Harvey Fu, Xiang Ren, and Robin Jia. How predictable are large language model capabilities? a case study on big-bench. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7493-7517, 2023.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In IEEE Conf. Comput. Vis. Pattern Recog. (CVPR), pages 1204-1213, 2022.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in neural informa- fion processing systems, 32, 2019.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang. Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centrie benchmark for evaluating foundation models. In Findings of the Association for Computational Linguistics, pages 2299-2314 2024.
14

---

Published as a conference paper at ICLR 2026
CONTENTS
Introduction
2
Related Work
2.1
Loss Scaling Laws
Downstream Task Performance Scaling
3
Pilot Study
4
Method
Clustering on Difficulty
Fitting
Extrapolation
Mapping from Predictable Subset to Tärget Evaluation Set
Experiments
Experimental Sctups
5.2
Prediction Experiments
5.3
Ablation Study
8
5.3.1
Prediction Moh models
5.3.2
Comparison of Clustering Methods
5.3.3
Different Extrapolation Formulas
10
6
Conclusion and Discussion
10
Improvements Clustering Algorithm
16
Improved MeanShift Algorithm
16
A.2 Smoothing Techniques
17
Proof Theorem
17
Additional Ablation Studies
19
The Criteria for Extrapolatable Subsets
19
Mapping Method
20
ncorporating Anchor Point Interpolation Mapping
20
Experimental Settings and Training Recipe
21
Performance Prediction for Continue-Pretrainee LLMS
22
Difficulty Distribution of Predictable Subset
23
Computational Cost
23
Limitations
24
15

---

Published as a conference paper at ICLR 2026
A IMPROVEMENTS OF CLUSTERING ALGORITHM
A.1 IMPROVED MEANSHIFT ALGORITHM
We iteratively apply the MeanShift algorithm with an adaptive cluster radius R and a minimum cluster size K. In each iteration, for the clustered samples, we examine whether the distance between each sample and its cluster center exceeds R, and relabel those samples that exceed this threshold as unclustered. For clusters containing fewer than K samples, we mark all samples in these clusters as unclustered. At the end of each iteration, we incorporate both the outliers from McanShift and our marked unclustered samples into the next round of clustering, continuing this process until no further changes occur in sample labels. We present the pseudocode in Algorithm 1
Algorithm 1 Iterative MeanShift Clustering Algorithm
1:
Calculate adaptive radius: R = min(estinmas)
(estimate bandwidth((Q)UU)
2: Initialize all labels in the evaluation set to -1
3: repeat
4: Perform MeanShift clustering with radius on all samples labeled -1
5: Assign new labels to clustered samples for each newly labeled sample do Calculate distance dist, to its cluster center if dist; >> R then
6:
7:
8:
9:
Reset label -1
10: end if
end for
12:
for each cluster do
13:
number of samples in cluster < K then Reset all samples in this cluster to -1 end end for
14:
15:
16:
17:
Renumber all non-(-1} newly labeled samples to avoid overlap with old labels
18:
until no label changes
In the experiment, is empirically set to 10, which has been verified through extensive experi- ments to be reasonable and robust value. To determine the clustering radius R, we employ the estimate estimate_bandwidtt function from the ditearn.clustee library. This utility automatically computes a bandwidth value that balances clustering granularity and stability based on the underlying distribution of the data. The function is governed by a quantile hyperparameter Q, which typically ranges from 0.1 to 0.5. Given our framework's ' stringent requirements for minimizing intra-cluster variance, we adopt a conservative value of Q = 0. in practice. Furthermore, considering that sample sizes and difficulty distributions vary significantly across different evaluation sets, the automatically estimated bandwidth may occasionally become excessively large. To mitigate this risk, we introduce a global upper bound U on the bandwidth. Specifically, we define represents the theoretical maximum distance between any two vectors in the difficulty feature space and 10 empirical constant. In our experimental setup, the feature vectors are 9-dimensional with each element inherently bounded within the range [0, 1]. Consequently, the maximum possible Euclidean distance is calculated as 2 x 9 = 3. yielding an upper bound of U = 0.3.
max max__dssten citesdddssccceeeee 10. where max_ max_distance distance
Filtering zero-performance samples. In the evaluation set, there may exist few extremely difficult problems that require sufficient model parameters to emerge. All small models may fail to solve these problems even after 100 evaluation attempts, resulting in difficulty feature vectors of all zeros. We refer to these as zero-performance samples. Their presence leads to two issues:
Zero performance on small models does not necessarily indicate zero accuracy on large models. For these samples, we cannot estimate when emergence will occur predict large model metrics.
2. During clustering, they may be confused with other low-performing but non-zero samples. Including them in the same cluster would lower the expected accuracy of that cluster, leading to inaccurate fitting and extrapolation later.
16

---

Published as a conference paper at ICLR 2026
Therefore, pre-filter these zero- prro-performance samples before clustering, treating them outliers that do not participate the clustering process. This approach obviates the necessity of considering their metrics under large models during subsequent extrapolation, and prevents disruption to the clustering of the remaining samples.
A.2 SMOOTHING TECHNIQUES
Metric fluctuations of individual samples in downstream tasks are not solely due limited sampling. Another potential factor is noise from uneven data distribution in recent training batches. Therefore, in addition to performing 100 evaluations to mitigate sampling variance, we evaluated 100 times on each of the adjacent checkpoints before and after the selected model. We then averaged these accuracy expectation values across three checkpoints, further reducing sampling variance while offsetting noise from uneven training data distribution. This approach also reduces the number of zero-po zero-performe rero-performance samples, further improving clustering and prediction effectiveness.
B PROOF OF THEOREM
use Lemma ! to derive scaling law downstream task performance (Theorem D.
Lemma 1 (Arithmetic--eomeeti mean difference). For any sequence of positive real numbers (ri)eer let:
Pa = Eis Ij be the arithmetic mean;
My IIi= T. 1/n be the geometric mean;
=²= Ci
- w)2 be the variance.
Then the difference between the arithmetic mean and geometric can be estimated as:
1
o²
A Pa P1
II
+ o(o2).
(5)
214a
1m1
Proof. Taking the logarithm of the geometric mean Pg:
log(pg)
log Tj.
(6)
n
Using Taylor expansion of log x around μc:
(x 11)
log x = log u
- 1
(
24²
We can simplify:
n
(I; Ma)
(x, Ma)2
log n4 log 1
+ ((1; [ka)2)
P/a
2/12
i=1
1
n2
1
=log
Pa
Ma)2
to
Σ
[ mnm
1
2n22
71
m
equal
Ø2
o² =log 1
+ o(02).
2n²
Therefore:
exp (- (-a23
))
Pa
14g 1a
241²
+o(02)
(8)
17

---

Published as a conference paper at ICLR 2026
When o² 2nd is small, this can be approximated as:
A o² 2/1a
Theorem A1 (Scaling Law for Downstream Task Performance). Consider a language model Mc trained with compute budget C and set of downstream tasks P. Under the following assumptions: Assumption 1 (Power-law scaling of answer loss): the expected answer loss follows:
Lp(C) == ((g.am))>-p [[q, Gone C)] = aC +7,
(10)
where 3,, > 0 are task-specific constants, with representing the irreducible loss.
Assumption (Unique deterministic answers): Each question has a unique deterministic answer. The model receives score 1 if and only if Mc outputs Oune and otherwise.
Assumption 3 (Accuracy decompoxition): The expected accuracy decomposes as:
[cp- = = (g >-P [p( Osroee 19, Mc)),
(11)
where E [0, is the random guessing baseline.
Then, the expected accuracy on task P can be modeled as:
oI(C) (o(((C))
Ep[Acc(C]] -9 (1 exp (-a0- 2,r(C))
(12)
where (C)(C)
B(g.oon)~pl LL aeree' C)l is the mean loss and of(C)
Var (oom)) (- arree's C)] is the loss variance across the task set.
Proof. For a task T = (q. Omne) P, under assumption 2, ame is deterministic and unique, thus p(amel Mc) can be decomposed into token-wise auto-regressive loss.
k
- -lo Mc)) = log
II
(13)
log (p(talg,ten:
(14)
L(g, Grrec; C).
(15)
Then take the exponential of both sides, and then take the expectation with respect to different tasks in the evaluation set p (g, ans) E P. We note that both Pans and lossans are functions of C..
Ep( 44, Mc)] Ep [eleeep-Le Grue: C)]
Σ
exp(-L(g Qrue C)). (16)
(q.amx))
We can adopt Lemma 1 to switch from arithmetic mean to geometric mean of loss, and apply the power law assumption
(C( + 2pz(C)
Σ
exp(-L(g, amei ()) = exp
Σ
L(g, Gmee C)
+ +o (o?((C)
71
71
EE
(g,00me E
loss scaling law
(17)
rf(C) (Cz(C)) + (
= (xap(-aC
(18)
where in the number of tasks in P. and μr, a² follow definitions in the proposition.
18

---

Published as a conference paper at ICLR 2026
Table Al: Prediction errors (EEJ, FEJ) across criteria of extrapolatable subsets.
Baseline
Ablate
Ablate b 0.05)
Ablate
w/o threshold
Metric/ Task Set
(a 1, b 0.1,
<
(a 0.5)
< 0.5)
EEI
FE4
FE
EEL
FEL
EEC
FEL
EE]
FEJ
GSM8k
0.31
2.68
0.31 2.68 0.31
2.68
0.31
2.68
4.05
4.35
MATH
0.84
0.79
0.84 0.79
0.84
0.79
1.04
0.94
0.40
0.38
BBH
0.54
0.47
0.54
0.54
0.47
0.17
5.33
TriviaQA
1.56
1.97
1.57 1.96
1.56
1.97
1.05
1.42
2.82
3.77
MBPP
1.61
2.42
1.61 2.42
1.61
2,42
1.61
2.42
1.55
1.73
AGIEval
1.11
1,64
1.11
1.64
1.11
1.64
0.08
1.35
2.24
2.38
DROP
1.44
1.05
0.24
1.05
0.29
0.96
1.83
MMLU-pm
1.26
1.39
1,26
1.39
1.26
1.39
2.88
3.27
1.00
1.10
Mean
1.08
1.55
0.94
1.55
0.94
1.54
1.11
2.21
2.45
2.61
Max
1.61
2.68
1.61
2.68
1.61
2.68
2.88
3.59
5.39
5.33
Finally, assumption 3 to align the answer passrate and the accuracy metric.
Br-p(Acc(C)] + (1 g) P ppp(pprpe / 4 Mc))
(19)
= (1 g)
exp(- (0(-I(( True C))
(20)
(g.ame) EPP
g + (1 -9) exp (exp(-aC-0 -)) 20(
2p) +0(((C))
(21)
Rationality of Assumption 3 Assumption is designed to accommodate tasks with finite swer sets. For such tasks, when calculating Acc(C), possibilities outside the answer set are disre- garded. When p(Onue Mc) approaches O, Acc(C') is the level of random guess, g. When (rree Mc.) approaches 1, Acc(C") is close to (anue g. .M..). This assumption implies a linear relationship between Acc(C') and the probability in the (0, 1) interval. The theorem itself is also effective for tasks with open answer sets, where the probability of a correct random guess can be assumed be (i.c.. g 0).
C ADDITIONAL ABLATION STUDIES
C.1 THE CRITERIA FOR EXTRAPOLATABLE SUBSETS
The criteria for fitting the extrapolation formula (Eq. are designed to ensure the following:
I a > 0 and b >> 0: These ensure that accuracy is an increasing function of compute. Larger values of and signify that task performance scales more distinctly with compute, leading fitting curves with better scaling properties and differentiability.
2 с 2 0: This ensures the extrapolated curve's maximum value is less than or equal to An excessively large implies that the fitting curve has very low ceiling, which is characteristic of task subsets with poor scaling properties. These are thus considered non-extrapolatable.
We conducted an ablation study on these parameters, as shown in Tab. A1 Starting from our baseline criteria (a > b > 0, 0 S e 1), we individually relaxed the constraints on a, b, and c, and also observed the effect of removing the thresholds entirely.
When the thresholds are removed entirely, the prediction performance degrades significantly. This because numerous task clusters with poor scaling properties are included in the extrapolation, impairing the overall result. In contrast, individually relaxing the thresholds for a, b, or still largely preserves the integrity of the filtering criteria. The performance shows remains nearly identical or only slightly decreases compared to the baseline, indicating that while the filtering step is important, method is not overly sensitive to the specific threshold values.
19

---

Published as a conference paper at ICLR 2026
Suftor to Fullett Rogslnn
Safoce th Fullict poggiing
Sabore to Fullett Miggeii
Sabbct to Pullict Hopplig
33
Clabar A00
Clatar ACC
Cluarr AGa
(a) 7 0.0025
(b) 7 0..05
(c) T 0.01
(d) T 0.02
Figure Al: Ablation of RMSE Threshold T,,
C.2 MAPPING METHOD
In the Mapping stage, we map the metries of the predictable subset to those of the full set, where a cubic smoothing spline is employed to model this relationship. During fitting, adjust the noise scale a to control the number of segments in the spline. After fitting, we calculate the root mean square error (RMSE). If the RMSE is below a predefined threshold, the fitting process terminates; otherwise, we further decrease a to induce more segments until the RMSE threshold is met.
We demonstrate the impact of different RMSE thresholds on the fitting performance in Fig. A1 It can be observed that the curve exhibits significant overfitting when T = 0.0025, whereas the fitted curve deviates from the target points when T 0.02. Consequently, as the RMSE threshold across all evaluation sets to achieve the optimal fitting performance.
uniformly adopt T = 0.005
C.3 INCORPORATING ANCHOR POINT IN INTERPOLATION MAPPING
We find that the mapping relationship from predictable subset metrics to full evaluation set metrics is similar across models with different training data and architectures. This allows leveraging pre- trained models "anchors" to refine the mapping and improve final estimation accuracy. In practice, we simply use an open-source model (Qwen2-72B) as a refinement anchor. We first derive an interpolation curve using only small model metries and fixed points 0), (1, 1), then assess anchor compatibility. This shared mapping implies estimable subset metrics are highly correlated with full-set metrics and less prone to interference from other model parameters than -ost-nntermeeiaee predictions.
We test two configurations:
COD (w/o anchor): Full COD pipeline, but no anchor points in the mapping phase;
COD (w. anchor): COD method with Qwen2-72B as a refinement anchor;
We list the performance of anchor models and the target model Tab, A2 In Fig. A2
we also compare the differences of using anchor points in the Mapping stage against not using them two evaluation datasets: MATH and MMLU-pro. The MATH dataset shows a clear discrepancy between the predictable subset and the full set, and adding anchor points significantly improves the fitting and prediction performance. In contrast, the MMI.U-rro dataset has a high proportion of predictable instances (listed in Sec. (53.3). and its metrics for the predictable subset are close to those of the full set, so introducing anchor points yields little difference.
Table A2: Performance comparison among the target model and anchor models.
Model
GSMSK
MATH
BBH
TriviaQA
MBPP
AGIEval
DROP
MMLU-pro
70B-Dense
88.55
48.02
81.69
80.66
68.00
58,20
76.82 77.56
57.28
Qwen2-72B
88.63
53.08
80.
84.23
71.60
64.16
56.93
To facilitate direct comparison, also apply the anchor point to baseline methods including End-to-ent -edd-t-end(exp) and (BNSSL, using the same anchor as the COD method. In practice, the anchor points are directly incorporated into the fitting process of the extrapolation formula.
20

---

Published as a conference paper at ICLR 2026
Subeeb Fille
Suzeen hultst Papsing
Submee Filient
Subeeb T.-.T.T.. Mazzing
02
Clistr Accemty
Actuney
Clarter Accomeo
(a) MATH w/o anchor
(b) MATH w/ anchor
(c) MMLUpro w/o anchor
(d) MMLUpro w/ anchor
Figure A2: Effectiveness of mapping with or without anchor points.
Table A3: Influence of anchor points in the mapping stage on prediction error (%). Errors 2% are considered accurate (green), while errors 5% are considered invalid (red). indicates lower better.
Overall
Individual Task Sets
Setting
Method
Mcan.,
Max]
GSM8k
MATH
BBH
TriviaQA
MBPP
AGIEval
DROP
MMLU-pro
w/o anchor
3.10
6.00
4.00
3.86
0.64
0.68
1.75
6.00
4.11
3.72
End-to-end(exp)
w/ anchor
3,08
6.42
0.56
5.75
0.93
2.25
2.82
6.42
1.10
4.80
w/o anchor
5.17
13.05
4.23
5.88
13.05
5.86
2.55
0.82
1.53
7.42
End-to-cnd(BNSL)
w/ anchor
3.60
6.65
0.18
5.77
1.28
4.17
4.22
6.65
1.43
5.10
w/o. anchor
2.65
4.98
3.10
3.99
0.97
2.38
1.59
4.98
2.86
1.32
COD
anchor
1.55
2.68
0.79
0.47
2.42
Tab. A3 shows that the COD method incorporating anchor consistently enhances prediction accuracy. However, the End-to-end(exp) and End-to-end(BNSL) baselines failed derive any benefit from the addition of anchor points. This suggests stable correlation between the predictable subset and full-set metrics across diverse models, enabling the use of existing model evaluations to improve predictions for models. In contrast. merely as fitting samples, aligning the prediction target solely with the scaling trend of the anchors. Yet, scaling trends differ significantly across models trained different data and architectures, manifesting as high variance across different capability dimensions; consequently, these methods fail produce effective estimations.
and
treat anchor points
Furthermore, since our clustering identifies intrinsic properties of evaluation sets, the derived pre- dictable subsets applicable new models.
D EXPERIMENTAL SETTINGS AND TRAINING RECIPE
Training recipe. To establish performance predictions for large language models, we conduct systematic experiments with a suite of smaller-scale models across different parameter counts. All our models are trained from scratch on a corpus of text data. We do not fix the data budget for all models; instead, we maintain a consistent Data-to-CPT (Compute Per Token) ratio for all models, as mentioned in Sec. 5. We list model configurations in Tab. A4
We adopt the in-house training data that comprises multilingual text corpora, with increased weighting for domains such as STEM, code, and general knowledge, following Llama3 (Grattafiori et al. 2024), Deepseck-v2 (Liu et al. 2024), Fineweb-EDU (Lozhkov et al. 2024), etc. We apply several de- duplication methods and data cleaning mechanisms to each data source to ensure high-quality tokens.
The model architecture is consistent with Llama3.1 (Grattafiori et al. 2024), incorporating Grouped- Query Attention (GQA) (Ainslie et al. 2023). SwiGLU activation function (Shazeer 2020). RM- SNorm (Zhang and Sennrich 2019) with Pre- Pre-normalization. etc. The models trained using BF16 precision with a sequence length of 8192 and a RoPE Su et al.] (2024) base of 500, 000. We employ the AdamW W optimizer with (0.9, 0.95), a weight decay of 0.1, and dropout rate of 0.1.
21

---

Published as a conference paper at ICLR 2026
Table A4: Model architecture specifications across different sizes.
122M
238M
41
652M
973M
1.9B
7B
12B
70B (Target)
Param. (M)
122
238
411
652
973
1,901
6,980
12,022
68,452
Compute Per Token (B)
1.535
2.684
4.275
6.378
9.060
16.436
54.761
91.609
475.131
Tokens (B)
45
108
153
923
1,544
8,012
Continue-Trainne Tokens (B)
3
5
12
18
33
114
191
1,000
Model Dimension
1,024
1,280
1,536
1,792
2,048
2,560
4,096
4,608
8,192
FFN Dimension
3,584
4,480
5,376
6,272
7.168
8,960
14,336
16,128
28,672
Heads
10
14
32
36
KV Heads
8
10
12
14
16
20
8
12
Table A5: Information of evaluation datasets.
Dataset
Domain
#Questions
#Shots
GSM8K
Math
1,319
MATH
Math
5,000
4
BBH
Reasoning
6,511
TriviaQA
Knowledge
17.944
MBPP
Coding
500
AGIEval
Comprehensive
8,063
DROP
Reading
9,536
MMLU-pro
Comprehensive
12,032
All models trained constant learning rate scheduler with few-step warmup stage. To determine the learning rate and batch size, we adopt the hyperparameter scaling laws from Liu et al.] (2024). Specifically, the optimal learning rate Yoopt and the optimal batch size Bops defined as power laws of the compute, measured in FLOPs: Nopt 01 C-b: and Bopt 02 Cd, where 01, b,. 02, b2 are parameters to be fitted. We perform a grid search on our small models to identify their optimal learning rates and batch sizes, and then extrapolate these findings to the bigger models.
Training Resources. The 7B dense model is trained on 923B tokens, consuming 52,800 H800 GPU-hours. The computational resources used for the other models can be estimated proportionally based on their respective compute requirements.
Evaluation settings and protocol. We conducted performance scaling estimation experiments across cight major LLM evaluation sets. These evaluation sets span a diverse range of capabilities, including Math, Reasoning, Knowledge, Coding, Reading, and general abilities. All pretrained LLMs were evaluated using a few-shot methodology to obtain the performance metrics. Detailed information is provided in Tab. A5 Our evaluation methodology aligns with that used for the Llama3 (Grattafioni et al. 2024) pre-trained models. We assess the models' capabilities directly through few-shot text completion tasks without any instruction tuning or Supervised Fine- Tuning (SFT). This evaluation method is chosen because a small amount of SFT data can significantly influence performance on downstream tasks, thereby not reflecting the inherent capabilities of the pre-trained model itself.
Software Framework. All models trained using the Megatron framework. The evaluation code an in-house implementation designed to consistent with the Llama3 (Grattafiori et al.) 2024) evaluation methodology.
E PERFORMANCE PREDICTION FOR CONTINUE-PRETRAINE LLMS
Leading industry pre-trained LLMs (e.g.. Deepseek-v3 (DeepSek-AA et al.- 2025).
Llama3 (Grattafiori et al., 2024), Qwen-2.5 (Yang al., 2024a)) adopt the Continual Training (CT) strategy of concentrating high-quality data towards the end of the pre-training process. This phase is typically accompanied by learning rate decay, enabling the model to fully absorb this high-quality data. Due significant changes in data distribution and the learning rate schedule, this approach often yields substantial improvements in metrics. Predicting a large model's final capability based solely on its performance during "stable" phase with consistent data distribution does not
22

---

Published as a conference paper at ICLR 2026
reflect its ultimate capability. Therefore, supplement this by providing metric predictions for the high-quality CT phase.
The relationship between model parameter scale and the volume of CT tokens is listed in Tab. A4 We conduct the same COD pipeline for CT models. We control the data distribution of the stable and decay phases for various smaller models, as well as their token-to- te--aarrmeerr ratio, be consistent with the large model targeted for prediction. The last checkpoint is used for evaluation. Based on prior clustering labels, we perform fitting, extrapolation, and mapping obtain the predicted performance for the large model.
token-to-parameter
Table A6: Predicted vs. actual metrics for an LLM with 70B parameters after high-quality continued pretraining. Errors 2% are considered accurate (green)
Evaluation Set
Predicted Metric
Actual Metric
Prediction Error
GSM8k
93.10
91.81
1.29
MATH
56.35
52.68
3.67
BBH
83.05
85.32
2.27
TriviaQA
79.29
84.05
4.76
MBPP
72.42
73,20
0.78
AGIEval
63.22
64.18
0.96
DROP
82.34
81.39
0.95
MMLU-pro
62.11
59.34
2.77
Results listed in Tab. A6 show that the proposed COD method achieves average prediction of 28%. We observe that MATH and TriviaQA exhibit relatively large prediction errors. We hypothesize that there are two main categories of reasons for this inaccuracy:
The CT data and the evaluation sets possess significant correlation. For example, math-related evaluation sets, modest amount of training can yield substantial improvements in performance metrics. such scenarios, the metrics for smaller models tend show greater volatility and have inaccurate evaluations.
2. The CT data exhibits inherent distribution bias, such that certain evaluation sets, such as TriviaQA, do not derive performance gains from it. This leads potential significant fluctuations in the metrics after the CT phase, thereby diminishing the accuracy of extrapolating to larger models.
F DIFFICULTY DISTRIBUTION OF PREDICTABLE SUBSET
We analyze the proportion of predictable subset tasks across different difficulty levels. The difficulty distributions of predictable subset versus complete sets for different evaluation benchmarks are illustrated in Fig. A3 We use the scores from the 12B model as the basis for difficulty classification. The results show that MMLU-pro and GSM8k evaluation sets have larger proportions of predictable subsets, indicating that most questions in these datasets exhibit good performance scaling properties. In contrast, many difficult questions with near-zero scores in the MATH evaluation set fall outside the predictable subset, requiring adjustment during the mapping phase. Meanwhile, BBH exhibits consistent proportions of its predictable subset across varying difficulty levels, as some questions display oscillatory patterns with limited improvement, even with increased computational resources.
The proportion the predictable subset can serve as a metric for assessing evaluation set quality. Evaluation sets with larger predictable subsets yield more reliable experimental conclusions from smaller models. When constructing evaluation sets, we recommend screening or supplementing unpredictable clusters and ensuring minimum number of questions for each difficulty feature to reduce metric volatility.
G COMPUTATIONAL COST
The extra computational overhead of running COD is not expensive compared to training a series of small models with increasing parameter sizes. The main additional cost comes from performing 100
23

---

Published as a conference paper at ICLR 2026
32r1
MATH
NYLU
1420
140
120
800
'22
6:0
500
Naa may Clattar
Nap Assuracy Clutter
24
Figure A3: Difficulty distribution comparison on 12B model between predictable subset and full evaluation set.
inference evaluations the evaluation set for each small model (the cost of clustering algorithm is negligible compared to the inference cost). The computational complexity is O(TMN), where 7 is the number of evaluation runs, M is the number of tokens for evaluation, and N is the maximum parameter size of the small models used for prediction. The corresponding token usage is O(TM).
In particular, for evaluation requiring 1M tokens, total of 100M tokens for small model inference is needed. In our experiments, the training token count for the 12B small model is 1.5547. Considering that training token typically 3 times more costly than inference token, the additional cost of COD is approximately 100M
1,5547 zs 0.002% of the training cost.
H LIMITATIONS
Compromised robustness due to excessive hyperparameter. The complete pipeline of our proposed COD method incorporates several hyperparameters designed constrain and refine the outcomes of various stages. These include the minimum intra-cluster sample size K, the adaptive bandwidth hyperparameter Q, and the maximum intra-cluster distance threshold U for the clustering phase; parameters a, b, and e for filtering extrapolatable subsets during curve fitting; and the RMSE threshold d'utilized during the mapping process.
Regarding the clustering- related hyperparameters, they can be omitted if pre-computed cluster assignments are reused; otherwise, we provide empirically validated default values as reliable priors. For the remaining hyperparameters, we present comprehensive ablation studies in the paper, demonstrating that the final predictive performance is robust and relatively insensitive these settings. Despite these mitigations, the reliance on a murti-parramter configuration may pose challenges to the COD method's ease deployment when generalizing to novel prediction scenarios.
Category of evaluation sets. The proposed flusteringoon-iifficn method requires sufficient number of test cases, as too few samples can lead to unstable cluster metrics and ineffective estimation. From evaluation set design perspective, an evaluation set with good predictive properties enables more effective generalization from small-scale to large-scale models, thus providing better guidance for model iteration.
Furthermore, we have not included multiple-choice tasks that require comparing the logits correct options to calculate scores. These tasks creating discrepancy between the answer loss and the model's 's true passrate, which violates the assumptions of the proposed Scaling Law for downstream task performance
24

---

Published as a conference paper at ICLR 2026
The prediction aceuracy for smaller models is unsatisfactory. Since our proposed COD (Clustering on Difficulty) method involves modeling the scaling of sample difficulty within the evaluation set, the clustering process requires certain scale of models to participate in pass rate evaluation. This ensures an accurate estimation of sample difficulty. However, when predicting the performance of relatively smaller target models (e.g., around 10B parameters), the proxy models used for clustering and fitting are typically limited to even smaller scales (e.g.. 2B). In such scenarios, a significant portion of samples may exhibit non-emergent or nascent emergent behaviors, making challenging to accurately model difficulty features that scale with compute. Under these specific constraints, methods that perform sample-wise extrapolation such as PassUntil (Hu et al.) 2024) tend to yield more robust predictive performance. Nonetheless, the primary utility of metric prediction lies in forecasting the downstream performance of significantly larger models. From this perspective, our COD method maintains broad applicability and significant value in mainstream scaling law research.
Chain-of-thougie performance prediction. Theorem T Jasumess that evaluation sets directly assess models' ability to provide answers. However, increasingly more evaluations allow models to think before providing answers. Recent works on inference time scaling (Snell et al. 2024; Bansal al. 2024) further demonstrate that for tasks involving mathematics, reasoning, and coding, training models to complete tasks through longer inference computation can significantly improve downstream task performance. In cases where the reasoning process answers are not unique, the relationship between a model's answer loss and passrate on a task may not necessarily follow the exponential relationship between the answer loss and the sample passrate. Although our COD framework still achieves reasonable prediction performance in such scenarios, its theoretical foundation lacks sufficient explanation for the performance scaling of thain-offthhogee based tasks. Therefore, consider improving prediction methods based on chain-of- tii---ffffhhhhhhh characteristies and expanding theoretical foundations as future work.
25


Reviews:
[{"review_id": "UB9ypl2wEz", "reviewer": "Reviewer_vwUL", "summary": "This paper tries to address the challenge of accurately predicting downstream performance of large language models (LLMs) before full-scale training. Existing scaling laws—either based on loss or direct performance–compute curves—fail to capture emergent behaviors and heterogeneous task difficulty.\nThe authors propose a Clustering-On-Difficulty (COD) framework that groups evaluation samples by their performance-scaling patterns using an improved MeanShift algorithm. Within each cluster, they fit a theoretically derived performance scaling law to extrapolate results from small models to large ones. Predictions from extrapolatable clusters are then mapped via a polynomial function to the full benchmark.\nExperiments on eight benchmarks (e.g., MATH, GSM8K, BBH, MMLU-pro) show that COD achieves an average better prediction error of a 70B model than baselines.", "questions": "Some typos: e.g. 1.36%ha in line 061, can them be fixed?\n\nAny specific evidence why this method could handle better emergent and heterogeneous behaviors?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The COD approach provides a clear decomposition of the performance-prediction problem into clustering, cluster-wise fitting, subset extrapolation, and subset-to-full mapping. This layered design could handle emergent and heterogeneous behaviors better.\n2. The improved MeanShift with radius and minimum-cluster constraints balances automatic cluster discovery and intra-cluster homogeneity, outperforming K-Means, DBSCAN, and baseline MeanShift on both intra-cluster metrics and prediction error.\n3. Eight diverse benchmarks and multiple baselines are covered, including detailed ablations (clustering, extrapolation formulas, mapping functions).\n4. The paper provides pseudocode, theoretical proofs, and extensive appendices on hyperparameters, smoothing, and computational cost.", "weaknesses": "1. COD introduces multiple stages (feature extraction, clustering, extrapolation, mapping) with several hyperparameters (radius R, min cluster K, thresholds a/b/c, polynomial degree). This may hinder practical adoption and stability.\n2. In what extent of error rate, could the preformance prediction be useful? Can this actually reflect the actual final benchmark performance?\n3. In introduct the error rate is 1.36%, but in the table of experiment results, the error rate seems to be 1.63, which is inconsistent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aCrfk1VpbJ", "reviewer": "Reviewer_D8BZ", "summary": "## Summary\n\nThis work  proposes a new framework for predicting downstream performance using smalller proxy models. It introduces Clustering-On-Difficulty (COD), which groups samples in eval tasks by estimated difficulty  to reduce variance and enable more stable predictions￼. The paper provides theoretical analysis to justify this approach, and shows, thorugh extensive experimental results, that their approach outperforms other competing methods. Overall it is a strong work, and should be, in my opinion, accepted to ICLR 2026.\n\n## Typos, missing references\n- line 61: \" 1.36%ha\"\n- [Bhagia et al 2024](https://arxiv.org/abs/2412.04403)", "questions": "I've made some educated guesses in my weaknesses section. Would be great if the authors could comment on them to ensure that I'm correct.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "- Method is effective, well justified, and works across large set of taks.\n- The emprical results are very convincing.\n- The paper includes extensive ablations in main text and appendix.\n- Paper is important contribuition in the field of estimating downstream performance, as it significantly improve ability to predict downstream performance.", "weaknesses": "- The core for the clustering approach seems to hinge on the estimation of difficulty per each instance in a benchmark. To do so, the paper seems to imply that models that are also target of scaling laws are used. Unclear if the proposed approach would work if using a different model family (i.e., how would this approach work if one has to bootstrap from external open-weights LLMs?)\n- To estimate task difficulty, it seems that one need a model already trained at largest target size to build clustering features? Is that the case?\n- The paper is missing an ablation showing how prediction accuracy changes if fewer/smaller models are used.\n- The main text is really difficult to follow due to missing details. The details are generally provided in the appendix, but the back-and-forth makes following the approach very hard. \n- The work is applied to MC tasks only. MC tasks have interesting discuitnuity properties [\\(Wiegreffe et al, 2025\\)](https://arxiv.org/abs/2407.15018), but are far from the only formulation of interest.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cOLZ0NKOyq", "reviewer": "Reviewer_fKzE", "summary": "The paper proposes Clustering-On-Difficulty (COD) to predict the downstream accuracy of a large model from smaller models. The pipeline is: (i) build a task‑wise pass‑rate vector across model scales and cluster tasks (improved MeanShift with radius & min‑size constraints) to reduce intra‑cluster variance; (ii) fit a cluster‑wise performance–compute scaling law derived from a loss scaling assumption; (iii) filter “non‑extrapolatable” clusters via parameter thresholds; (iv) map the predicted accuracy on the “predictable subset” to the full evaluation set with a monotone polynomial (quartic by default), optionally calibrated with anchor models. On eight eval sets (GSM8K, MATH, BBH, TriviaQA, MBPP, AGIEval, DROP, MMLU‑pro), COD reports 1.36% average prediction error across eight key LLM benchmarks, outperforming loss‑intermediate, end‑to‑end, pass‑rate, and BNSL baselines.", "questions": "See cons", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "1. Directly predicting downstream performance (not just loss) is operationally valuable.\n2. The author spends much effort on training nine models (122M→70B) under a consistent recipe and eight diverse benchmarks. Such results are valuable for the community.\n3. The paper is well-written and easy to understand.", "weaknesses": "1. The generalizability of the model is questionable. All selected benchmarks are widely adopted in the community. How does the proposed generalize to unseen benchmarks?\n2. The author only compares with their own baselines. Many performance prediction solutions on downstream benchmarks are missing in the comparison.\n3. The COD experiments heavily rely on heuristic hyperparameters such as a, b & c. This limits its generalizability to other cases.\n4. Typos: in the abstract, the mean error is 1.36%, in Table 1, it is 1.63%. Which one is correct?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pFIKF5pKJr", "reviewer": "Reviewer_DRNX", "summary": "This paper introduces a cluster-based method for task scaling law (i.e., predicting larger model performance based on smaller model performance). The method has four steps: (1) cluster test examples based on their difficulty features (i.e., how small language models perform on the example); (2) fitting scaling law for each cluster; (3) extrapolation for each cluster; (4) mapping back to the full evaluation set performance. On an internal model series, the proposed method outperforms baselines such as end-to-end prediction or loss-intermediate prediction. Further ablation is done on clustering methods and scaling law formula.", "questions": "* Line 151: \"three random clusters\" - does this mean data points are randomly assigned to clusters, or three clusters randomly selected from your work in later chapters? Very confusing at this point.\n* Line 214: \"we generate 100 samples\" - does this mean sampling 100 answers for every input query?\n* Line 337: \"we calibrate ... using evaluation results from existing models as anchors\". Does this mean you fit the parameters in Eq. (5) using another model's performance? After reading appendix C.3, it seems this anchoring step is critical as removing it will results in deteriorated performance (worse than the end-to-end baseline). I have three follow-up questions regarding this: (1) Do baseline methods have access to this extra information? (2) Can you elaborate more on the in-house MoE model you used here? How large is it? (3) Can you report the performance of the calibration model and the target model (70B) on the 8 tasks in Table 1?\n\n(I'm willing to adjust my rating if the questions above are resolved.)\n\nSuggestions:\n* I was very confused when reading the abstract. Terminologies like \"difficulty scaling features\" and \"support subset\" were not defined. \"hindering prediction by smaller models\" is confusing, do you mean \"extrapolation from performance using smaller models\"? \"the exclusion of tasks\" - do you exclude tasks or test examples? (My current understanding is that you exclude some test examples.)\n* Suggested reference:\n  * Establishing Task Scaling Laws via Compute-Efficient Model Ladders (https://arxiv.org/abs/2412.04403)\n  * Observational Scaling Laws and the Predictability of Language Model Performance (https://arxiv.org/abs/2405.10938)\n  * How Predictable Are Large Language Model Capabilities? A Case Study on BIG-bench (https://arxiv.org/abs/2305.14947)", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* The paper presents an effective and reasonable approach for downstream task scaling law prediction. The pilot study points out limitations about several questionable assumptions in task scaling law, which I found insightful. The proposed method is reasonable and intuitive.", "weaknesses": "* Some details of the proposed method is unclear or missing. See questions below.\n* Results were limited to one model series. The paper could be more convincing if the method was shown to be effective on various model families.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)