CAPE: Generalized Convergence Prediction Across Architectures Without Full Training
Authors: Alireza Pourali
Arian Boukani
Hamzeh Khazaei
Venue: Accepted by TMLR ()

Abstract: Training deep neural networks to convergence is expensive and time-consuming, especially when exploring new architectures or hardware configurations. Prior work has primarily estimated per-iteration or per-epoch cost under fixed training schedules, overlooking the critical challenge of predicting how long a model will take to converge. We present \textit{CAPE} (Convergence-Aware Prediction Engine), a lightweight and probing-based framework that predicts the number of epochs required for convergence before any full training occurs. CAPE performs a brief probe at initialization using a small batch of data to extract analytical and dynamical features such as parameter count, dataset size, learning rate, batch size, gradient norm, Neural Tangent Kernel (NTK) trace, and initial loss. These features jointly characterize the model’s optimization landscape and serve as input to a meta-model trained to forecast convergence horizons under a validation-based early-stopping criterion. CAPE achieves strong predictive correspondence to true convergence epochs, with a Pearson correlation of 0.89 across diverse architectures and datasets, demonstrating accurate and consistent convergence prediction across model families. By enabling zero-shot prediction of full-dataset convergence behavior, CAPE provides a practical tool for rapid model selection, hyperparameter exploration, and resource-aware training planning.

Published in Transactions on Machine Learning Research (12/2025)
CAPE: Generalized Convergence Prediction Across Archi- tectures Without Full Training
Alireza Pourali
aliresop@yorrku.ca
Deportment of Electrical Engineering and Computer Science
York University
Arian Boukani
arbo@yorku..cr
Department Electrical Engineering and Computer Science York University
Hamzeh Khazaei
hid@yorkk.cc
Department of Electrical Engineering and Computer Science
York University
Reviewed on OpenReview: /b net/ forum? id- JuBBg
Abstract
Training deep neural networks to convergence is expensive and time-consuming, especially when exploring now architectures or hardware configurations. Prior work has primarily estimated per-iteration or per-epoch cost under fixed training schedules, overlooking the critical challenge of predicting how long a model will take to converge. We present CAPE (Convergence-Aree Aware Prediction Engine), a lightweight and probing-based framework that predicts the number of epochs required for convergence before any full training occurs. CAPE performs brief probe at initialization using small batch of data to extract analyt- ical and dynamical features, including parameter count, dataset size, learning rate, batch size, gradient norm, Neural Tangent Kernel (NTK) trace, and initial loss. These features jointly characterize the model's optimization landscape and serve as input to a meta-model trained to forecast convergence horizons under validation-based early-stopping criterion. CAPE achieves strong predictive correspondence to true convergence epochs, with a Pear- son correlation of 0.89 across diverse architectures and datasets, demonstrating accurate and consistent convergence prediction across model families. By enabling zero-shot predic- tion of full-dataset convergence behaviour, CAPE provides a practical tool for rapid model selection, hyperparameter exploration, and resource-aware training planning.
1 Introduction
Deep neural networks (DNNs) require efficient training methods because their development depends on ample computational resources and extended training periods. The current techniques (Bergstra & Bengio, 2012 Li al. 2018: Parker-Holder
al. 2020) require extensive experimental testing find optimal hyperparameters and training schedules, which results in both time consuming delays and unnecessary resource consumption. The Neural Tangent Kernel (NTK) theory stands out significant contribution to understanding how deep neural networks converge in recent research (Arora et al. 2019 Jacot et al. 2018 Loe et al.) 2019 Wang et al. 2022 Mu et al. (020). which analyzes the training dynamics in high-dimensional settings. However, both theoretical and empirical works highlight critical limitations: NTK validity depends on tight rescaling conditions (Boix-Adsers & Littwin 2023). extensions with regularization are still confined near initialization (Clerico & Guodj] 2024), and empirical scaling behaviours deviate significantly from NTK
predictions (Vyas et al. 2023). These gaps limit applicability to real-worrd scenarios. Furthermore, studies on
meta-learning (Ji et al. 2020 Ye al. 2021
Chen al. 2020 Guan et al. 2022 Harrison et al. 2022 Zhou

---

Published in Transactions on Machine Learning Research (12/2025)
et al,, 2019 Tack et al. 2022: aim improve learning efficiency by leveraging knowledge from past tasks, but cannot often predict training time for held-out configurations (i.e., new model-datasee dataset- hyperparameter tuples). Work on predicting computational cost (Justus et al. 2018 Pourali et al. 2025 Geoffrey et al, 2021 has shown promise, but often relies on linear models that fail to capture the inherent non-lincarities of DNN training. To address these limitations, we introduce CAPE (Convergence- Aware Prediction Engine), a novel framework that combines analytical descriptors with probing-based feature extraction to accurately predict the number of epochs required for convergence under validation-basee early-stopping criterion.
Our proposed framework employs lightweight, probing- grobing-based feature extraction method to characterize DNN at initialization, climinating the need for full training runs. This design draws inspiration from recent advances in linking initialization dynamics to optimization and generalization behavior (Wang et al. 2022; Ronen et al. 2019 Marion & Berthier 2023). By extracting analytical and dynamical features such as the number of parameters, dataset size, learning rate, batch size, gradient norm, initial loss, and a proxy for the Neural Tangent Kernel (NTK) trace, CAPE captures both the structural complexity and carly optimization smoothness of the model. These features form meta-dalasel that spans diverse neural architectures (MLPs, CNNS, RNNs, and Transformers) and datasets. The meta-dataset is then used to train a regression-basse based meta-model that learns the mapping between the extracted features and the number of epochs required for convergence, as determined by validation-based early-stopping criterion. This meta- meta-learning approach enables CAPE to generalize effectively to unseen architectures and datasets, providing accurate and efficient convergence predictions that surpass traditional analytical and linear regression baselines (Zancato et al.
2020 Kaplan et al. 2020: Hoffmann et al. 2022).
The key advantage of our approach lies in its ability to predict the number of epochs required for model to reach convergence before initiating full training. This capability provides substantial benefits throughout the deep learning workflow. Researchers and practitioners can leverage these predictions to make informed choices regarding model selection, hyperparameter tuning, and the design of efficient training schedules. By accurately forecasting convergence behavior in advance, CAPE enables more effective allocation of compu- tational resources and reduces the time spent on trial-and-error experimentation. The proposed framework overcomes the limitations of existing methods by combining probing- based feature extraction with the gen- cralization capabilities of meta-learning, offering practical and scalable solution for convergence prediction in deep neural networks. Ultimately, this work contributes to the broader goal of improving the efficiency and accessibility of deep learning research by allowing practitioners to focus model design and evaluation rather than exhaustive training cycles.
To summarize, our contributions are the following:
(i) We introduce probing-based convergence prediction framework that estimates the number of epochs required for deep neural networks (DNNs) to reach convergence, without executing full training runs.
(ii) We extract both structural and dynamical features, including parameter count, dataset size, batch size, learning rate, gradient norm, initial loss, and a proxy for the Neural Tangent Kernel (NTK) trace by probing the model at initialization using a small batch of data.
(iii) We construct a meta-dataset that spans diverse architectures (MLPs, CNNs, RNNs, Transformers) and datascts, capturing convergence behavior across a wide range of model configurations.
(iv) We develop a regression-based meta-model using a Random Forest ensemble, trained on the con- structed meta-dataset to learn the complex mapping between initialiaation-tine features and the number of epochs required for convergence under a validation-basee early-stopping criterion. This meta-model generalizes effectively to held-out architectures and datasets not seen during meta- training.
(v) We demonstrate that our system enables early estimation of training cost, allowing rescarchers and practitioners to make informed decisions about model selection, resource budgeting, and training schedules, improving overall efficiency in deep learning workflows.
2

---

Published in Transactions on Machine Learning Research (12/2025)
2 Related Works
2.1
Convergence Prediction in Deep Learning
Recent works on convergence analysis (Allen-Zhu et al. 2019 Zou el al. 2020 Ji el al. 2020 Gao et al. 2024
Zancato et al. 2020) in deep learning focus heavily on theoretical guarantees derived from over parameterized networks and NTK-based formulations. These methods show that gradient descent can converge globally when nelworks are sulficiently wide and initialized under specific distributions, assuming either data separa- bility or proximity initialization. Beyond NTK, other lines of work establish convergence guarantees under weaker conditions such as gradient domination (Weissmann et al. 2025), logistic loss on two-layer networks (Gopalani et al. 2024). small initialization regimes (Kumar & Haupt 2025 and optimizer-specifie analyses of RMSPropp Adam (Zhang et al. 2025). For Transformers, convergence analysis has also considered im- plicit bias and self-attention dynamics (Vasudeva et al. 2024). Some models reformulate training dynamics as stochastic diffcrential equations in function space, enabling closed form predictions of timo-to-accuracy using the eigenvalues of the NTK (Zancato et al. 2020 Wan et al. 2021 Lee et al. 2019). However, these approaches are mostly confined to fine-tuning or pre-trained regimes, often requiring the network to remain
in a small perturbation region around initialization (Allen-Zhu al., 2019 Zancato al. 2020). While
such works provide valuable theoretical insights, their reliance on restrictive assumptions limits computa- tional practicality and precludes advanced prediction of convergence before training begins, gap our system directly addressos.
2.2 Learning Curve Extrapolation and Training Time Estimation
Learning curve extrapolation is widely used to estimate final performance from early training signals, espe- cially within AutoML and neural architecture search workflows. Traditional approaches use Bayesian curve fitting, Gaussian processes, ensemble- based estimators that require observing partial training trajectories [Domhan et al. 2015 Klein et al. 2017). More recent meta-learned models like LC-PFN (Adriaensen et al. 2023) reduce reliance on handcrafted priors and instead fit generalizable predictors over many tasks, enabling one-shot extrapolation of loss or accuracy. However, such methods still depend on partial curve data and are inapplicable when no training has occurred. Some frameworks like MOTE-NAS jointly predict resource use and model quality but focus more on cost or latency than convergence (Zhang et al. 2024b). Our work differs by offering zero-shot convergence prediction, estimating the number of epochs required for training to converge using initialization-oni
features, without relying on any portion of the learning curve (Zela et al. 2020: Bahri et al.) 2024). Complementary studies have demonstrated that learning rate schedules, particularly cooldown phases, play a critical role in shaping convergence dynamics (Dremov et al., 2025) further motivating the need for predictors that generalize beyond handcrafted or schedule-dopendent training trajectories.
2.3 Meta-Learning and Probing-Bard Based Estimation
Meta-learning has proven effective for transferring training knowledge across tasks and model types, espe- cially when applied to optimizer adaptation, initialization heuristics, and performance modeling (Ye et al., 2021 Zhang et al. 2024a Guan et al.) 2022). Convergence in distributed and federated settings has also been studied, where Local SGD exhibits accelerated rates for over-parameten peemtparaeterrieed models (Qin al. 2022 complementing meta-generallizttion perspectives. Some approaches use task-conditioned priors or learned regularizers that adapt to optimization landscapes dynamically, often leading to improved generalization acToss datasets and objectives (Tac et al. 2022 Jiang et al., 2021 Grant et al. 2018). Probing- based ostimation complements this by extracting features like gradient norm, NTK trace proxies, and paramcter counts at initialization to guide predictions about training dynamics (Zhu el al.
2022 Xia el. al. 2020).
These methods often avoid full training by leveraging small-batch statistics and meta-trained regressors, reducing computational overhead (Adriaensen et al., (2023] Wang Ma. 2022). Despite their strengths, prior work typically focuses on few-shot learning or inner-loop optimization efficiency rather than directly predicting full-model convergence from scratch, which our work uniquely addresses.
3

---

Published in Transactions on Machine Learning Research (12/2025)
3 Predicting Convergence Without Training
In this section, we formalize the task of predicting the convergence epochs of a deep neural network (DNN) without executing full training. Rather than relying on lcarning-cury extrapolation or empirical tuning, CAPE estimates the number of epochs required reach a predefined convergence threshold, as determined by a validation-based early-stopping criterion. This estimation uses only analytical and dynamical features extracted initialization, enabling zero-shot convergence prediction across architectures and datasets.
Assumption 1 (Fixed and Reproducible Initialization). Model parameters 0 E Rp are initialized with a deterministic scheme (e.g., Xavier or He) and fized random seed. All probing features are computed at this inilialization point Oo; parumeters remain unchanged during probing.
Let X' and y denote the input and output spaces, respectively, and let the dataset be = X V. We write the network as
C
fa X Y 0 E R".
Given a task-appropriat loss :: f: y + Rzo, we define the empirical training and validation losses at epoch by
Lirain (t) = Drrain (A.A)
Σ
(z).)
(t) = D
(foco (f)) v),
where Dusin U Dval is fixed split and g(t) are the parameters after t training epochs (used only define the ground-truth label; CAPE does not execute these epochs at inference time).
Definition (Convergence Epoch Count). The convergence epoch count Teony is defined as the smallest epoch at which the validation loss (t) fails to improve by more than tolerance 5 for consecutive epochs:
Teony min {t N Lval(t p') Lval(t) < 5, Vp' S p)
(1)
This criterion aligns with standard early-stopping practice in deep learning, replacing fixed loss ratios with validation-baset based stopping rule.
Assumption 2 (Expected Monotonic Decay). While stochastic optimizers such as SGD or Adam introduce noise in the loss trajectory, we assume that the expected validation loss [[)) decreases mono- tonically in expectation, ensuring that Eq. [ yields a well-defined Teonv-
To predict Toonv without performing training, we extract a feature vector z E IRd from the model at initial- ization and from a small subset of the training data.
Assumption 3 (Representative Probing Subset). A randomly sampled subset D Proobe DD with [D probel < N, provides a sufficient approximation of the model's early-layer dynamics and curvature statis- tics, allowing the probing features z (defined in Section to capture ronvergence-velevant behavior.
Probe-batch vs. full-dataset convergence. All analytical and dynamical features (e.g., log V[],, NTK trace, initial loss) are computed using the small subset Pprobe- However, the ground-truth convergence label Teonv is measured on the full training dataset using the validation-basee criterion in Eq. ! This distinction enables CAPE to remain zero-shot at inference time while grounding its labels in realistic full-dataset behavior.
NTK
We then train regression function g Rd + R that maps the probing features to a predicted conver- gence epoch count feouv- A meta-dataset M (() righh))) is constructed from diverse architectures, datasets, and training configurations. The mela-regressor is trained by minimizing the mean squared error in log space:
M Σ 1=1
min 1 9EG
TG) M (log log conv
(2)
Assumption 4 (Meta-Generalizattonn
The mege-regressor trained on M generalizes to held-out orchitectur-dalacee dalaset pairs drawm from the same underlying distribution.
4

---

Published in Transactions on Machine Learning Research (12/2025)
Proposition 1 (Asymptotic Consistency). If the meta-dataset M is sufficiently diverse and the hy- pothesis class G is rich function family (e.g., universal opprozimalors), then as M - 00, the predictor g converges in probability to the true mapping @f Proobe: 1,, (,,,,)) - Tconv.
This formulation enables convergence prediction in the early-stopping sense without performing full training, relying solely on computational and structural signals extracted at initialization.
4 Probing-Based Feature Extraction and Meta-Learning
CAPE's implementation comprises three stages: (1) designing initialization-time probing features that cap- ture both structural and dynamical properties of a model; (2) constructing diverse meta-dataset from multiple architectures and datasets; and (3) training meta-regrossor to predict convergence epochs from these features. This section details each stage of the system.
4.1 Probing Feature Design
We extract compact set of features from randomly initialized model using a small probe subset of the training data. As stated in Assumption 1, initialization is fixed and reproducible, and no parameter updates are performed, making the procedure efficient and architect ure-anostt..........
Let feo denote the neural network initialization with paramcters 6 Ro, and let D Probe C D be a randomly sampled subset with size B < [D], as described in Assumption 3. The probing function is defined
z = @(ff, Proobe: N,
(3)
where 1j the learning rate, B is the batch size, is the total dataset size, and a an architecture identifier (e.g., MLP, CNN, RNN, Transformer). Each element influences the expected optimization dynamics. The following initialization-time features are computed:
Parameter Count P. Measures the representational complexity the model, reflecting its total number of learnable parameters:
P= 10o|
Initial Loss Co. Quantifies the model's empirical loss at random initialization, reflecting how well un- trained features aligu with the target distribution:
Lo
Σ (z),))) (s.y)
(5)
B
It reflects the alignment between random features and target distributions.
Average Gradient Norm G²
Sensitivity of the loss to parameter updates at initialization:
G B Σ (vo,
(6)
(L.
Larger values typically indicate steeper loss surface and faster initial descent, whereas smaller values suggest flat or ill-conditioned regions.
NTK Trace Proxy A curvatureesemsitive quantity capturing how model outputs vary with parameter perturbations:
1
Σ (x))
(7)
B
DED
5

---

Published in Transactions on Machine Learning Research (12/2025)
Log- Trannforrmed Feature Vector. To reduce scale variance and improve numerical stability, all scalar quantities are log-transformed before regression:
logg))) log(G"), (o) log)))) logg()n)))B))))) log(B),
All features can be computed within a second on modern hardware, allowing large-scale or real-time evalu- ation.
4.2 Meta-Datased Construction
For each model dataset configuration, we pair the extracted features 2 with the ground-truti convergence epoch count T(i) defined by Eq. The resulting meta-dataset is
M
{(2 T )
(9)
spanning diverse architectures, datasets, and training conditions to promote generalization.
4.3 Meta- Regrrserrrssor Training
Given M,, we train regression model R - that maps probing features to predicted convergence epochs. We employ a Random Forest regressor (Breiman 2001) due to its robustness to feature scale, interpretability, and strong empirical performance on heterogencous feature sets. The model is trainod on log-transformed inputs and largets to reduce scale sensitivity and stabilize variance across configurations, minimizing the objective in Eq. 2 Regularization implicitly achieved through bootstrap aggregation and random feature sampling, while model selection and hyperparameter tuning are performed via k-fold vross-validttion on the training meta-set.
5 Experiments
We evaluate CAPE across four architecture families such as MLPs, CNNs, RNNs, and Transformers, each instantiated using compact yet representative models capture diverse convergence behaviors. All experi- ments follow unified probing and validation-based convergence framework, ensuring comparability across architectures.
Model familics and training protocol. Each model family is represented by widely used architectures scaled for elficient convergence measurement. MLP-based models include MLP-Mizer (Tolstikhin el al. 2021), ResMLP (Touvron et al. 2022 and AS- AP (Lian et al. 2021 CNNs are represented by ResNet- 50 He et al. 2016), DenseNet- 121 (Huang et al. 2017), and MobileNet V2 (Sandler et al. 2018). RNNs in- clude LSTM [Hcchreieer Schmidhuber 1997). GRU (Cho et al. 2014). and BiLSTM (Huang et al. 2015 while Transformers cover DeiT- Tiny (Tourron et al. 2021 and DistäBERT (Sanh et al. 2019
All mod- 2x 2x10-33,
els are trained under consistent hyperparameter grid with learning rates LR E {5x10- 10 batch sizes B E {8, 16, 32, 64, 128, 256), and optimizers Adam, Adafactor, Adam W and SGD.
Datasets. We evaluate each model on the datasets that align with its standard benchmark usage to ensure architectural relevance and consistent convergence characteristics:
MLPs:
AS-MLP and MiP-Mixer are evaluated using TinyImageNet (Dnng et al. 2009 and
STL10 Coates el. al. 2011); ResMLP is evaluated using CIFAR-100 (Wei et al. 2021 and TinyIma- geNet.
ResNet-50 is evaluated using TinyImageNet and CIFAR-10 Wei et al. CIFAR-100 and TinyImageNet; and MobileNet on CIFAR-10 and STL10.
CNNs:
2021 DenseNeet-122 on
RNNs:
GRU and LSTM are evaluated using IMDB MMas al. 2011 and AG NEWS
Gulli 2005
BILSTMM on SST2 (Socher et al. 2013) and IMDB.

---

Published in Transactions on Machine Learning Research (12/2025)
Transformers: DistilBERT is evaluated using SST2 and IMDB; DeiT-Tiny CIFAR-100 and Tinylm- ageNet.
All datasets are preprocessed following standard normalization and tokenization protocols. Image datasets are resized to their canonical benchmark resolutions consistent with prior work (e.g., CIFAR and Tiny- ImageNet); while text datasets use subword or whilespace lokenization with vocabularies capped al 30k tokens.
Mcta-feature extraction. To estimate the convergence epoch Teony from initialization, we extract six probing features from a single training batch per model datasel pair: parameter count (log P), learning rate (log LR), batch size (log B), squared gradient norm (log G²), NTK trace proxy (log7), and initial loss (log La). For each configuration, the ground-truth label Teonv is defined loss fails to improve by at least the chosen tolerance threshold (5 x 10 for five consecutive epochs. This tolerance level aligns with widely used early-stopping practices in modern deep learning, where improvements below the order of 10-3 to 10 are typically treated as numerically insignificant and attributed to routine stochastic variation. All probing features were computed from a single mini-batch corresponding to the smallest batch size defined for each -ataset configuration. Although this represents a minimal subset of the training data, CAPE consistently achieved accurate convergence predictions across architectures and datasets, underscoring the robustness of its initialization-lime feature extraction.
the first epoch at which the validation
We train Random Forest regressor predict log Toan from these features, using 200 estimators, maximum depth of 8, bootstrap sampling, and mcan squared error as the split criterion. Predictions are exponentiated to recover Teony in epoch units. We choose Random Forest for its robustness to noisy features, low sensilivily to scaling, and strong performance on heterogeneous tabular datasets.
Hardwarc and runtime. All experiments were conducted on a workstation equipped with an NVIDIA RIX 4000 Ada GPU, an Intel Core i9- 19-14900K CPU, and 64 GB of RAM. The full meta-dataset generation, covering all model families (MLPs, CNNs, RNNs, and Transformers), multiple datasets, and exhaustive sweeps over learning rates, batch sizes, optimizers, required over 400 GPU hours of compute time.
Evaluation protocol. We evaluate CAPE under three out-of-sample regimes that mirror realistic deploy- ment: (i) 5-fold cross validation (CV) with shuffled folds; (ii) (LOMM)), where each model family is held out in turn. For every regime, we generate strictly out-of-fold predictions: in CV, a standard 5-fold cross-validattin procedure is applied, whereas LODO and LOMO, a Leave-One- One-Group-Out splitter grouped by dataset or model used. To ensure reprodueibility, the code and datasets used in our experimental evaluation are publicly available in GitHub repository
(LODO), where each
dataset is held out in turn; and (iii) Leave PeaeeOOne-Moeele
Baselines. To the best of our knowledge, prior work has demonstrated zero-shot prediction of the number of training epochs Teonv required for convergence under validation-based early-stopping criterion, for a novel combination of architecture, dataset, and byperparameters, using only single batch probing features and a learned regressor. An ablation of alternative regressors for CAPE and baseline models is reported in Appendix A.3 We compare against three represontative baselincs, both parametric and non- parametric, each evaluated under the same out-of-sample protocol:
(L) CAPE (probe-only) trains an identical Random Forest but only on the two probe features log G2 and log
(2) Tearning-Curve Extrapolation (LCF) DDmman et al. 2015) predicts T Teony from an exponential fit to the validation loss prefix; the decay/offset hyperparameters are learned within each training fold and the amplitude is calibrated on the test prefix before applying the carly-stopping rule.
(3) Scaling- Law uses ridge regressor on (log P. log N) only.
htaps:///gthhub..om//aaaaaat

---

Published in Transactions on Machine Learning Research (12/2025)
Table 1: CAPE vs. baselines across evaluation protocols. Lower MAE/RMSE and higher PearsonR, indicate better performance.
Method
Evaluation protocol
MAE
RMSE
PearsonR.
CAPE (full feature set)
CAPE
Cross-fold (5-fold)
4.63
8.10
0.89
CAPE
Leave-one-datasse--ouu
6.85
10.57
0.81
CAPE
Leave-one-model-ont
7.27
11.04
0.79
Baselines
Probe-only (log G² log r)
Cross-fold (5-fold)
13.10
21.33
0.05
Probe-only (log G² log r)
Leave-ome-dataset-ont
13.90
21.69
-0.09
Probe-only (log G2 log r)
Leave-one-moodel-out
15.12
23.12
0.16
Learning-curve extrapolation Learning-cury extrapolation Learning-cure curve extrapolation Scaling-law (log P. log N)
Cross-fold (5-fold)
13.53
29.22
0.64
Leave-one-datasst-ouu
16.09
33.68
0.65
Leave-one-model-on mut
14.43
31.08
0.66
Cross-fold (5-fold)
11.87
17.23
0.28
Scaling-law (log P. log N)
Leave-one-daaasse--uut
11.81
17.29
0.27
Scaling-law (log P. log N)
Leave-one-mmodeloout
12.34
17.82
0.18
Metrics. We report three complementary metries for CAPE: Mean Absolute Error (MAE), Root Mean Squared Error (RMSE) and Pearson correlation coefficient (r). For the baselines we report MAE, RMSE, and r. All metrics are computed on the out-of-fold predictions produced by the corresponding regime.
Mean Absolute Error (MAE). Measures the average absolute difference between the predicted and actual convergence epochs:
MAE
predd Tact,el-
Root Mean Squared Error (RMSE). Emphasizes larger prediction deviations by squaring the resid- uals before averaging:
1 n1
RMSE
(Trdd Tact,i)².
Pearson Correlation Coefficient (r). Quantifies the linear correlation between predicted and actual convergence epochs:
E (Tpred,s - T prooa) T Tact)
Ei (Tpred, prea)) Di (Tact,i T Tact)²
5.1 Comparison with Baselines
Evaluation and comparison. Across all protocols, CAPE attains substantially lower absolute errors and markedly higher correlation than the baselines (Table D. In cross-fold evaluation, CAPE achieves MAE 4.63 and RMSE 8.10 with Pearson correlation 0.89. When holding out entire datasets models, CAPE maintains strong accuracy (MAE 6.85/7.27; RMSE 10.57/11.04) and robust rank correlation (PearsonR 0.81/0.79). The probe-only variant, which removes all static/conted features, shows large errors and near-zero (or negative) correlations, indicating that probe signals alone are insufficient. Learning-curve extrapolation, despite consuming early validation prefixes, underperforms CAPE on both error and correlation, highlighting the advantage of CAPE's initialization-tiin time features coupled with meta-regression. The simple scaling-law baseline captures coarse trends from (log P, log N} but lacks the fidelity to match CAPE's accuracy
8

---

Published in Transactions on Machine Learning Research (12/2025)
correlation. Collectively, these results demonstrate that CAPE's integrated feature design and training protocol provide reliable, generalizable convergence-epoch predictions under diverse evaluation regimes.
Relation to neural scaling laws. Neural scaling laws posit approximate power-law relationships bet ween model/dataset scale and performance or compute (Kaplan et al. 2020 Hoffmann et al., 2022). Our scaling- law baseline follows this spirit by regressing convergence epochs on {log lo N). While such macroscopic laws capture broad monotone trends, they deliberately abstract away optimization specifics (e.g. learning rate, batch size, curvature/gradient geometry). By contrast, CAPF; augments static scale with initialization- time probes (log log.) and meaining-conted features, enabling it account for short-horizon dynamics that materially affect when early stopping is triggered. Empirically, this yields substantially tighter fit (lower MIAE/MMSE and stronger correspondence with ground truth than scale-only predictors, indicating that convergence timing is not solely function of (P,N)
N) but also of local geometry and optimizer scale.
Scaling-law versus Probe-based predictors. Scaling-law estimators are most effective in the pre- training design phose, where only coarse descriptors, such as model paramcter count and dataset size are known. They provide inexpensive and interpretable approximations of asymptotic behavior, allowing prac- titioners to forecast compute requirements or performance trends under fixed training protocols. However, because these estimators assume stationary training regime and abstract away optimization dynamics, they often fail to capture non-scaling effects introduced by changes in hyperparameters, architectural inductive biases, or data distributions. In contrast, CAPE operates at the inilialization and probing level, integrat- ing both structural and early-dynamic signals (e.g., gradient norm, NTK trace, optimizer configuration) to model short-horizon convergence behavior. This makes CAPE particularly suitable when the training setup deviates from canonical scaling assumptions, for instance, when adjusting batch size or learning rate, em- ploying adaptive optimizers, or applying early stopping criteria. In such regimes, CAPE consistently achieves lower prediction error and higher agreement with realized convergence compared to scale-only baselines, of- fering a more robust and context-aware characterization of training dynamics while maintaining lightweight computational overhead.
Comparison with learning-curve extrapolation. Our LCE baseline fits an exponential model T(()- ac- c to the initial portion of each run's 's validation-loss trajectory. The decay and asymptote parameters (b,c) c) are estimated from the training data and applied to held-out runs, after which the fitted curve is ex- trapolated to the full training horizon and an early-stopping rule is simulated using each run's patience and tolerance parameters. Both CAPE and LCE predict the same convergence target and are evaluated under identical experimental protocols with shared metrics. Across all settings, CAPE achieves substantially lower MAE and RMSE and higher correlation than LCE (Table 1 This result suggests that early validation trajectories, though informative in principle, are often noisy and sensitive to optimization hyperparame- ters, whereas CAPE's initialization-time geometric probes and contextual features yield more stable and generalizable convergence predictions.
5.2 CrchttArch and Cross-Datases Convergence Prediction
Evaluation Protocol. As shown Table 2 we report per-model performance of CAPE under three ovaluation regimes: 5-fold cross-validation (CV), Leave--ne-Datasctt--O0 (LODO), and Leave-One-Model- Out (LOMO). Each entry summarizes mean absolute error (MAE) and root mean squared error (RMSE) in epochs.
Observations. CAPE maintains low prediction error (typically 2-8 epochs) across most architectures and evaluation regimes. Recurrent families (BILSTM, GRU, LSTM) and several CNN/MLP variants (DenscNet121, ResMLP, Mixer) show stable performance, indicating that the probe-based fcatures gen- eralize well beyond the specific training domains. Interestingly, CAPE occasionally performs better under cross-domain regimes than in standard cross-validation. For example, DefT-Tiny and DistilBERT achieve lower MAE and RMSE under LODO, indicating that datasot-level generalization can benefit Transformers whose convergence dynamics are relatively stable across datasets with similar optimization profiles. Like- wise, the LSTM model attains slightly lower error under LOMO, suggesting that recurrent. architectures
9

---

Published in Transactions on Machine Learning Research (12/2025)
Table Per-model convergence prediction for CAPE across evaluation regimes. MAE/RMSE (epochs).
Lower is better for
CV (5-fold)
LODO
LOMO
Model
MAE
RMSE
MAE
RMSE
MAE
RMSE
DeiT-Tiny
4.42
5.72
3.84
4.99
4.90
6.28
DistilBERT
2.34
3.37
1.79
2.55
2.95
3.84
AS-MLP
12.63
16.23
14.46
18.64
19.30
23.05
BILSTM
5.21
9.59
6.84
12.98
5.74
10.65
DenseNet121
2.20
3.50
4.25
4.94
3.58
4.31
GRU
2.67
4.40
7.22
10.15
5.28
7.45
LSTM
5.19
9.20
8.89
13.13
4.86
8.63
MLP-Mixer
5.25
11.18
5.99
11.37
6.51
10.07
MobileNet
4.94
6.14
11.23
12.01
9.81
12.11
ResMLP
3.56
5.67
4.49
7.12
4.99
8.29
ResNet-50
2.53
3.28
6.31
7.56
12.07
13.29
share transferable temporal-gradient statistics that help CAPE interpolate across unscen sequence models. In contrast, convolutional and MLP-based families (DenseNet121, ResNet-50, ResMLP, Mixer) depend strongly dataset-specific features such imput resolution, normalization depth, and augmentation scheme, causing CV to remain their most favorable regime. Finally, AS-MLP exhibits the largest its broad convergence range (26-67 epochs), where proportionally similar relative errors translate into large absolute deviations in epoch counts. Moreover, LOMO remains the most challenging regime overall, as it requires extrapolation to unseen architectures; nevertheless, CAPE maintains competitive MAE (often <10 epochs) across most model familics, demonstrating its ability capture transferable convergence signatures across both datasets and architectures.
due to
5.3 Cross Dataset Behavior under Batch-Size Variation
We evaluate CAPE under the Leavo-One-Dataset-OO (LODO) protocol while systematically sweeping key hyperparameters, batch size, learning rale, and optimizer lype, to examine its robustness to training config- uration shifts. For each model, training is performed on available datasets except one, which reserved for evaluation. Specifically, DeiT-Tiny: ResNet-50, DenseNet- 121, MLP-Mixer, and ResMLP are tested on TinylmageNet, DistilBERT and GRU IMDB, and BiLSTM on SST-2 as the unscen datasets. Figure [ summarizes CAPE's
performance across varying batch sizes for all model fami- lies. Overall, predicted convergence epochs remain closely aligned with the actual values, indicating that the meta-regressor generalizes well across unseen datasets. For recurrent models (BiLSTM, GRU), CAPE slightly overestimates convergence at smaller training batch sizes but achieves stable agreement as batch size increases, reflecting that probe-derived features computed using the smallest batch size per model-datasct configuration, captures optimization dynamics robustly despite changes in batch-dependent gradient noise during training. Transformer-bent based architectures (DeiT-Tiny, DistilBERT) show consistently accurate pre- dictions with only mild variability batch size grows, suggesting that probe statistics such as gradient norm and NTK trace effectively encode convergence behavior largely independent of mini-batch scale. Among con- volutional and MbP-based archilectures (DenseNet-121, MLP--ixer, ResMLP, ResNet-50), predicted values closely track actual convergence trends, with minor underestimation observed for ResMLP and ResNet-50 at intermediate batch sizes. These deviations likely arise from datasot-specific optimization effects on Tiny- ImageNet, where higher input variability and feature diversity slightly delay convergence relative the probe-based expectation.
10

---

Published in Transactions on Machine Learning Research (12/2025)
12
Astaal (epocho
Actul tepocas)
Canvergene (qpochs)
76
Prodiced (spoche)
Prodicted
144%
5.0%
22.80%
101 0%
3.2% 14.7 25.1%
19.7%
Conveegenns
12.2%
11.0%
20.0%
9,35
11%
4.8%
22.3% 83S
16
10
33
32
125
BILSTM
Dell-Timy
DistiBERT
GRU
DemseNet-EE
MLP Mixer
ResMLP
ResNet-50
Figure 1: LODO evaluation across batch-size variations. Predicted versus actual convergence epochs are shown for cach model, with relative errors (%) indicated above each configuration.
5.4 Cross-Dataset Behavior under Learning-Ran Rate Variation
Figure presents CAPE's Cravv-One-aattse-- Out evaluation across different learning rates, illustrating the model's robustness to optimizations ion-iiiaaiionllllle ion-scale variation. Overall, the predicted and actual convergence epochs remain closely aligned, with most relative errors below 15%, confirming that CAPE effectively models con- vergence dynamics across broad range of learning rates. For recurrent architectures (BILSTM, GRU), prediction accuracy slightly degrades at intermediate learning rates, likely due to increased gradient oscilla- tion that perturbs early probe statistics. Transformer models (DeiT-Tiny, DistilBERT) exhibit highly stable predictions at lower and moderate rates, with minor overestimation appearing only at the highest setting, indicating that probe-derived features generalize well across smooth optimization regimes. Convolutional and MT.P-based models (DenseNet-121,, MI.P-Mixer, ResMLP, ResNet-50) maintain strong agreement over- all, with mild peerprediction observed at elevated learning rates, particularly for ResMLP and ResNet-50, where faster initial loss decay leads to slightly premat ure predicted convergence. These results demonst rate that CAPE generalizes reliably to unseen datasets under varying learning- scales, preserving accurate convergence estimation across diverse model familics.
Acael lopddl (ppeced
Accai (spedio) Predicte (epocte)
Conveggaee (spocks)
@egeeegeeeeccc.... (opocite)
4..%%
A.1%
10.9
25.7%
is 15
5.1%
Convergesoo
1.5%
5.5%
15 25
4%
50%,
5--4 16-1
2613
16-1 1e- DeiT-Tany
263
So4 16-1 DistiBERT
26-1
Se-f 16-1
26/5
16-3
263
164
1e-1
3-3
56-1
16-3
5o-l 163 26-1 361
BILSTM
GRU
DenseNe-121
MLP-Mixer
ResMLP
ResNei-SU
Figure 2: LODO evaluation across learning-raad variations. Configurations correspond to typical values such as x 10-4, 10-3, and 2 x 10-3, 3, with predicted versus actual convergence epochs compared for each model.
5.5
Behavior under Optimizer Variation
Figure g iilustrates CAPE's Leave-One- Dathe---ttstt-Out results across optimizer variations, evaluating the sys- tem's robustness to optimization dynamics. Across all architectures, predicted convergence epochs remain closely aligned with actual values, confirming CAPE's ability to generalize across distinct update rules. Re- current models (BiLSTM, GRU) show stable predictions with deviations typically under 10% when switching between AdamW and SGD, while Transformer architectures (DeiT-Tiny, DistilBERT) maintain low error across Adam, AdamW, and Adafactor, indicating that probe features effectively encode optinizer-invarianl characteristics of early training dynamics. Across all models, CAPE accurately reflects the longer convergence trajectorics characteristic of SGD compared to adaptive optimizers such as AdamW. This behavior demon- strates that CAPE not only generalizes across optimizers but also faithfully captures convergence patterns across diverse architectures.
11

---

Published in Transactions on Machine Learning Research (12/2025)
Acael (opada) Prokeedi
Actal (spedae)
Converpence (cpocch)
Convergence (cpocia))
Produced teaecte)
3.7%
100%
-0
3.0%
-0
6.1%
16.9%
33.7%
29.%%
17.0%
3.0%
19
Adaen SGB
Adan
AdmW
Addiette Admi DESiBERT
Adami SED GRU
Adamt:: 50D DenseNat-121
Adaett:
Mii
AdteWW: SGa
Adamn ResNet-50
SGD
BILSTM
Dail-Tiny
MLP-MMser
ResMLP
Figure 3: LODO evaluation across optimizer variations. Fach model family tested with two optimizers (e.g., AdamW and SGD for CNNs/MLPs, Adam and AdamW for Transformers, Adafactor and AdamW for sequence models), comparing predicted and actual convergence epochs.
6 Conclusion and Limitation
We introduced CAPE, a lightweight, probing-based framework that predicts the epochs to convergence without full training. By combining structural descriptors (e.g., parameter and dataset scale, batch size, learning rate) with initialization time probes (gradient-nona) normm statistics, NTK-trace proxy, initial loss), CAPE learns a meta-regressor that gencralizes across architoctures and datasets under validation- basod carly- stopping rule. Empirically, CAPE achieves low absolute error and strong linear correspondence to ground truth, with Pearson correlation of 0.89 in cross-fold evaluation and remaining robust under LODO and LOMO protocols. Across MLPs, CNNs, RNNs, and Transformers, CAPE consistently outperforms static scale-only, prefix-based (LCE), and probe-only variants, enabling practical er-ante convergence estimation for model selection, hyperparameter exploration, and compute planning.
Hardware and runtime mapping. While CAPE is hardware-agnostic by predicting epochs rather than wall-clock time, deployment often requires translating epochs into runtime or cost. Augmenting the feature set with hardware factors (e.g., GPU aepaittcttrr/mmmmoo bandwidth, vCPU count, RAM, storage) would enable end-to-end time and cost prediction.
Meta-dataset coverage. Accuracy depends on the diversity of meta-training data. Broader cov- erage additional modalities, longer sequences, larger image resolutions, and stronger augmentation regimes- further improve robustness under distribution shift.
Scope of probes and criteria. Probes are computed at initialization and labels follow a specific early- stopping criterion. Extending to schedule-aware settings (e.g,, cosine/step decay, cooldown phases) and alternative stopping rules could increase fidelity in scenarios with non-monotone trajectorics.
Extrapolation limits. Although CAPE generalizes well to unseen datasets and model families, extreme regimes (very large/small learning rates or batch sizes, unusual optimizers) may require targeted meta-data or mild task-specific calibration.
Modality-dependent convergence behavior. The current early-stopping rule applies a patience of five consecutive epochs, which aligns well with the training dynamics of the vision datasets included in our meta-dataset. However, the notion of "epoch" varies across modalities: text datasets often contain fewer samples, exhibit tokenization and sequence length dependent variability, and may complete epoch with substantially fewer effective parameter update cycles than vision tasks. Developing modality aware convergence definition, polentially adjusting patience thresholds or stabilization criteria to reflect dataset granularity would require redefining the convergence unit, regenerating the meta-dataset, and recalibrating all evaluation metrics to preserve architectural consistency. We leave this future work, as exploring modality specific convergence signals, particularly for NLP workloads, represents a promising direction for expanding the generality of CAPE.
12

---

The journal of Published in Transactions on Machine Learning Research (12/2025)
References
Steven Adriaensen, Hcrilalaina Rakotoarison, Samuel Müller, and Frank Hutter. Efficient bayesian learning curve extrapolation using prior-data fitted networks. Advances in Newrol Information Processing Systems, 36:19858 -999888--1986,, 2023.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterization. In International conference on machine learning, pp.
PMLR, 2019.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Advances in neural information processing systems, 32, 2019.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Proceedings of the National Academy of Sciences, 121(277)0211188122, 2024.
James Bergstra and Yoshua Bengio. Random search for hyper-paramest optimization. machine learning research, 1311)22113305 2012.
Enric Boix- Adssra and Etai Littwin. Tight conditions for when the ntk approximation is valid. Transactions on Machine Learning Research, 2023.
Leo Breiman. Random forests. Machine learning, 45(1):5 32, 2001.
Jiaxin Chen, Xiao-Ming Wu, Yanke Li, Qimai Li, Li-Ming Zhan, and Fu-lai Chung, A closer look at the training strategy for modern meta-learning. Advances in neural information processing systems, 33:396- 406, 2020.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rmn encoder-decoder for statisti- cal machine translation. arXiv preprint arXiv:1406. 1078, 2014.
Eugenio Clerico and Benjamin Guedj. A note on regularised ntk dynamics with an application to pac-baycsian training. Transactions on Machine Learning Research, 2024.
Adam Coates, Andrew Ng, and Honglak Lec. An analysis of single-layer networks in unsupervisod feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 1-5-223. JMLR Workshop and Conference Proceedings, 2011.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pp. Teee, 2009.
Tobias Domhan, Jost Tobias Springenberg, Frank Hutter, et al. Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves. In IJCAI, volume 15, pp. 3460-8, 8, 2015.
Aleksandr Dremov, Alexander Hägele, Atli Kosson, and Martin Jaggi. Training dynamics of the cooldown stage in warmup-stable-decay learning rate scheduler. Transactions on Machine Learning Research, 2025.
Cheng Gao, Yuan Cao, Zihao Li, Yihan He, Mengdi Wang, Han Liu, Jason Klusowski, and Jianqing Fan. Global convergence in training large-scale transformers. Advances in Neural Information Processing Sys- tems, 37:29213-29284 2024.
X Yu Geoffrey, Yubo Gao, Pavel Golikov, and Gennady Pekhimenko. Habitat: A [Runtime- Based] com- putational performance predictor for deep neural network training. In 2021 USENIX Annual Technical Conference (USENIX ATC 21), pp. 503 521, 2021.
Pulkit Gopalani, Samyak Jha, and Anirbit Mukherjee. Global convergence of sgd for logistic loss on two layer neural nets. Transactions Machine Learning Research, 2024.
13

---

Published in Transactions on Machine Learning Research (12/2025)
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient-based meta-learning as hierarchical bayes. In International Conference on Learning Represenlations, 2018.
Jiechao Guan, Yong Liu, and Zhiwu Lu. Fine-grained analysis of stability and generalization for modern meta learning algorithms. Advances in Newral Information Processing Systems, 35:18487-18880 -85000 2022.
Antonio Gulli, The anatomy of a news search engine. In Proceedings of the 14th International World Wide Web Conference, pp. 880 881, Chiba, Japan, 2005.
James Harrison, Luke Metz, and Jascha Sohl-Dickstein. A closer look at learned optimization: Stability, robustness, and inductive biases. Advonces in neural information processing systems, 35:3758-385 -7733, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Tn Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770 778, 2016. 1780,
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Newral computation, 1997.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Jean Lespiau, et al. Training compute-optimal large language models. arXiv:2203. 1 15556, 2022.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolu- tional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4700-4708, 2017.
Zhiheng Huang, Wei Xu, and Kai Yu. Bidirectional lstm-erf models for sequence lagging. ar preprinl arXiv:1508.0190 2015.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and gencralization in neural networks. Advances in newral information processing systems, 31, 2018.
Kaiyi Ji, Jason D Lee, Yingbin Liang, and Vincent Poor. Convergence of meta-learning with task-specific adaptation over partial parameters. Advances Newral Information Processing Systems, 33:11 15:11490-1111 11500, 2020.
Weisen Jiang, James Kwok, and Yu Zhang. Effective meta-regullariztionn by kernelized proximal regulariza- tion. Advances in Neural Information Processing Systems, 34:26212 26222, 2021.
Daniel Justus, John Brennan, Stephen Bonner, and Andrew Stephen McGough. Predicting the computational cost of deep learning models. In 2018 IEEE international conference on big data (Big Data), pp. 3873-3882 IEEE, 2018
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, et al. Scaling laws for neural language models. arXiv:2001.08300 2020.
Aaron Klein, Stefan Falkner, Jost. Tobias Springenberg, and Frank Hutter. Learning curve prediction with bayesian neural networks. In International conference on learning representations, 2017.
Akshay Kumar and Jarvis Haupt. Early directional convergence in deep homogencous neural networks for small initializations. Transactions on Machine Learning Research, 2025.
Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems, 32, 2019.
Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-basse based approach to hyperparameter optimization. Journal of Machine Learning Research, 18 (185):1-52, 2018.
Dongze Lian, Zehao Yu, Xing Sun, and Shenghua Gao. As-mlp: An axial shifted mlp architecture for vision. arXiv preprint ar 25::2107.0880 2021.
14

---

Published in Transactions on Machine Learning Research (12/2025)
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annuual Meeling of the Associa- tion for Computational Linguistics: Human Language Technologies (ACL-HLT), pp. 142 150, Portland, Oregon, USA, 2011. Association for Computational Linguistics.
neural networks. Advances in Neural Information Processing Systems, 36:64996 65029, 2023. Pierre Marion and Raphaël Berthier. Leveraging the two-limescale regime to demonstrate convergence of
Fangzhou Mu, Yingyu Liang, and Yin Li. Gradients features for deep representation learning. In 8th International Conference on Learning Representations, ICLR 2020, 2020.
Jack Parker-Holder, Vu Nguyen, and Stephen J Roberts. Provably efficient online hyperparameter optimiza- tion with population-bant bandits. Advances in neural information processing systems, 1111177002211111 2020.
Alireza Pourali, Arian Boukani, and Hamzeh Khazaei. PreNeT:: Leveraging computational features to predict deep neural network training time. In Proceedings of the 16th ACM/SPEC International Conference Performance Engineering, ICPE '25, pp. 81 91, 2025.
Tiancheng Qin, S Rasoul Etesami, and Cesar A Uribe. Faster convergence of local sgd for over-parameeerizee models. Transactions on Machine Learning Research, 2022.
Basri Ronen, David Jacobs, Yoni Kasten, and Shira Kritchman. The convergence rate of neural networks for learned functions of different frequencies. Advances in Neural Information Processing Systems, 32, 2019.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceediigg of the IEEE conference on compuler vision and pattern recognition, pp.
2018.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: 2019.
smaller, faster, cheaper and lighter. arXiv preprint ar
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1631-1642, Seattle, Washington, USA, 2013. Association for Computational Linguistics.
Jihoon Tack, Jongjin Park, Hankook Lee, Jacho Lec, and Jinwoo Shin. Meta-learning with self-improving momentum larget. Advances in Neurul Information Processing Systems, 35:6318 6332, 2022.
Tla O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lueas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mlp architecture for vision. Advances in neural information processing systems, 34:24261 24272, 2021.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347 10357. PMLR, 2021.
Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeck, et al. Resmlp: Feedforward networks for image classification with data-efficient training. IEEE transactions pattern anolysis and machine intelligence, 45(4):5314-534) 2022.
Bhavya Vasudeva, Puncesh Deora, and Christos Thrampoulidis. Implicit bias and fast convergence rates for self-attention. Transactions on Machine Learning Research, 2024.
Nikhil Vyas, Yamini Bansal, and Preetum Nakkiran. Empirical limitations of the ntk for understanding scaling laws in deep learning. Transactions Machine Learning Research, 2023.
15

---

Published in Transactions on Machine Learning Research (12/2025)
Ruosi Wan, Zhanxing Zhu, Xiangyu Zhang, and Jian Sun. Spherical motion dynamics: Learning dynamics of normalized neural network using sgd and weight decay. Advances in Newral Information Processing Systems, 34:6380-600 10000 2021.
Haonan Wang, Wei Huang, Ziwei Wu, Hanghang Tong, Andrew J Margenot, and Jingrui He. Deep active learning by leveraging training dynamics. Advances in Neural Information Processing Systems, 35:25171- 25184, 2022.
Mingze Wang and Chao Ma. Early stage convergence and global convergence of training mildly parameterized neural networks. Advances in Neural Informalion Processing Systems, 35:743 756, 2022.
Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: study using real-world human annotations. arXie preprint arXiv:2110.1900 12088, 2021.
Simon Weissmann, Sara Klein, Waiss Azizian, and Leif Döring, Almost sure convergence of stochastic gradient methods under gradient domination. Transactions on Machine Learning Research, 2025.
Mengzhou Xia, Antonios Anastasopoulos, Ruochen Xu, Yiming Yang, and Graham Neubig. Predicting performance for natural language processing tasks. In Procecdings of the 58th Annual Meeting of the Association for Compulational Linguislics, 2020.
Feiyang Ye, Baijiong Lin, Zhixiong Yue, Pengxin Guo, Qiao Xiao, and Yu Zhang. Multi-objective meta learning. Advonces in Neural Information Processing Systems, 01221338221355 2021.
Luca Zancato, Alessandro Achille, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Predicting training time without training. Advances in Neural Information Processing Systems, 146633661455 2020.
Arber Zela, Julien Niklas Siems, Lucas Zimmer, Jovita Lukasik, Margret Keuper, and Frank Hutter. Surro- gate nas benchmarks: Going beyond the limited search spaces of tabular nas benchmarks. In International Conference on Learning Representations, 2020.
Qi Zhang, Yi Zhou, and Shaofeng Zou. Convergence guarantees for rmsprop and adam in generalized-smooth non-convex optimization with affine noise variance. Transactions machine learning research, 2025.
Yilang Zhang, Alireza Sadeghi, and Georgios Giannakis. Meta-learning universal priors using non-injective change of variables. Advonces in Neural Information Processing Systems, 17:111965--11
2024a.
Yu-Ming Zhang, Jun-Wei Hsich, Xin Li, Ming-Ching Chang, Chun-Chich Lee, and Kuo-Chin Fan. Mote-nas: multi-objective training-based estimate for efficient neural architecture search. In Proceedings of the 38th International Conference on Neural Information Processing Systems, pp. 100845 100869, 2024b.
Pan Zhou, Xiaotong Yuan, Huan Xu, Shuicheng Yan, and Jiashi Feng. Efficient meta learning via minibatch proximal update. Advances in Newral Information Processing Systems, 32, 2019.
Zining Zhu, Soroosh Shahtalebi, and Frank Rudziez. Predicting fine-tuning performance with probing. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11534- 11547, 2022.
Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Gradient descent optimizes over-parameterized deep relu networks. Machine learning, 109:467 -92, 2020.
16

---

Published in Transactions on Machine Learning Research (12/2025)
A Appendix
A.1 Proof of Proposition 1: Asymptotic Consistency
Proposition 1 (Asymptotic Consistency). Assume that:
The probing feature extractor ((Je. Pprobe) produces bounded feature vectors z E Rd
The ground-truth mapping h (z) : Tooyy measurable,
The hypothesis class 9 contains sequence of functions that approximate h" arbitrarily well (i.e., G is dense in (P22),
The training examples (z ()), )oul) are iid. samples from a fixed distribution DM-
Then, the empirical risk minimizer
M Σ
9M E arg min (log((((1)
log T conv
geg M j=1
satisfies
lim (xrT:om)) (l)g) log T Temm)) )²| -0000
inf IE (z. (log((z) log ) 9E07
That is, 9M is consistent estimator of the log-space convergence predictor.
Proof. Let = Rd x Rxo, where each sample (z, Toon)) ~ DMM Define the per-example loss function as Lig 2, Toony) = (log (z) log Tconn)2
assuming g(z) > to ensure the logarithm is well-defined.
Define the population risk: and the empirical risk over M samples:
R(g) (i JDD [Lg Teonv)],
1 M
RM( = Σ
z(g (
M 1=1
Let 9M E arggming mingeg Ra We aim to show that
R(gM) inf R(g) M + 00.
gEg
Step 1: Uniform Convergence. the loss class Lg == {(z, T + T[g; 2, T E 9} has finite pseudo-dimension or bounded covering mumber, then by the uniform law of large numbers:
sup RM R(g) 40 as M
gEG
This holds if g is Lipschitz (e.g., tree neural regressors with bounded weights), and both log T and log (zz) are bounded, e.g., via clipping regularization.
Step 2: Approximation Error. By assumption, there exists g" E g such that R(g") = inf Therefore, G can approximate the Bayes-optimal predictor arbitrarily closely.
inf geg R(g).
Step 3: Conclude Consistency. From uniform convergence and richness of G, we have:
lim R(gM) inf R(g), M
9Eg
which implics the consistency of the meta-regressor:
lim E(a.T:...) M M0000
(log - log T T inf R(g)).
gEg
17

---

Published in Transactions on Machine Learning Research (12/2025)
A.2 Supporting Lemma: Generalization of Empirical Risk Minimization
Lemma 1 (ERM Convergence). Let G be hypothesis class of real-valued functions mapping from E to Rs0, and assume that:
The per-sample loss L(g; u, Tonny) gl) log T Tcnoe) )
The examples (z, ) are ii.. from fixed distribution DM-
The function class g has finite VC-dimension or bounded Rademacher complexity,
logg(z) and log Teony are uniformly bounded almost surely.
Then, the empirical risk minimizer
M
Σ
(logg(2(0) - -log Toll)
9M
arg min gEg M jml
satisfies, with high probability,
inf B[L(g; T Toon))] ((MM),
gEg
where (MM + as M xc.
Proof. This follows from standard learning theory results. Since the loss bounded and Lipschitz in (z), and has finite capacity, uniform convergence holds over Lg {I(g: 9):
Rg)(g) - (R()) + as M - oc.
sup gEg
Then, by consistency of ERM under uniform convergence, the excess risk of gM over the best-in-class predictor vanishes:
R(gar)
inf R(g) + 0. gEG
A.3 Ablation Study on Regressor Choices
To justify the regression models used for CAPE and the basclines, we present a comprchensive ablation over three regression families of Ridge, Random Forest, and Gradient Boosling. Table summarizzss performance across 5-fold CV, LODO, and LOMO regimes using MAE, RMSE, and Pearson correlation.
Regressor choice for CAPE. For CAPE, which incorporates both structural descriptors and probing- based features, non-linear tree-based regressors (Random Forest and Gradient Boosting) consistently out- perform Ridge across all regimes: they reduce MAE by 35 75-45% in cross-fold CV and by 25-40% in LODO, while also yiclding substantially higher Pearson corrclations. A similar pattern holds for the CAPE (probe- only) variant: although it operates solely on log G2 and log tree-based models still achieve lower prediction error than Ridge, indicating that even the probe features exhibit non-linear relationships with the conver- gence horizon. These results collectively show that the mapping from initialization-tine statistics to Toonv is strongly non-linear, making ensemble regressors a more suitable choice. Accordingly, the main CAPE model uses Random Forest regressor.
18

---

Published in Transactions on Machine Learning Research (12/2025)
Table 3: Ablation of regressor choices for CAPE, CAPE (probe-only), and the scaling-lar baseline. Lower
MAE/RMSE and higher PearsonR indicate better performance.
Method
Evaluation protocol
MAE
RMSE
PearsonR
CAPE (full feature set)
CAPE Random Forest
Cross-fold (5-fold)
4.63
8.10
0.89
CAPE Random Forest
Leave-one-datasset-on
6.85
10.57
0.81
CAPE Random Forest
Leave-one-model-ont
7.27
11.04
0.79
CAPE
Gradient Boosting
Cross-fold (5-fold)
4.96
8.10
0.89
CAPE Gradient Boosting
Leave-one-dataset-out
7.08
10.73
0.80
CAPE-Gradie Boosting
Leave-one-model-ort
9.04
12.86
0.71
SAPE-Ridge
Cross-fold (5-fold)
8.39
11.84
0.75
CAPE-Ridge
Leave-ome-dataset-oou
9.99
13.86
0.66
CAPE Ridge
Leave-one-moddel-ont
15.22
19.58
0.38
CAPE (probe-only: log lo,
Frobe-only-Randeee Random Forest
Cross-fold (5-fold)
13.10
21.33
0.05
Probe-only Random Forest
Leave-oneedatasset-ott
13.90
21.69
-0.09
Probe-only Random Forest.
Leave-one--oodellonu
15.12
23.12
-0.16
Probe-only Gradient Boosting
Cross-fold (5-fold)
13.31
21.03
-0.1
Saao-oonl-Grraiinn Boosting
Leave-one-dataset-ont
13.44
20.92
0.13
Probe-ouly Gradient Boosting
Leave-one-moodellont
13.94
21.71
-0.18
Probe-only-Rine Ridge
Cross-fold (5-fold)
13.30
21.01
0..8
Probe-only Ridge
Leave-one-dataset-ontt
13.54
21.26
-0.27
Gobb-only-Ridce
Leave-one-moddl-ont
13.16
18.66
0.31
Scaling-law baseline (log P, log N)
Scaling-law Ridge
Cross-fold (5-fold)
11.87
17.23
0.28
Scaling-law-Ridge
Leave-one-dataset-ont
11.81
17.29
0.27
Scaling-law-Rian Ridge
Leave-one-model-ont
12.34
17.82
0.18
Scaling-law-Rande Random Forest
Cross-fold (5-fold)
10.56
16.09
0.45
Scaling-lawe Random Forest
Leave-one-datassl-onn
11.74
16.98
0.35
Scaling Random lan-nadomm Forest
Leave-one-modellontt
14.22
20.01
0.07
Scaling-lawe Gradient -araddientt Boosting
Cross-fold (5-fold)
10.55
16.10
0.45
Scaling-lawe Gradient Boosting
Baating-lw-Grrdiin
Leave-one-dataset-ont
11.73
16.96
0.35
Scaling- law Gradient Boosting
Leave-one-moodel-ont
14.18
19.96
0.07
The scaling-law baseline uses only two static features, (log P, log N}, which induce a nearly linear relationship with convergence steps. While tree-based regressors sometimes achieve slightly better performance in CV CV/LODO, the LOMO regime whose goal is to assess cross-model generalization- shows that Ridge achieves: (i) the lowest RMSE, (ii) the highest Pearson corre- lation, and (iii) the most stable behavior across architectures. Becanse the scaling-laws law baseline is meant represent structurally simple, parametric predictor, ridge regression is the most faithful and robust choice for this baseline, consistent with prior scaling-lay law literature and supported empirically by its superior LOMO performance.
Regressor choice for the scaling-law bascline.
19


Reviews:
[{"review_id": "5Yurki7IOU", "reviewer": "Reviewer_zTMP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes CAPE, a system that predicts training steps to convergence without full training. It extracts features (parameter count, gradient norm, NTK trace, learning rate, batch size, dataset size) at initialization from a small probe batch, then uses an XGBoost meta-regressor to predict convergence steps across MLPs, CNNs, RNNs, and Transformers.\n\nStrength:\n1. Addresses a practical problem for resource planning and model selection\n2. Comprehensive evaluation across four architecture families\n3. Achieves MAE of 3-9 steps using single-batch probing\n\nWeaknesses:\n1. Defines convergence on probe batch (Eq. 3) but claims to predict full-dataset convergence (Eq. 4) without empirical validation\n2. ε ∈ {0.1, 0.15, 0.2} means loss drops to only 10-20% of initial value—not actual convergence\n3. Only toy datasets (32×32 images), short horizons (15-60 steps); no realistic-scale experiments\n4. MAE increases 3-10× on held-out datasets (25-36 vs. 2-9) leading to poor generalization", "claims_and_evidence_yn": "No", "claims_explanation": "The paper's central claim lacks validation. It defines convergence T* on a small probe batch D_probe but claims to predict full-dataset convergence without testing this relationship. No correlation analysis demonstrates probe-batch convergence indicates full-training convergence.\n\nThe convergence criterion ε ∈ {0.1, 0.15, 0.2} is too permissive—reaching 10-20% of initial loss captures only early training, not convergence. No test accuracy is reported at these points. The MAE of 3-9 steps represents 5-60% relative error over 15-60 step horizons.\n\nEvaluation is limited to tiny datasets (MNIST, Fashion-MNIST, CIFAR-10/100). Performance collapses on distribution shift: MAE increases from 2-9 to 25-36 on held-out datasets. No experiments at realistic scales (ImageNet, >1000 steps) exist.", "audience_yn": "Yes", "audience_explanation": "Predicting training costs before execution addresses a real need in ML workflows. The probe-based meta-learning approach is underexplored. Some researchers will be interested in this.\n However, interest is limited by: (1) unvalidated probe→full-training assumption, (2) unrealistic convergence criteria and experimental scope, (3) missing connection to scaling laws literature that provides complementary predictions from model/data scale alone.", "requested_changes": "1. Directly compare convergence steps on probe batches vs. full datasets. Report correlation coefficients. If weak, reframe paper honestly.\n2. Define convergence via validation accuracy plateauing, not arbitrary loss thresholds. Report test accuracy at predicted convergence. Use longer horizons (100-1000+ steps).\n3. Include at least one large-scale dataset (ImageNet-level) with training runs requiring >1000 steps.\n4. Add thorough discussion of neural scaling laws literature (Kaplan et al. 2020, Hoffmann et al. 2022). Analyze relationship between CAPE predictions and power-law scaling. Discuss when scaling law vs. probe-based predictions are preferable.\n5. why LCE with 60 steps underperforms 1-step CAPE. Add iso-budget comparison.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IMY6ghNmjG", "reviewer": "Reviewer_mJvB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a novel method to predicting the training steps needed to reach a desired small loss value. The author(s) first makes an overview of existing related studies, followed by introducing their method, CAPE. Then experiments are made to i) compare the accuracy of CAPE with other prediction methods, ii) investigate the importance of different features, iii) compare the generalization ability of CAPE with other prediction methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "From the experiments in Section 5, the proposed prediction method, CAPE does seem to outperform some other methods significantly in terms of both accuracy and generalization abilities. Here I assume the experimental results shown in the paper are reliable.", "audience_yn": "Yes", "audience_explanation": "The work tackles an essential and interesting topic in training ML models. I strongly agree with the author(s) that current theoretical results are of little use in predicting this in practice, and I agree that having a rough idea of the training steps needed would be helpful in selecting models. The work would also be helpful in developing more useful theories related to convergence of optimizers.\n\nBesides, the paper has a clear structure and provides detailed discussions.", "requested_changes": "My main concerns are related to the selection of features: \n\n1. When computing the average gradient norm and NTK trace proxy, how is $\\theta$ chosen? Is it just one fixed point, e.g., the initialization parameter of the model? In that case the gradient norm cannot imply the steepness of the loss landscape as it does not provide information for gradient of loss at other parameters. PLUS: This might be the reason why (log of) gradient norm and trace proxy only contribute incrementally to the prediction ability of CAPE. \n\n2. I wonder why the initialization parameter, and possibly the loss value at initialization, are not included among the features? It is well-known that initialization of a model greatly affects the training result. \n3. I wonder why the use of model architecture is not included among the features? In particular, how can CAPE know about what model it is given to when it tries to make predictions? \n\nThere are also some technical questions: \n\n1. What does it mean by saying loss “decays smoothly” — e.g. the SGD is a discrete algorithm and makes no sense to talk about its smoothness. \n2. In assumption 3, how sufficient is “sufficient approximation of the model’s training dynamics”? \n3. The notation “L” in eq (3) should be “$\\mathcal{L}$”?", "broader_impact_concerns": "N.A.", "additional_comments": "N.A.", "extra_scores": {}, "extra_text": {}}, {"review_id": "vs6NpLPZuh", "reviewer": "Reviewer_VP7T", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces CAPE (Convergence-Aware Prediction Engine), a system that predicts the number of training steps a deep neural network (DNN) needs to converge on a sampled dataset without actually training it. By briefly \"probing\" the model at initialization with the dataset, CAPE extracts static features like parameter count and dynamic features like gradient norm and a Neural Tangent Kernel (NTK) trace proxy that are then fed into a meta-model trained on a diverse range of architectures and datasets to predict the convergence steps.", "claims_and_evidence_yn": "No", "claims_explanation": "This paper claims to predict the number of steps required for convergence on a full dataset. However, the actual methods only predict the steps to achieve a relative training error decrease on a very small sample of such a dataset. Breaking down these discrepancies: \n* Only a very small portion (32-128 samples) of the full dataset is used to determine the threshold of \"convergence\". No analysis is given whether this is a reliable proxy for larger datasets, where mini-batch gradient descent is typically used. A clear counter-example is CIFAR-100 with 100 classes, where 32 samples is clearly insufficient. This problem is explicitly assumed in Assumption 3 without any empirical evidence.\n* That same small portion is currently used for both training and validation, instead of using held-out data to measure validation loss to determine convergence. The current method measures how well a model can memorize data, while using unseen data as validation shows how the model will generalize to unseen examples, which is much more desirable.\n* The convergence threshold is set as a fixed fraction the initial training loss, but the initial point in the loss landscape may not be a good indication of how much lower the \"converged\" loss may be, even relatively. The standard method for proximity to convergence from the early stopping literature is using a \"patience\" parameter $p$, measuring the validation loss after each training epoch and using the point where the validation loss stops decreasing for $p$ epochs.", "audience_yn": "No", "audience_explanation": "In its current state, where CAPE only predicts the steps to achieve a relative training error decrease on a very small dataset, this paper is not interesting to TMLR's audience.", "requested_changes": "The fastest way to modify this paper so that all claims are supported are to change the wording of the claims to specify their limits, i.e that CAPE only predicts the steps to achieve a relative training error decrease on a very small dataset. However, as stated above, this is less interesting to the TMLR audience. \n\nThe next obvious change is to change the collection of the meta dataset such that all networks are trained on the full datasets (utilizing only the training split), using a held-out validation set to measure validation loss to determine convergence via patience. However, this will be quite costly per meta-dataset sample compared to the current methods. \n\nThe above are the main issues that are crucial to this work, and fixing them as further discussed will likely require a lot of the paper to be overhauled. I also note further concerns with the current state of the paper as follows.\n* For learning-curve extrapolation (LCE) described in 5.2, 60 steps is supposed to be used to collect full training curve information, but the abstract lists \"typical convergence horizons\" on 15-60 steps (unless these are log-space?). How is LCE not predicting the correct value if it should often be contained within its training curve? This suggests an error in implementation, such as in the model-fitting step.\n* Please report relative errors, not absolute, for the number of training steps. Claiming a mean absolute error of 3-9 steps is befuddling to audience members that may be familiar with current large model training runs being many more orders of magnitude, unless \"3-9\" is in log-space?\n* Referring to \"Bounded Relative Accuracy\" as just \"AvgAccuracy\"/\"average accuracy\" is not accurate. If you continue to choose to report this metric, you need to both use a consistent and descriptive name as well as provide an explicit interpretation of what it represents, as it is not a standard/well-known metric.\n* Some variables are overloaded. For example, $k$ is used for the model output dimension of the feature vector dimension. $T*$ is also doubly defined in Equations 1 and 3. Equation 6 seems extraneous. The batch size and learning rate sweeps do nto needed to be repeated three times.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)