Guess-and-Learn (G&L): Measuring the Cumulative Error Cost of Cold-Start Adaptation
Authors: Roland Arnold
Venue: Accepted by TMLR ()

Abstract: Evaluation of machine learning models typically emphasizes final accuracy, overlooking the cost of adaptation: the cumulative errors incurred while learning from scratch. Guess-and-Learn (G&L) v1.0 addresses this gap by measuring cold-start adaptability - the total mistakes a model makes while sequentially labeling an unlabeled dataset. At each step, the learner selects an instance, predicts its label, receives the ground truth, and updates parameters under either online (per-sample) or batch (delayed) mode. The resulting error trajectory exposes adaptation speed, selection quality, and bias - dynamics invisible to endpoint metrics. G&L defines four tracks (Scratch/Pretrained $\times$ Online/Batch) to disentangle the effects of initialization and update frequency. We formalize the protocol, relate it to classical mistake-bound theory, and estimate a heuristic ``oracle reference band'' for MNIST as a plausibility reference. Baseline experiments on MNIST and AG~News, spanning classical methods (Perceptron, $k$-NN), convolutional architectures (CNN, ResNet-50), and pretrained transformers (ViT-B/16, BERT-base), reveal systematic differences in early-phase efficiency: smaller models can adapt with fewer initial errors, while pretraining benefits vary by domain. Across settings, current models remain well above the oracle band, highlighting an adaptability gap. By quantifying the mistake cost of early learning, G&L complements conventional benchmarks and provides a reproducible framework for developing learners that are not only accurate in the limit but also reliable from the first examples.

Published in Transactions on Machine Learning Research (02/2026)
Guess-and-Learn: Benchmarking the Cumulative Error of Cold-Start Adaptation
Roland Arnold Independent Researcher
rolond. w. www....wdooolmmml...ooo com
Reviewed on OpenReview: https: openreview. net/ formm?? id= ickkhjjRpp
Abstract
Evaluation of machine learning models typically emphasizes final accuracy, overlooking the cost of adaptation: the cumulative errors incurred while learning from scratch. Guess-and- Learn (G&L) v1.0 addresses this gap by measuring cold-start adaptability the total mistakes model makes while sequentially labeling an unlabeled dataset. At each step, the learner selects an instance, predicts its label, receives the ground truth, and updates parameters under either online (per-sample) or batch (delayed) mode. The resulting error trajectory exposes adaptation speed, selection quality, and bias dynamics invisible to endpoint metrios. Online Online/Batch) Batch) to disentangle the effects
G&L defines four tracks (Scratch/ caaetch/Prettinnee of initialization and update frequency. We formalize the protocol, relate it to classical mistake-bound theory, and estimate heuristic "oracle reference band" for MNIST as plausibility reference. Baseline experiments on MNIST and AG News, spanning classical methods (Perceptron, k-NN), convolutional architectures (CNN, ResNet-50), and pretrained transformers (ViT-B/16, BERT-base), reveal systematic differences in carly-phase efficiency: smaller models can adapt with fewer initial errors, while pretraining benefits vary by domain. Across settings, current models remain well above the oracle band, highlighting an adaptability gap. By quantifying the mistake cost of early learning, G&L complements conventional benchmarks and provides reproducible framework for developing learners that are not only accurate in the limit but also reliable from the first examples.
1 Introduction
Modern machine learning systems achieve high accuracy, but this success often depends more scale than adaptability. Most models require large amounts of labeled data, learn slowly from scratch, and adapt poorly to new domains where in-domain labels are scarce. In contrast, truly efficient learners would achieve competence with fewer examples and recover more quickly under distribution shift.
Evaluation practice, however, is dominated by two metrics: final accuracy on a held-out test set and label efficiency (mcasuring how many labelod samples are needed to reach a target performance). While useful, both overlook the cost of adaptation the total mistakes a model makes while learning. Two models may reach similar accuracy but differ drastically in how many errors they incur along the way.
Guess-and-Learn (G&L) addresses this gap by treating cumulative error, not label count, as the primary metric. At each step, a learner selects an instance, predicts its label, receives the ground truth, and updates parameters. The benchmark tracks cumulative mistakes over the full labeling process, producing an error trajectory that directly reflects adaptation speed and efficiency. This perspective especially relevant domains where early mistakes carry real costs: degraded user experience interactive systems Amershi et al. (2014), increased safety risk in autonomous systems, wasted resources when labels expensive obtain.

---

Published in Transactions on Machine Learning Research (02/2026)
Our central question is: How many total errors does model make when labeling every instance in dataset, starting from zero in-domain knowledge? Formally, at step the cumulative error is
Fx 1
where Os is the model's prediction, Vi the ground truth, and 1(- the indicator function.
oracle the answer can be remarkably low. On MNIST, probabilistic analysis Mitzenmacher & Upfal (2017) suggests plausible floor in the single digits (see Appendix C and under rcalistic assumptions accounting for ambiguous instances Yadav & Bottou (2019). mastery is estimated to be possible after fewer than twenly errors. In practice, even stale-of-hhe-rl models make tens hundreds of times more mistakes, underscoring a clear adaptability gap.
For humans- or an idealized clustering-andd-ressnnii
G&L is not a replacement for cxisting benchmarks, but a complementary diagnostic. By quantifying the mistake cost of early learning, it provides a concrete, reproducible metric for assessing model adaptability and for driving the development of learners that are not only accurate in the limit but also reliable from the very first examples.
Contributions.
This paper's contributions are:
A new evaluation protocol (G&L) for measuring cold-start adaptation, defined by four tracks that isolate initialization and update frequency.
A heuristic oracle reference band for MNIST to provide a plausible performance floor for early-phase errors.
A reproducible experimental framework, including open-source code with a pinned commit and fixed seeds.
An analysis of training policy effects (e.g., update cadence, weight resets) that highlights capacity-agility trade-off in early adaptation.
2 Purpose & Scope
The goal Guess-and-Leard (G&L) not simply to produce another leaderboard, but to establish a benchmark that makes early-phase adaptability measurable. By quantilying the mistakes made during cold-start learning, G&zL turns variation in efficiency into concrete, comparable target for model improvement.
Cold-start setting. In G&L, the learner begins with:
Zero in-domain labeled instances.
Optional (ask-agnostic prior knowledge (e.g,, pretrained weights).
requirement to predict label before receiving the ground truth.
No option to abstain from prediction.
Freedom to select the next instance to label by any strategy.
These conditions hold throughout. Any form of manual seeding, few-shot prompting, oracle hints constitutes warm start and is outside the scope of G&L v1.0.
Protocol. The protocol involves sequential, full-pool labeling process. At each step, the learner selects unlabeled example from the pool, (ii) predicts its class, (iii) receives the ground-truth label from oracle, and (iv) updales its parameters based on fixed schedule. Every prediction contributes to cumulative error. The primary performance measure is the final cumulative error, EN, after labeling a pool of N instances.
Available tittps://githie htt.h//.www/.oaaaaaataaatttaaeaa tearn-benchmark (Commit: 5653338).
2

---

Published in Transactions on Machine Learning Research (02/2026)
Tracks. G&L v1.0 defines four tracks that isolate the influence of initialization and update frequency:
Scratch Online (SO): random initialization; update after every labeled example.
Scratch-Batch (SB): random initialization; update after batches size >
Pretrained Online (PO): initialized from pretrained weights; update after every labeled example.
Pretrained-Batee (PB): initialized from pretrained weights; updale after batches of size K 1.
These tracks separate the influence of prior knowlodge (scratch vs. pretrained) from the influence of update schedule (per-step vs. batch).
Oracle reference band (MNIST). To contextualize MNIST results, include heuristic oracle band: estimate of the minimum plausible number of errors under idealized assumptions (e-g., perfect clustering with immediate generalization after one labeled example per class). Its width reflects uncertainty from factors such as class imbalance, intra-class variation, and label noise Northcutt et al. (2021). This heuristic estimate is not a formal bound but an illustrative reference, included to highlight the large gap between current. systems and efficient learning under ideal conditions (Appendix C
3 Relation to Existing Paradigms
Guess-and-Leam (G&L) borrows familiar scttings from existing paradigms but alters the objective mcasure the cumulative error cost of cold-start adaptation.
Active learning. Pool-based active learning seeks to minimize label queries needed to reach target accuracy Cohn et al. (1994 Settles (2009). G&L uses the same interactive pool-selection setting but inverts the objective: it measures the total mistakes made while labeling the entire pool. Two reference points are useful: (i) random baseline: for balanced classes, uniform guessing incurs E[Ex]= = N(1 - 1/C); and (ii) an existence floor: at least C mistakes are unavoidable the worst case, since the learner must encounter least one example of each class before knowing it exists Angluii (1988). The combination of pool- baseb
selection with sequential model updates often termed Interactive Machine Learning Ammersii et al. (2014)
Fails & Olsen Jr (2003), where the Iearner's goal is to minimize the cost of the interaction (cumulative errors) rather than just the final generalization error.
Online learning. The G&LL Scratch Online (SO) track is a direct empirical analogue of the mistake- bound model Littlestone (1988). For linearly separable data with margin and diameter R, the Perceptron has the classic bound Ey S. R R2//² Novikoff [1962); Littlestone (1988). G&L-SO instantiates this setting on modern, high-dimensional data, with the added twist that the learner actively selects the sequence of examples.
KWIK learning (abstention). "Knows What Tt Knows" frameworks allow abstention until confidence is high, thereby avoiding mistakes Li et al. (2008). G&L forbids abstention; every instance must be predicted. This models mandatory-decision scenarios (e.g., medical triage, autonomous navigation) and ensures the full cost of uncertainty is counted.
Curriculum & self-paced learning. Acquisition strategies impose a curriculum Bengio et al. (209 Kumar al. (010)). Uncertainty sampling tends toward "hard-first", while confidence sampling is "easy-first" Heuristically, if strategy maintained per-step margin "t 2 Ymin; the Perceptron bound tightens to
R²
EN
<
7
This motivates margin-aware sampling, though it not a guarantee for the heuristics we evaluate.
Prequential evaluation. G&L's prodict-then-uddtt loop mirrors prequential evaluation data streams Dawid (1984): Gama al. (2013), with three differences: (i) a fixed pool in cold-start; (ii) cumulative error is the primary metric; and (iii) the learner controls instance order.
3

---

Published in Transactions on Machine Learning Research (02/2026)
4 Methodology
4.1 G&L Protocol
At each step the learner:
(i) selects one unlabeled instance using a chosen acquisition strategy;
(ii predicts its label;
(iii receives the ground-truth label from an oracle;
(iv) updates its parameters according to a predefined schedule.
Predictions are mandatory (no abstontion), and errors are accumulated over time. The primary metric is final cumulative error Ex after all N instances are labeled.
4.2 Benchmark Tracks
We cross two factors- initialiialiiioii and update schedule to form four tracks (Table Online tracks update after each instance; batch tracks update after groups of size (default 50 unless otherwise specified).
Table 1: G&L tracks formed by the cross of initialization and update schedule.
Track
Initialization
Update Schedule
Batch Size (K)
SO
Scratch
Online
G&L SB
Scratch
Batch
> 1 (default K K=50)
G&L-PO
Pretrained
Online
G&T.-PB
Pretrained
Batch
> 1 (default K=50)
4.3 Acquisition Strategies
We evaluate six established strategies:
Random: uniform sampling without replacement.
Confidence (easy-first): select æ maximizing max, Po(y x) Bengio et al. (0099)
Least-confidence:: select x that minimizes max, Po(v a:) Lewis & Gale 1994).
Margin: select with smallest gap Pı Pa between the top two class probabilities (2007); Tong & Koller (2001).
Schein & Ungar
Entropy: select with largest Eo Po x)) log Po x) Settles 2009).
K-Center Greedy: selects such that the maximum distance from any unlabeled point to its
nearest labeled example is minimized Sener & Savarese (018)).
These are interpretable baselines to reveal how acquisition logic shapes the error trajectory; we do not claim optimality for G&L.
4.4 Implementation Details
All models produce class-probability vectors Po æ) with prediction 9 arg max, Po(y 2). Models producing logits use a softaax Bridle (1990); k-NN probabilities derive from distance-weighter neighbor votes (normalized) Cover & Hart [1967). . For k-NN, 'learning' consists of adding each newly labeled instance weighted predictions. Ties (in acquisition
the pool of reference examples used for subsequent prediction) are broken uniformly at random. The pool is processed without replacement until exhaustion. Unless noted, results averaged over fixed seeds.
4

---

Published in Transactions on Machine Learning Research (02/2026)
4.5 Experimental Setup
Datasets. MNIST (image classification) LeCun et al. (1998) and AG News (texi classification) Zhang et al. (2015). For G&L, the unlabeled pool is drawn from each dataset's official test set and processed without replacement, solely to standardize the unlabeled pool for measuring adaptation cost; we do not report conventional test accuracy here, so this does not create leakage for our target metric.
Models. Vision (MNIST): K-NN (k=7) Cover & Hart (1967 Fix & Hodges (951]). Perceptron Rosenblatt small 3-layer CNN LeCun et al. (1998), Res.ett-50 He et al. (2016) ViT-B/16 Dosovitskiy al.
1958
2021 Vaswani et al. 2017 Tezt (AG News): k-NN, linear Perceptron,
Devlin et al. (0019)
Vaswani et al. [0077]
Pretrained tracks. In PO (Pretrained Online), pretrained backbones are used as frozen feature extractors with a newly initialized linear head trained online. In PB (Pretrained -Btchh), all parameters are fine-tuned end-to-end on the batch schedule.
Acquisition. All six strategies from 4.3 are used.
Updates. Online tracks update every step; batch tracks use K > (default K=50), For ablation analysis, we vary this paramcter across K E {10,50 50, 2000
Reporting. Unless noted, results are averaged over seeds and reported for n=300 early-phase subsets; full-pool curves are included for MNIST to contextualize the oracle band.
Hyperparameter tuning. To reflect realistic cold-start usage, hyperparameters (e.g., learning rates, architecture depth) were selected based on standard conventions for final accuracy convergence rather than explicitly tuned to minimize cumulative error. This focuses the evaluation on the adaptability of standard architectures they are lypically instantiated.

---

Published in Transactions on Machine Learning Research (02/2026)
5 Results and Analysis
We organize findings into four themes: model adaptability, strategy effects, training policies, and cost performance trade-offs.
5.1 The Adaptability Gap
Most models are far from optimal in early adaptation. On MNIST, the best-performing ViT-B/16 track pretrained- batch (PB)-accuul) accumulated 136.0 + 4.75 errors at n = 300 under Random acquisition (mean standard error across sceds), finding visualized Figure 1 In contrast, a conceptual clustering oracle-- requiring only one representative example per suggests 7-12 mistake plausibility band under idcalizod assumptions (see Appendix C We note that datasets with significant label noise would inherently possess higher error floors, though the magnitude of the current adaptability gap suggests that substantial room for improvement remains regardless of this floor. For AG News, whose class boundarics are more ambiguous (e.g., World vs. Business), and where prior work reports nontrivial disagreement and label noise Northcuti
et al. 2021 we do not provide an oracle estimate.
G&L Track Comparison on Full MNIST Pool
Model (Track)
Perceptron (SO)
VIT-B-16 16 (PB)
1750
Reelistic Gracle (7-12 Emors)
1500
Cumulative Errors
750
500
250
0
2000
4000
8000
8000
10000
Number of Labeled Samples
Figure 1: MNIST cumulative error trajectorics under Random acquisition. We compare the Perceptron (SO) and ViT-B/16 (PB) against heuristic oracle band (7-12 errons). All models remain well above the band during early adaptation, highlighting the adaptability gap.
5.2 Capacity vs. Agility
Adaptability is not a simple function of size. cold-start, smaller online learners can outperform larger models carly on (Figure 2). On the first 300 MNIST samples, the Perceptron incurs fewer cumulative errors than both CNN and N-. High capacity (ResNet-50, ViT-B/16) does not immediately translate into efficient boundary formation when labeled data are scarce.

---

Published in Transactions on Machine Learning Research (02/2026)
MNIST CSL-S8_ (n-300, Rendom)
MNIST GBL-PB (n-300, R ResNe150, VIT-8-16
Perceptron KNN, CNN
Porcoptton
Rushett)
LNB
43-18
1-NN
Curdatoo Eioes
Outuktive
150
100
0
Soargrao
Ldbicee Stmpper
Figure 2: Capacity vs. agility on MNIST (n=300, Random acquisition). Left: small SO models (Perceptron,
k-NN, CNN). Right: pretrained PB50 models (ResNet-50, ViT-B/16). Smaller online learners adapt faster.
5.3 Impact of Acquisition Strategies
Effects are task- and model-dependent,, as shown in Figure On MNIST with CNN (SO), Confidence (casy-first) yields the lowest cumulative error over the first 300 samples. Uncortainty-based strategics (Entropy, Margin, Least-Confidence) underperform even Random sampling, likely because they front-load ambiguous cases that inflate early mistakes. On AG News with BERT- base (PO), strategy differences are small and often within variance bands in the first 300 samples; strong priors dominate.
However, geometric sampling proves highly effective in specific regimes: as shown in Table 2 K-Center Greedy reduces cumulative error by 15.9% for k-NN MNIST and 11.7% for ResNet-50 compared to random sampling. This highlights clear "geometric win" for models that rely heavily on feature-space distance, though the advantage diminishes for text models (AG News) where density-based uncertainty often prevails.
Table 2: Maximum potential error reduction: Comparing the peittrreformingggg strategy (post-hoc) against the random baseline for each track. This highlights the upper bound of efficiency gains available with current methods. Min Error reports the cumulative error count (n = 300). Green indicates error reduction (improvement); red indicates degradation.
Dataset
Model
Track
Best Strategy
Min Error
Random
A Error (%)
AG News
BERT- -bse
G&LLPBBso
Confidence
84.8 + 11.8
101.2 + 8.9
-16.2
AG News
Text k-NN
G&IL-BBo
Confidence
128.4 + 11.5
138.6 + 9.2
-7.4
AG News
Text Perceptron
G&-L-SO
Confidence
141.8 9.9
189.0 4.3
-2.00
MNIST
K-NN
G&LLSB3o
K-Center Greedy
104.4 7.3
124.2 + 9.3
-15.9
MNIST
CNN
G&LL-SO
Confidence
69.2 - 6.6
92.8 = 8.4
-25.4
MNIST
Perceptron
G&LL-OO
Confidence
98.8
143.2 - 6.4
-31.0
MNIST
50
G&LL-PO
K- K-Center Greedy
238.2 7.7
4.3
-11.7
MNIST
ViT-B/16
G&:L-PBso
K- K-Center Greedy
3.0
135.4 14.1
+11.1
5.4 Protocol Validation and Ablations
We performed ablation studies to validate the robustness of the G&L protocol and quantify the impact of update frequency and model persistence.

---

Published in Transactions on Machine Learning Research (02/2026)
MNIST CNN, 80 (1-300)
News BERT-bese, PO (h=300)
Random
Randon
Camkderce
dentidence
Loast-Oarf.
Least-Cort
Entipyy
200
Enroyy
100
KoGanler Graedy
Groady
Co
50
700
%
120
Sourgrao
Labacied Stempher
Figure Strategy effects. Left: MNIST- CNN, SO (n=300): Confidence (casy-first) yields the lowest hoaggg///nari//assttooiddeeee /lesst--oofiieeeeee are worse than Random. Right: AG News BERT-base, PO (n=300): curves are tightly clustered; strategy choice has minimal impact over the first 300 samples.
cumulative error; entropy paaropyymargi/////ee
Update Cadence (K). We analyzed the trade-off between computational cost and adaptation speed by varying the batch size E {10, 50, 200) (Figure Left). While more frequent updates (K 10) generally reduce cumulative error by allowing the model to react faster to mistakes, the relationship is non-monotonic; K 200 unexpectedly outperforms the intermediate K 50. This suggests trade-off where improved gradient estimation from larger batches can partially offset the penalty of delayed feedback.
To verify that G&L measures true adaptation rather than repeated independent. fitting, we evaluated a "forgetful" baseline that resets the model to its original pretrained weights before every batch (Figure 4 Right). As expected, discarding the learned state significantly degrades performance. This negative control serves as a sanity check to confirm that the beuchmark isolates the contribution of the update steps.
Impact of Continuous State (Weight Reset).
5.5
Cost-Performance Trade-offs
Plotting final error against mean wall-clock time (л=300) reveals distinct cost-peeformance frontiers by datasct and track. Each point plots the mcan performance of spocific model-and-stratesn combination; plots are faceted avoid mixing regimes.
On MNIST, SO models define the most efficient frontier. A-NN achieves the lowest error (~80-100) at moderate cost, while the Perceptron is faster but higher-error; the CNN performs poorly in SO. Deferring updates in SB degrades performance. Pretrained vision models (ResNet-50, ViT-B/16) in PO/PB are costly and not competitive on this dataset.
On AG News, pretrained models are essential. In scratch tracks, Text k-NN is a reasonable bascline and outperforms the fast. but error-prone Text. Perceptron. BERT-base in PB (fine-tuning) attains the lowest. errors (~80-18) 130) at the highest cost; PO (frozen features) is weaker. Aggregating across strategies shifts frontiers (e.g., SO MNIST toward k-NN).
8

---

Published in Transactions on Machine Learning Research (02/2026)
A-bbtaioon MNIST (CNN, SB, entropy, n=300)
Roset vs No Reser AG News (BERT-bas basse, PB, 50, antropy, n=300)
250
200
100
Curulaiia
Combetee
80
80
100
50
100
150
200
250 300
%
50
100
150
200
250
300
Labeled Samp
Labeled Samples
Figure 4: Ablations. Left: An ablation on the batch size K for CNN on MNIST (SB track, Entropy acquisition). The impact of update frequency is non-monotonic; while more frequent updates (K = 10) are most effective, a very large batch size (K' 200) unexpectedly outperforms an intermediate one (K = 50). Right: AG News BERT-base, PB50 Entropy (resetting between batches increases errors).
WNIST
VT.O L/s
Toan The Cror
Og
AG News 50
AG News 58,58
AG News PO
AG News PB. 52
10
Moun Dro DART
Wian Maih Clock Tiee (0)
Mees Mool- T9re
Meon Wall Choo: Time
Moor Cloc Time
Figure 5: Cost performance (n=300): mean wall-clock time (log-scale) vs. mean final error, faceted by dalaset (MNIST, AG News) and track (SO, SB, PO, PB). Each point represents the mean performance of specific model and strategy pairing.

---

Published in Transactions on Machine Learning Research (02/2026)
6 Discussion
6.1 Key Findings and Interpretations
Adaptability remains challenging. Even powerful pretrained models accumulate hundreds of early mistakes simple tasks and remain far above the plausible 7-12 error band on MNIST.
Agility can outweigh capacity. On simpler tasks and in carly stages, smaller online models (e.g., Perceptron) oflen incur fewer errors than high capacity architectures, underscoring capacily-agiln tension. The substantial cumulative error observed in high capacity models during early adaptation on clean, highly structured benchmarks such as MNIST suggests that this inefficiency reflects structural properties of current large scale architectures and their training regimes rather than merely dataset noise. If heavily parameterized models struggle to leverage sparse, sequential feedback under controlled, low-noise conditions, it is unlikely that this limitation disappears solely as tasks become more complex. Importantly, there is no theoretical justification to assume that rapid, error-effcient adaptation must inherently trade off against ultimate representational capacity.
Strategy is context-dependent. Uncertainty-based methods can hurt when they front-load ambiguous cases (e.g., MNIST with CNN), yet matter little when strong priors dominate (e.g., BERT on AG News).
Training policies are critical. Update frequency and weight retention substantially affect cumulative error; retaining learned information between batches is especially important.
Cost persorreform is regime-speciffc. No single model excels across all tracks. The value of pretraining and sensitivity to batch updates are task dependent, arguing for regime-aware evaluation.
6.2 From Diagnosis to Design
The diagnostics point conerete opportunities:
Architectures & optimizers designed for rapid, stable per-sample or small-batch updates.
Transfer & meta-learning aimed at improving cold-start error efficiency, not only final accuracy.
Hybrid update schemes that leverage large-model capacity without sacrificing online agility.
Acquisition-upl update co-design optimize instance selection jointly with update dynamics.
6.3 Limitations and Future Work
G&T. v1.0 prioritizes a clear, reproducible protocol, establishing baselines on curated benchmarks. This focus entails specific scope limits that suggest natural directions for future research:
Data scale and noise: We utilized clean, small, and standard datasets to isolate model adaptability from data quality issues. Real-world data is often larger and noisier, and while establishing baselines in controlled regimes is prerequisite for measuring performance in complex environments, reliably scaling G&L to these regimes is natural direction for future work.
Metric incentives: In regimes with extreme class imbalance, cunulative error tracking yield degenerate solutions. A "lazy" model that predicts only the majority class may incur lower cumulative error than an agile model that actively explores to find rare instances. Future extensions for such pathological cases may need cost-sensitive metrics (e.g., cost-weighted error) to align with downstream utility.
Oracle assumptions: The protocol currently forces the learner to accept oracle labels as absolute truth. A more advanced "skeptical learner" extension could allow models to flag potential label noise for human review, rewarding the identification of dataset errors rather than penalizing the model for disagrecing with them.
Ambiguity awareness: The current protocol enforces strict single-label classification. However, real- world inputs often contain intrinsic ambiguity validly belong to multiple categories simultaneously.
10

---

Published in Transactions on Machine Learning Research (02/2026)
Future iterations could support multi-label or probabilistic acquisition strategies, rewarding models that capture this nuance rather than forcing single hard decision.
Generalization: The protocol measures adaptation efficiency within fixed pool; it does not explicitly assess post-adaptation performance on out-of-distribution domains or open-sct recognition lasks where the label space expands over time.
7 Conclusion
Guess-and-Learn addresses a clear gap by measuring the cumulative error cost of learning from a cold-start. By tracking mistakes hep-by-step, it captures how quickly a model becomes useful information that final accuracy and label-efficiency do not provide.
The lour-track design (scratch vs. pretrained, online batch) and full error trajectories expose actionable differences in adaptation: sensitivity to initialization and update policy, the variable impact of acquisition strategies, and regime-specific cost-performance tradc-offs. These distinctions matter in practice: reducing early errors lowers annotation cost, mitigates risk in safety-critical settings, and improves user experience.
the same time, results show a persistent adaptability gap. On MNIST, modern models make many more early mistakes than a plausible oracle. Closing this gap requires optimizing for agility as well as for final accuracy. We offer G&L not only as a benchmark but as design driver: a concrete target and a reproducíble way assess progress toward learners that adapt faster, make fewer mistakes, and become reliable sooner.
8 Acknowledgements
We thank the Action Editor and the anonymous reviewers for their constructive feedback, which helped improve this manuscript.
References
Saleema Amershi, Maya Cakmak, William Bradley Knox, and Todd Kulesza. Power to the people: The role of humans in interactive machine learning. Al Magazine, 35(4):105- 120, 2014.
Dana Angluin. Queries and concept learning. Machine Learning, 2(4):319-342, 1988.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning, In Proceedings of the 26th Annual International Conference on Machine Learning, pp. 41-48, 2009.
John Bridle. Probabilistic interpretation of foed-forward classification network outputs, with relationships to statistical pattern recognition. In Newrocomputing, pp. 227 236. Springer, 1990.
David Cohn, Les Atlas, and Richard Ladner. Improving generalization with active learning. Machine Learning, 15(2):201 221, 1994.
Thomas Cover and Peter Hart. Nearest neighbor pattern classification. IEEE Transactions on Information Theory, 13(1):21-27, 1967.
A Philip Dawid. Present position and potential developments: Some personal views: Statistical theory: The proquential approach. Journal of the Royal Statistical Society: Series A (General), 147(2))278-29)) 1984.
Jacob Devlin, Ming-Wa- Wei Chang, Kenton Loc, and Kristina Toutanova. BERT: Pro-training of deep bidiroctional transformers for language understanding, In Proceedings of the 2019 Conference of the North Americon Chapter of the Association for Computational Linguisties, pp. 4171-4186 2019.
Alexey Dosovitskiy et al. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Jerry Alan Fails and Dan R Olsen Jr. Interactive machine learning. In Proceedings of the 8th International Conference Intelligent User Interfaces, pp. 39 39-45, 2003.
11

---

Published in Transactions on Machine Learning Research (02/2026)
Evelyn Fix and Joseph L Hodges. Discriminatory analysis. nonparametric discrimination: Consistency properties. Technical report, USAF School of Aviation Medicine, 1951.
Edward W Forgy. Cluster analysis of multivariate data: efficiency versus interpretability of classifications. Biometrics, 21(3):768-74) -807 1965.
Joao Gama, Raquel Sebastião, and Pedro Pereira Rodrigues. On evaluating stream learning algorithms. Machine Learning, 90(3):317-38 346, 2013.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
M Pawan Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. In Advances in Neural Information Processing Systems, volume 23, 2010.
Yann LeCun, Léon Botton, Yoshua Bengio, and Patrick Haffner. Gradient- learning applicd to document recognition. Proceedings of the IEEE, 86(11):2278 2324, 1998.
David D Lewis and William A Gale. A sequential algorithm for training text classifiers. In SIGIR '94, pp. 3-12, 1994.
Lihong Li, Michael L Littman, and Thomas Walsh. Knows what it knows: framework for self-aware learning. In Proceedings of the 25th International Conference on Machine Learning, pp. 568 575, 2008.
Nick Littlestone. Learning quickly when irrelevant attributes abound: A new linear-threshold algorithm.
Machine Learning, 2(4))285-318, -38, 1988.
Stuart Lloyd. Least squares quantization in PCM. IEEE Transactions on Information Theory, 28(2):129- 137, 1982.
James MacQueen. Some methods for classification and analysis of multivariate observations. In Proceedings of the Fifth Berkeley Symposium Malhemalical Stalistics and Probobilily, volume 1, pp. 281 297, 1967.
Michael Mitzenmacher and Eli Upfal. Probability and computing: Randomization and probabilistic techniques in algorithms and data analysis. Cambridge University Press, 2017.
Rajeev Motwani and Prabhakar Raghavan. Randomized algorithms. Cambridge Universily Press, 1995.
Curtis G Northeutt, Lu Jiang, and Isaac L. Chuang. Confident learning: Estimating uncortainty in datasct. labels. Journal of Artificial Intelligence Research, 70:1373-14 1111,,11,,,111 2021.
Albert B Novikoff. On convergence proofs for perceptrons. In Proceedings of the Symposium on the Mathematical Theory of Automata, volume 12, pp. 615 622, 1962.
Frank Rosenblatt. The perceptron: probabilistic model for information storage and organization in the brain. Psychological Review, (6)38)88 1958.
Andrew 1 Schein and Lyle H Ungar. Active learning for logistic regression: evaluation. Machine Learning, 68(3):235-200 2007.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations, 2018.
Burr Settles. Active learning literature survey. Technical report, University of Wisconsin-Maddisn Department Computer Sciences, 2009.
Simon Tong and Daphne Koller. Support vector machine active learning with applications to text classification. Journal of Machine Learning Research, 2:45-66 2:45-66,4 2001.
Ashish Vaswani et al. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.
12

---

Published in Transactions on Machine Learning Research (02/2026)
Chhavi Yadav and Léon Bottou. Cold case: The lost MNIST digits. arXiv preprint arXi:: 2019.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems, volume 28, 2015.
13

---

Published in Transactions on Machine Learning Research (02/2026)
A Experimental Reproducibility
Environment. All experiments were run on single NVIDIA A100 (40 GB) GPU, using PyTorch 2.0.1, Hugging Face Transformers 4.54.0, and Python 3.11.
Scale. The analysis is based on 579 independent training runs, which produced 484,300 data records. Seeds. Random sceds [0, 2,3,444 4] were fixed across runs.
Code and data. The complete source code, configuration files, and logged results to reproduce this benchmark are publicly available at https://github. com/RolandWArDn saliaroooodgguss-aaaaa gurggguuuss---ae aund--earn-beechaaae (Commit: 5653338). The repository README md provides step-by-step reproduction instructions.
B Supplemental Visualizations
Figure Early-stage adaptability on AG News (n=300). Pretrained BERT shows a clear advantage over scratch models, reflecting the benefit of task-aligned priors.
Early-Stage Adaptability on AG News (n=300)
Model (Track)
Text K-NN (SO)
140
Toxt Porcoptron (8O)
BERT-base (P8)
120
Cumulative Errors
100
10
200
250
300
Labeled Samples
Figure 6: AG News carly-stage adaptability (n=300).
Figure Stale-stage adaptability MNIST (π=300). The Perceptron adapts quickly and outperforms deeper pretrained models carly on, illustrating the capacity-agility tradc-off.
14

---

Published in Transactions on Machine Learning Research (02/2026)
Early-Stage Adaptability on MNIST (n=300)
Model (Track)
175
Perceptron (SO)
ResNet50 (PB)
VIT-B-16 (PB)
150
125
Cumulative Errors
100
75
100
150
200
250
300
Labeled Samples
Figure 7: MNIST carly-stage adaptability (n=300).
C Heuristic Oracle Floor for MNIST
We provide a practical, heuristic estimate plausible floor on errors for MNIST the "oracle floor" based on a simple cluster-firee first oracle and an audit of residual ambiguity. This an illustrative reference, not a formal bound.
Cluster-First Oracle
Setup. (i) Embed the unlabeled pool; (ii) partition into C=10 clusters (e.g., k-means MacQueen [1967]
Lloyd
1982
Forgy 1965 )); (iii) query one medoid per cluster and assign its label to the cluster.
Mapping cost. Associating C abstract clusters to C class names is a coupon-collector variant with feedback
Mitzenmacher & Upfal (0017); Motwani Raghavan (1995). The expected number of mapping errors is
E Fmapp
C Hc-
He
(2)
j=1
For MNIST (C=10), this gives EmEm]]] Ho 7.07.
Residual ambiguity. Even with near-perfect clustering, sources remain:
Boundary cost (Ehouna): cluster-purity audits suggest ~2 borderline cases (e.g., vs. 7, 0 vs. 6).
Noise/ hatie//mbbiuttyy cost (Enoise): label audits identify ~446 6 ambiguous mislabeled instances (e.g., via Cleanlab; see Northcutt et al. (2021) Yadav & Bottou (2019)).
These overlap with one another and with mapping steps; they not simply additive.
15

---

Published in Transactions on Machine Learning Research (02/2026)
Oracle band. Combining the mapping expectation (~7 errors) with small overlapping residual yields MNIST oracle band of roughly 7-12 errors. This represents a minimum plausible range under idealized conditions, with remaining mistakes attributable to intrinsic dataset ambiguity rather than learner deficiencies.
16


Reviews:
[{"review_id": "iET8AAMReG", "reviewer": "Reviewer_EYSi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a learning paradigm called \"Guess and Learn\" that integrates the pool-based active learning where one can observe the full pool of samples and selects arbitrary one sample, and the online learning where one must make immediate decisions for each selected sample and suffer from a cumulative regret. The paper considers the variants of training from scratch or training from pretrained, selecting per sample or per batch, and different query strategies including random, selecting the most confidence, and selecting the least confidence. The experiments are conducted mainly on 2 types of dataset and model combinations: MNIST and CNN, AG News and BERT. The author(s) suggest 5 key findings: (1) pretrained model may still suffer in the initial stage; (2) smaller models may learn faster in the initial stage; (3) uncertainty sampling strategy may hurt the performance from the cold start; (4) batch size and resetting or not are important in the performance; (5) no dominant combination that outperforms others in all scenarios.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The experimental results support the above five claims, although not so completely and thoroughly. In fact, I think the author(s) are too ambitious to \"define\" a learning protocol for all the tasks and models while the content only contains two very limited datasets (MNIST in vision and AG News in NLP) and two very limited models (CNN and BERT). I wanted to say that the author(s) should include more experiments spanning over different datasets and models to systematically support their claims, but I think it is too unrealistic. Nevertheless, the learning protocol differs from previous literature. That is the only reason I selected \"Yes\" for this question.", "audience_yn": "No", "audience_explanation": "1. All the ingredients of this \"Guess and Learn\" paradigm have already been investigated: online learning for cumulative regret and immediate decision, active learning for intentionally deciding which sample to query, tuning the batch size, training from a pretrained weight, and so on. As for the paradigm itself, the most important two ingredients are the online learning and the active learning. But that's where the problem becomes weird: for the online learning, you cannot observe the full sample pool, and you have to make a decision every time a new sample \"arrives\", which models some real-world scenes such as the online nature of customers in some platforms; for the pool-based active learning, you can observe the full pool and decide which one to query, and the goal of the learning focuses more on the \"learner\" side (finally reaching a good model and querying a label suffers a constant cost rather than related to the prediction result) rather than the \"decision-maker\" side (suffering a cumulative regret). The two ingredients actually contradicts with each other. I cannot imagine a real-world scene that this learning paradigm is practical. Could you please help me remind one where only this paradigm can deal with but not the online convex optimization or pool-based active learning alone?\n\n2. The five key observations are all well-known results, and there is nothing new. We all know that pretrained models need fine-tuning and that's why fine-tuning is adopted. Also, smaller models require fewer samples and are of smaller sample complexities are known. From the area of active learning, people have already known that the uncertainty sampling suffers when the labeling budget is small and the model is trained from cold start, and that's why covering-based methods are introduced (see \"Active Learning Through a Covering Lens\" for just one example). As for the batch size's impact on the learning outcome, there are also extensive results on it (for example, batch size and learning rate, batch size and smoothness, etc.). As for the delayed feedback model in the online learning setting, it is also well-studied. As for the \"resetting\" strategy in the training, I don't quite see why you need to reset to the initial pretrained model weights after a batch for a meaningful learning paradigm; that discards all the efforts in the training. I think the reason for the \"sublinear regret\" in Figure 4 Right is due to the \"entropy\" querying strategy, and the pretrained model makes fewer mistakes on those it is confident with. Finally, I don't think the \"no dominance among all\" observation is new, and if the author(s) want to establish a publishable result from that perspective, they should focus on a small area with more extensive data and models.", "requested_changes": "N/A", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DzQIrjkCUM", "reviewer": "Reviewer_U1j3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an evaluation protocol called \"Guess-and-Learn (G&L)\" aimed at revisiting the adaptability of machine learning models during the \"Cold-Start\" phase. The authors point out that current evaluation systems overly rely on Final Accuracy or Label Efficiency (the number of labels needed to reach specific performance), thereby ignoring the \"Adaptation Cost\" accumulated by the model while learning from scratch. To quantify this cost, G&L defines the core metric as Cumulative Error (E_N), which is the sum of errors committed by the model during the process of sequentially labeling, predicting, and updating parameters on the entire dataset.\n\n### **strengths**\n\n- The paper reveals a counter-intuitive phenomenon through detailed experiments (especially the analysis in Figure 2 and Figure 5 ): in the cold-start phase, \"bigger\" is not necessarily \"better\".\n- It uses the official test set as the unlabeled pool and processes the entire pool without replacement during experiments, which to a certain extent mitigates the experimental differences caused by uncertainty.\n\n### **weaknesses**\n\n- The paper mainly relies on MNIST and AG News as experimental platforms. These two datasets are widely considered \"solved\" toy problems and cannot represent the real challenges faced by modern machine learning.\n- The baseline methods are outdated and ignore SOTA Active Learning strategies. This leads to the paper's conclusions being limited by \"weak baseline\" settings based on simple uncertainty heuristics.\n- The Oracle Band (7-12 errors) proposed in the paper , while intuitive, is derived based on strong assumptions of \"perfect clustering\" and \"one-shot generalization\", lacking a more universal theoretical lower bound.\n- The definition of \"Cold-Start\" is too narrow and does not consider Prompting and Zero-Shot. Especially for current pre-training-dominant models like BERT and CLIP, the mainstream solution for cold-start is usually Zero-Shot Prompting or In-Context Learning (ICL).", "claims_and_evidence_yn": "No", "claims_explanation": "The submission relies on widely considered \"solved\" toy datasets like MNIST and AG News that cannot represent real modern challenges , and employs outdated baseline methods that ignore SOTA Active Learning strategies, limiting the conclusions to \"weak baseline\" settings based on simple uncertainty heuristics.", "audience_yn": "Yes", "audience_explanation": "The submission reveals a counter-intuitive phenomenon through detailed experiments (especially the analysis in Figure 2 and Figure 5 ): in the cold-start phase, \"bigger\" is not necessarily \"better\".", "requested_changes": "- Experiments on more challenging datasets like CIFAR-100 or GLUE. Relying solely on toy datasets makes it difficult to prove that the trade-off is truly universal.\n- Include SOTA strategies like k-Center Greedy. A benchmark should be compared against the best, not just weaker baselines.\n- For pretrained tracks, it would be best to add a Zero-Shot baseline. If CLIP without updates outperforms fine-tuning, that would be a crucial finding.\n- Explicitly state the Oracle's limitations in the main text, noting that it might be inaccurate on datasets with high noise.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mD6i0QlzVq", "reviewer": "Reviewer_K2nJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Guess and Learn, a novel and well-motivated evaluation framework for measuring the ‘cold-start’ adaptation cost of machine learning models. The primary contribution is a shift in the evaluation paradigm from traditional final accuracy to cumulative error, defined as the total mistakes a model makes while sequentially learning an entire unlabelled dataset. This metric captures the practical costs of early learning in real world applications. The G&L protocol is rigorously defined through a sequential ‘predict then update’ loop and features an insightful four track design that systematically disentangles the effects of model initialization and update frequency. Through empirical studies on MNIST and AG News, the paper provides key findings, including the existence of a significant \"adaptability gap\" between current models and a plausible performance floor, and a clear ‘capacity and agility’ trade-off where smaller models can be more efficient in early stages. Overall, the work delivers a reproducible framework that provides a new lens for assessment, urging the community to optimize not just for final accuracy but for rapid and reliable adaptation from the very first examples.\\\nKey strength:\\\n1.The paper addresses a critical but overlooked aspect of model evaluation. The concept of \"cumulative error cost\" is highly relevant for practical applications where early mistakes are costly.\\\n2.The G&L framework is simple, intuitive, and clearly defined. The four-track system is a major strength, allowing for a nuanced and insightful analysis of how pretraining and update policies separately contribute to adaptation performance.\\\n3.The experiments are thorough and lead to valuable insights. The concepts of the \"adaptability gap\" and the \"capacity-agility trade-off\" are well-supported by the results and provide a new vocabulary for discussing model behavior.\\\n4.The authors' commitment to providing an open-source framework is commendable. G&L has the potential to become a valuable diagnostic tool and a standard benchmark for the community, driving research in new directions.\\\nKey weaknesss:\\\n1.The method was only tested on two clean, standard datasets. We don't know if the results would be the same for larger and more chaotic real-world data.\\\n2.The process of checking every single data point one-by-one can be very slow and expensive. This makes it difficult to use with the huge datasets and models common today.\\\n3.The \"oracle reference band\" used for comparison is an educated guess, not a proven, perfect score. Therefore, the \"adaptability gap\" is a helpful illustration, but not a precise measurement.\\\n4.The models' hyperparameters were likely optimized for traditional metrics like final accuracy. These settings may be suboptimal for minimizing cumulative error, meaning the reported performance might not reflect the true potential of these models under the G&L framework.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper's strength begins with its clear and rigorous definition of the Guess-and-Learn (G&L) framework. The authors meticulously detail the sequential learning protocol and its four distinct tracks (SO, SB, PO, PB), creating a well-defined and systematic methodology. This clear framework makes the paper's central claims testable and provides a solid foundation for the experimental work that follows.\\\nBuilding on this strong foundation, the claims are further substantiated by a comprehensive and thorough set of experiments. The authors validate their hypotheses across a diverse range of models and data types, using targeted analyses and ablations to directly support key findings like the \"adaptability gap\" and the \"capacity vs. agility\" trade-off. The results are presented with scientific rigor, including variance bands from multiple seeds, which makes the evidence both accurate and compelling.", "audience_yn": "Yes", "audience_explanation": "This paper offers a fresh and much-needed perspective on model evaluation by focusing on a very practical question: what is the cost of learning from scratch? Instead of just measuring final accuracy, the proposed G&L framework quantifies the cumulative error, the total number of mistakes a model makes as it learns. This is a far more relevant metric for many real-world scenarios where early performance and reliability are critical. Because of this practical focus, the findings will be relevant to a broad TMLR audience, especially those in active learning, transfer learning, and anyone working on benchmarks for more efficient and reliable ML systems.", "requested_changes": "I recommend this paper for acceptance. The work presents an interesting framework and the main claims are adequately supported by the experiments. I have no critical objections, but the following suggestions would help strengthen the submission.\\\n1.Acknowledge the Limited Scope of Evaluation (To strengthen):\\\nThe paper's conclusions are supported by the experiments, but this support is based on two standard, clean datasets. The work would be more robust if the authors added a brief discussion to the limitations section about the potential challenges of applying G&L to more complex, noisy, or imbalanced real-world data.\\\n2.Provide Context on Hyperparameter Tuning (To strengthen):\\\nThe paper would benefit from a brief clarification on how model hyperparameters were handled. A simple statement confirming whether they were tuned for a traditional metric like final accuracy, rather than the proposed cumulative error metric, would provide important context for interpreting the results and understanding the trade-offs involved.", "broader_impact_concerns": "I have no broader impact concerns regarding this work. The paper introduces a new methodology for evaluating machine learning models, not a new application or model with direct societal implications.I think this work does not take any new ethical risks, and a broader impact statement is not necessary.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)