BurnyCoder/eureka-digits-haiku-models

eureka-digits-haiku

Experimental checkpoints. Input definitions, preprocessing and usage limitations are specified by the registered protocol and reproduction guide.

Experiment	Method	Seed	Split	Accuracy	Seconds	Device
class_prior_s0	class_prior	0	test	0.102778	0.026	cpu
logistic_s0	logistic	0	test	0.955556	0.107	cpu
pilot_regularized_s0	regularized	0	validation	0.966574	6.805	cuda
regularized_s0	regularized	0	test	0.963889	0.486	cuda
regularized_s1	regularized	1	test	0.969444	6.630	cuda
regularized_s2	regularized	2	test	0.969444	2.553	cuda
sensitivity_s0	sensitivity	0	test	0.963889	2.525	cuda
sensitivity_s1	sensitivity	1	test	0.969444	2.583	cuda
sensitivity_s2	sensitivity	2	test	0.969444	2.481	cuda
shuffled_labels_s0	shuffled_labels	0	test	0.061111	2.465	cuda
shuffled_labels_s1	shuffled_labels	1	test	0.116667	2.470	cuda
shuffled_labels_s2	shuffled_labels	2	test	0.108333	2.367	cuda
unregularized_s0	unregularized	0	test	0.963889	2.426	cuda
unregularized_s1	unregularized	1	test	0.969444	2.393	cuda
unregularized_s2	unregularized	2	test	0.969444	2.501	cuda

The mean paired accuracy difference is 0.000000, with conditional 95% bootstrap interval [0.000000, 0.000000]. The registered decision is inconclusive. The sample standard deviation across seed effects is 0.000000. The interval uses 10000 paired resamples of test examples and conditions on the trained models.

• analysis_kind: "confirmatory"

• objective: "Exercise the complete research harness on a bounded reproducible study; negative and inconclusive effects are valid."

• question: "Does AdamW weight decay improve held-out accuracy of a small MLP on the bundled digits data?"

• architecture: [64, 128, 10]

• optimizer: "AdamW"

• learning_rate: 0.001

• batch_size: 64

• epochs: 50

• methods: ["regularized", "unregularized", "sensitivity", "shuffled_labels", "logistic", "class_prior"]

• seeds: [0, 1, 2]

• data_boundaries: "Frozen stratified 60/20/20 train/validation/test partition. Split seed 42. Divide pixels by the documented constant 16. Pilot reports validation only. No learned preprocessing or selection uses test data. Generalization is limited to this random example split, not new writers or datasets."

• analysis_plan: "Use 10000 paired bootstrap resamples of test example indices with RNG seed 2026, preserving pairing across methods and all trained seeds. This interval conditions on the trained models and does not estimate training-population uncertainty. Separately show each seed effect and its sample standard deviation. Treat other comparisons as descriptive, with no post-hoc primary selection."

• hyperparameter_rationale: "Small prespecified network and fixed training budget provide fast feedback on an established task. The three decay values are a descriptive local sensitivity check, not a test-selected sweep. Three seeds expose limited initialization sensitivity; they do not support broad population claims."

• S01: Decoupled Weight Decay Regularization (AdamW), Loshchilov & Hutter, ICLR 2019

• S02: Adam: A Method for Stochastic Optimization, Kingma & Ba, ICLR 2015

• S03: torch.optim.AdamW documentation (PyTorch 2.15)

• S04: PyTorch adam.py: decoupled weight decay update (torch 2.14.1 installed locally, plus upstream main)

• S05: sklearn.datasets.load_digits documentation

• S06: Optical Recognition of Handwritten Digits (UCI ML Repository), Alpaydin & Kaynak 1998

• S07: Understanding deep learning requires rethinking generalization, Zhang et al., ICLR 2017

• S08: A Simple Weight Decay Can Improve Generalization, Krogh & Hertz, NIPS 1991 (vol. 4)

• S09: Understanding Decoupled and Early Weight Decay, Bjorck et al., 2020

• S10: Towards Understanding Convergence and Generalization of AdamW, Zhou et al., IEEE TPAMI 2024

• S11: Troubling Trends in Machine Learning Scholarship, Lipton & Steinhardt, 2018

• S12: How to avoid machine learning pitfalls: a guide for academic researchers, Lones

• S13: How I Think About My Research Process (sequence), Neel Nanda

• S14: How I Think About My Research Process: Explore, Understand, Distill, Neel Nanda, 26 Apr 2025

• S15: Construction of Confidence Intervals, Dalitz, arXiv:1807.03582

• S16: safetensors torch API documentation (save_file, load_file)

• S17: scipy.stats.bootstrap documentation (percentile method, paired option)

• S18: oceCens percci documentation (percentile interval, citing Efron & Tibshirani 1993, p. 168)