SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(a) For all NAS methods you compare, did you use exactly the same NAS benchmark, including the same dataset (with the same training-test split), search space and code for training the architectures and hyperparameters for that code? [Yes] We always used the same training pipelines during search and final evaluation.
(b) Did you control for confounding factors (different hardware, versions of deep learning libraries, different runtimes for the different methods)? [Yes] We kept hardware, software versions, and runtimes fixed across our experiments to reduce effects of confounding factors.
(c) Did you run ablation studies? [Yes] We compared different kernels (hWL, WL, NASBOT, and GCN) and investigated the surrogate performance in Section 6. We also analyzed the distribution of architectures during search, impact of the flexible parameterization of the convolutional blocks, test error vs. number of parameters and FLOPs in Appendix J.3.
(d) Did you use the same evaluation protocol for the methods being compared? [Yes]
(e) Did you compare performance over time? [Yes] We plotted results over the number of iterations, as typically done for black-box optimizers, and report the total search cost in Appendices J.3, K.3, L.3 and M.3. For the activation function search, DARTS cell search, and transformer-based searches, we did not compare performance over time, as this is typically not done for these experiments.
(f) Did you compare to random search? [Yes]
(g) Did you perform multiple runs of your experiments and report seeds? [Yes] We ran all search runs on the hierarchical or cell-based NAS-Bench-201 over three seeds, the search run for an activation function for one seed, the search on the DARTS search space on four seeds, transformer searches for one seed, and surrogate regression experiments over 20 seeds. All seeds are reported.
(h) Did you use tabular or surrogate benchmarks for in-depth evaluations? [N/A] At the time of this work, there existed no surrogate benchmark for hierarchical NAS.
3. Best practices for reporting important details
(a) Did you report how you tuned hyperparameters, and what time and resources this required? [N/A] We have not tuned any hyperparameter.
(b) Did you report the time for the entire end-to-end NAS method (rather than, e.g., only for the search phase? [Yes] We report search times in Appendices J.3, K.3, L.3 and M.3. For the hierarchical NAS-Bench-201 experiments, the training protocols are exactly the same for the search and final evaluation (except for CIFAR-10), and, thus, there are no additional costs for final evaluation. For the activation function search experiment, we used the same training protocol for training and evaluation. For the DARTS experiment, we re-trained the discretized architecture on CIFAR-10/ImageNet which required ca. 1.5/5 GPU days on a single/eight NVIDIA RTX 2080 Ti GPU(s). For the nanoGPT experiment, we report the final error on the given validation set. For the sentiment analysis experiment, we used the same training and evaluation protocol.
(c) Did you report all the details of your experimental setup? [Yes] We report all experimental details in Section 6, Appendices J.1, K.2, L.1, L.2 and M.1.