Package: nashpobench2api
NAS-HPO-Bench-II APIOverviewNAS-HPO-Bench-IIis the first benchmark dataset for joint optimization of CNN and training HPs.It helpsa fair and low-cost evaluation/comparisonof joint optimization (NAS+HPO) methodsa detailed analysisof the relationship between architecture/training HPs and performancesOur experimental analysis supportsthe importance of joint optimization.
Please seeour paperfor details.This repo providesAPIfor NAS-HPO-Bench-II to make benchmarking easy.
You canquery our data when evaluating modelsin the search process of AutoML methodsinstead of training the models at a high cost.If you use the dataset, please cite:@InProceedings{hirose2021bench,
  title={{NAS-HPO-Bench-II}: A Benchmark Dataset on Joint Optimization of Convolutional Neural Network Architecture and Training Hyperparameters},
  author={Hirose, Yoichi and Yoshinari, Nozomu and Shirakawa,  Shinichi},
  booktitle={Proceedings of the 13th Asian Conference on Machine Learning},
  year={2021}
}The code for training models ishere.Dataset OverviewThe total size of the search space is192K.
The dataset includesthe exact data of all the models in the search space for 12 epoch trainingthe surrogate data predicting accuracies after 200 epoch trainingArchitecture Search SpaceThe overall CNN architecture is constructed bystacking cellsrepresented as a directed acyclic graph (DAG).
Each edge in the graph indicates one of the four operations.3x3 convolution(ReLU activation, 3x3 convolution with stride 1, then batch normalization)3x3 average poolingwith stride 1Skip, which outputs the input tensorZero, which outputs the zero tensor with the same dimension as the inputIt is based onNAS-Bench-201and the only difference is that we exclude the 1x1 convolution operation from the options.Training HP Search SpaceThe combination ofeight initial learning ratesandsix batch sizesare used.HyperparameterOptionsBatch Size16, 32, 64, 128, 256, 512Learning Rate0.003125, 0.00625, 0.0125, 0.025, 0.05, 0.1, 0.2, 0.4InstallationRunpip install nashpobench2api, and download the API dataset fromGoogle Drive(93.7MB), then put the data in some directory (default: ./data).
This API supports python >= 3.6 (and no external library dependencies).If you want to run the codes inbench_algos, runpip install -r requirements.txt.Getting StartedCreate an API instance to get access to the dataset.from nashpobench2api import NASHPOBench2API as API
api = API('/path/to/dataset')You can get 12-epoch valid accuracy (%) and train+valid training cost (sec.) of the specified configuration.acc, cost = api.query_by_key(
	cellcode='0|10|210',
	batch_size=256,
	lr=0.1 )Here,cellcoderepresents one of the architectures in the search space. As shown in the figure below, the numbers in the cellcode mean the type of operations, and the position of the numbers shows the edge '(A) | (B)(C) | (D)(E)(F)'.In the querying process, theapiinstance remembers and shows the log (what you have queried).
You can reduce the log if setverbose=Falsewhen initializingapi.When the querying process has finished, you can get the test accuracy of the configuration with the best valid accuracy in the queried configurations.results = api.get_results()resultsis a dictionary with the keys below.KeyExplanationacc_transa transition of valid  accuraciesapihave queriedkey_transa transition of keys (=cellcode, lr, batch_size)apihave queriedbest_acc_transa transition of the best valid accuracies (%)apihave queriedbest_key_transa transition of the best keys (=cellcode, lr, batch_size)apihave queriedtotal_cost_transa transition of train+valid costs (sec.)final_accs12-epoch and 200-epoch test accuracies (%) of the key with the best valid accuracyapihave queriedYou can reset whatapihave remebered, which is useful when multiple runs.api.reset_log_data()The examples of benchmarking codes are in thebench_algosdirectory.
Especially,random_search.pyis the simplest code and easy to understand (the core part israndom_search()).Work in ProgressUpload the dataset as DataFrame for visualization/analysis.Upload codes for a surrogate model.Upload the trained models.