Package: dynast
DyNAS-TDyNAS-T (DynamicNeuralArchitectureSearchToolkit) is a super-network neural architecture
search NAS optimization package designed for efficiently discovering optimal deep neural network (DNN)
architectures for a variety of performance objectives such as accuracy, latency, multiply-and-accumulates,
and model size.BackgroundNeural architecture search, the study of automating the discovery of optimal deep neural network architectures for tasks in domains such as computer vision and natural language processing, has seen rapid growth in the machine learning research community. The computational overhead of evaluating DNN architectures during the neural architecture search process can be very costly due to the training and validation cycles. To address the training overhead, novel weight-sharing approaches known as one-shot or super-networks [1] have offered a way to mitigate the training overhead by reducing training times from thousands to a few GPU days. These approaches train a task-specific super-network architecture with a weight-sharing mechanism that allows the sub-networks to be treated as unique individual architectures. This enables sub-network model extraction and validation without a separate training cycle.To learn more about super-networks and how to define/train them, please see oursuper-network tutorial.AlgorithmsEvolutionary algorithms, specifically genetic algorithms, have a history of usage in NAS and continue to gain popularity as a highly efficient way to explore the architecture objective space. DyNAS-T supports a wide range of evolutionary algorithms (EAs) such as NSGA-II [2] by leveraging thepymoolibrary.A unique capability of DyNAS-T is the Lightweight Iterative NAS (LINAS) that pairs evolutionary algorithms with lightly trained objective predictors in an iterative cycle to accelerate architectural exploration [3]. This technique is ~4x more sample efficient than typical one-shot predictor-based NAS approaches.The following number of optimization algorithms are supported by DyNAS-T in both standard and LINAS formats.1 Objective(Single-Objective)2 Objectives(Multi-Objective)3 Objectives(Many-Objective)GA*'ga'NSGA-II*'nsga2'UNSGA-II*'unsga3'CMA-ES'cmaes'AGE-MOEA'age'CTAEA'ctaea'MOEAD'moead'*Recommended for stability of search resultsSuper-networksDyNAS-T included support for the following super-network frameworks suchs asOnce-for-All (OFA).Super-NetworkModel NameDatasetObjectives/Measurements SupportedOFA MobileNetV3-w1.0ofa_mbv3_d234_e346_k357_w1.0ImageNet 1Kaccuracy_top1,macs,params,latencyOFA MobileNetV3-w1.2ofa_mbv3_d234_e346_k357_w1.2ImageNet 1Kaccuracy_top1,macs,params,latencyOFA ResNet50ofa_resnet50ImageNet 1Kaccuracy_top1,macs,params,latencyQuantization-aware OFA ResNet50inc_quantization_ofa_resnet50ImageNet 1Kaccuracy_top1,model_size,params,latencyOFA ProxylessNASofa_proxyless_d234_e346_k357_w1.3ImageNet 1Kaccuracy_top1,macs,params,latencyTransformerLTtransformer_lt_wmt_en_deWMT En-Debleu(BLEU Score),macs,params,latencyBERT-SST2bert_base_sst2SST2latency,macs,params,accuracy_sst2BootstrapNAS--accuracy_top1,macs,params,latencyVision Transformervit_base_imagenetImageNet 1Kaccuracy_top1,macs,params,latencyImageNet:When using any of the OFA super-networks, the ImageNet directory tree should have a separate directory for each of the classes in bothtrainandvalsets. To prepare your ImageNet dataset for use with OFA you could follow instructions availablehere.WMT En-De:To obtain and prepare dataset please follow instructions availablehere.BootstrapNAS: BootstrapNAS is currently only avaiable through the Python interface. To read more how to use DyNAS-T on BootstrapNAS search space, please refer tothe example notebook.Intel Library SupportThe following software libraries are compatible with DyNAS-T:Intel Neural Compressor (INC)Intel OpenVINO NNCF BootstrapNASGetting StartedTo setup DyNAS-T from source code runpip install -e .or make a local copy of thedynastsubfolder in your
local subnetwork repository with therequirements.txtdependencies installed.You can also install DyNAS-T from PyPI:pipinstalldynastInstalling DyNAS-T withpipwill make adynastcommand available in your CLI.Running DyNAS-TThedynast/cli.py(you can usedynastcommand to invoke this script) template provide a starting point for running the NAS process. An evaluation is the process of determining the fitness of an architectural candidate. Avalidationevaluation is the costly process of running the full validation set. Apredictorevaluation uses a pre-trained performance predictor.supernet- Name of the pre-trained super-network. See list of supported super-networks. For a custom super-network, you will have to modify the code including thedynast_manager.pyandsupernetwork_registry.pyfiles.optimization_metrics- These are the metrics that the NAS process optimizes for. Note that the number of objectives you specify must be compatible with the supporting algorithm.measurements- In addition to the optimization metrics, you can specify which measurements you would like to take during an full evaluation.search_tactic-linasLightweight iterative NAS (recommended) orevolutionary(good for benchmarking and testing new super-networks).search_algo- Determines which evolutionary algorithm to run for thelinaslow-fidelity inner loop or theevolutionarysearch tactic.num_evals- Number of evaluations (full validation measurements) to take. For example, if 1 validation measurement takes 5 minutes, 120 evaluations would take 10 hours.seed- Random seed.population- The size of the pool of candidates for each evolutionary generation.50is recommended for most cases, though this can be treated as a tunable hyperparameter.results_path- The location of the csv file that store information of the DNN candidates during the search process. The csv file is used for plotting NAS results.dataset_path- Location of the dataset used for training the super-network of interest.Single-ObjectiveExample 1a.NAS process for the OFA MobileNetV3-w1.0 super-network that optimizes for ImageNet Top-1 accuracy using a simple evolutionary genetic algorithm (GA) approach.dynast\--supernetofa_mbv3_d234_e346_k357_w1.0\--optimization_metricsaccuracy_top1\--measurementsaccuracy_top1macsparams\--results_pathmbnv3w10_ga_acc.csv\--search_tacticevolutionary\--num_evals250\--search_algogaExample 1b.NAS process for the OFA MobileNetV3-w1.2 super-network that optimizes for ImageNet Top-1 accuracy using a LINAS + GA approach.dynast\--supernetofa_mbv3_d234_e346_k357_w1.2\--optimization_metricsaccuracy_top1\--measurementsaccuracy_top1macsparams\--results_pathmbnv3w12_linasga_acc.csv\--search_tacticlinas\--num_evals250\--search_algogaMulti-ObjectiveExample 2a.NAS process for the OFA MobileNetV3-w1.0 super-network that optimizes for ImageNet Top-1 accuracyandmultiply-and-accumulates (MACs) using a LINAS+NSGA-II approach.dynast\--supernetofa_mbv3_d234_e346_k357_w1.0\--optimization_metricsaccuracy_top1macs\--measurementsaccuracy_top1macsparams\--results_pathmbnv3w10_linasnsga2_acc_macs.csv\--search_tacticevolutionary\--num_evals250\--search_algonsga2Example 2b.NAS process for the OFA ResNet50 super-network that optimizes for ImageNet Top-1 accuracyandmodel size (parameters) using a evolutionary AGE-MOEA approach.dynast\--supernetofa_resnet50\--optimization_metricsaccuracy_top1params\--measurementsaccuracy_top1macsparams\--results_pathresnet50_age_acc_params.csv\--search_tacticevolutionary\--num_evals500\--search_algoageMany-ObjectiveExample 3a.NAS process for the OFA ResNet50 super-network that optimizes for ImageNet Top-1 accuracyandmodel size (parameters)andmultiply-and-accumulates (MACs) using a evolutionary unsga3 approach.dynast\--supernetofa_resnet50\--optimization_metricsaccuracy_top1macsparams\--measurementsaccuracy_top1macsparams\--results_pathresnet50_linasunsga3_acc_macs_params.csv\--search_tacticevolutionary\--num_evals500\--search_algounsga3Example 3b.NAS process for the OFA MobileNetV3-w1.0 super-network that optimizes for ImageNet Top-1 accuracyandmodel size (parameters)andmultiply-and-accumulates (MACs) using a linas+unsga3 approach.dynast\--supernetofa_mbv3_d234_e346_k357_w1.0\--optimization_metricsaccuracy_top1macsparams\--measurementsaccuracy_top1macsparams\--results_pathmbnv3w10_linasunsga3_acc_macs_params.csv\--search_tacticlinas\--num_evals500\--search_algounsga3An example of the search results for a Multi-Objective search using both LINAS+NSGA-II and standard NSGA-II algorithms will yield results in the following format.Quantization-aware SearchThis approach allows you to run search on your FP32 super-network and find optimal model configurations w.r.t. both architecture and Post-Training Quantization policy. DyNAS-T's implementation usesIntel® Neural Compressoras an underlying backend for quantizing models. This search approach is specific to the CPU, and so--device=cpuhas to be used.Example 4.Quantization-aware search on OFA ResNet50 super-network.dynast\--results_pathdynast_ofaresnet50_quant.csv\--dataset_path/ML_datasets/imagenet/ilsvrc12_raw\--supernetinc_quantization_ofa_resnet50\--devicecpu\--batch_size128\--search_tacticlinas\--measurementslatencyaccuracy_top1\--optimization_metricslatencyaccuracy_top1\--seed42Distributed SearchSearch can be performed with multiple workers using theMPI/torch.distributedlibrary. To use this functionality, your script should be called withmpirun/mpiexeccommand and an additional--distributedparam has to be set (DyNAS([...], distributed=True).Note: When run withtorchrun, unless explicitly specified,torch.distributedusesOMP_NUM_THREADS=1(link) which may result in slow evaluation time. Good practice is to explicitly setOMP_NUM_THREADSto(total_core_count)/(num_workers)(optional for MPI).Example 5.Distributed NAS process with two OpenMPI workers for the OFA MobileNetV3-w1.0 super-network that optimizes for ImageNet Top-1 accuracyandmodel size (parameters)OMP_NUM_THREADS=28mpirun\--report-bindings\-xMASTER_ADDR=127.0.0.1\-xMASTER_PORT=1234\-np2\-bind-tosocket\-map-bysocket\dynast\--supernetofa_mbv3_d234_e346_k357_w1.0\--optimization_metricsaccuracy_top1macs\--results_pathresults.csv\--search_tacticlinas\--distributed\--population50\--num_evals250References[1] Cai, H., Gan, C., & Han, S. (2020). Once for All: Train One Network and Specialize it for Efficient Deployment. ArXiv, abs/1908.09791.[2] K. Deb, A. Pratap, S. Agarwal and T. Meyarivan, "A fast and elitist multiobjective genetic algorithm: NSGA-II," in IEEE Transactions on Evolutionary Computation, vol. 6, no. 2, pp. 182-197, April 2002, doi: 10.1109/4235.996017.[3] Cummings, D., Sarah, A., Sridhar, S.N., Szankin, M., Muñoz, J.P., & Sundaresan, S. (2022). A Hardware-Aware Framework for Accelerating Neural Architecture Search Across Modalities. ArXiv, abs/2205.10358.Legal Disclaimer and NoticesThis “research quality code”  is for Non-Commercial purposes provided by Intel “As Is” without any express or implied warranty of any kind. Please see the dataset's applicable license for terms and conditions. Intel does not own the rights to this data set and does not confer any rights to it. Intel does not warrant or assume responsibility for the accuracy or completeness of any information, text, graphics, links or other items within the code. A thorough security review has not been performed on this code. Additionally, this repository may contain components that are out of date or contain known security vulnerabilities.
ImageNet, WMT, SST2: Please see the dataset's applicable license for terms and conditions. Intel does not own the rights to this data set and does not confer any rights to it.