Package: ml-benchmark
Basht - ABenchmarkingApproach forSustainabileHyperparameterTuningTuning
This repository supplies a job to benchmark hyperparameter tuning.Start upPrerequisitesUbuntu >= 16.04 (not tested on MAC OS)Python >= 3.7.0PyTorchNote: If you run your benchmark on GPU make sure to installCudaand install the correct PyTorch Version, which fits your Cuda version.InstallClone the repository withgit clone <url>Create a Python environment withpython -m venv .venv.Activate your environment withsource .venv/bin/activateUpgrade pip withpip install pip --upgradeIf not already installed installPyTorch and TorchvisionTo install the benchmark and use it locally , clone the repository and switch to the root_folder of the repository and type inpip install -e .. Otherwise you can also install the package via PyPi withpip install ml-benchmarkRayTune ExampleFor running the raytune example experiment add the following after the process above:pip install ray[tune]pip install pandasOptuna Minikube ExampleFor minikube, install therequirements.txtinexperiments/optuna_minikubeif present. Also make sure you installedminikube.
Before starting the minikube example you need to start your minikube vm withminikube start. Then executeexperiments/optuna_minikube/optuna_minikube_benchmark.py.Class ExplanationClassDescriptionMNISTTaskUse it to get the Data for the Model. Please do not change its configurationMLPObjectiveThe Job that needs to be executed. Adjustments should not be neccessary.MLPThe model that is trained over the MNIST Task.Benchmark MethodolegyEach implementation uses a commonexperiment-docker-containerthat represents the full lifecycle of a benchmarking experiment, see the lifecycle figure.LifecycleThe Lifecycle consists of 7 steps, that we describe in detail in the following:.StepDescriptionDeployDescribes all deployment operations necessary to run the required components of a hyperparameter optimization (HPO) framework to run the HPO task. With the completion of this step the desired architecture of the HPO Framework should be running on a platform, e.g,. in the case of Kubernetes it referes to the steps nassary to deploy all pods and services in kubernetes.SetupAll operations needed to initialize and start a trial. For instance the handover of relevant classes to other workers, or the scheduling of a new worker.TrialA Trial defines the loading, training and validation of a model with a specific hyperparameter setting. A hyperparameter setting is one combination of hyperparameters, which can be used to initialize a model. E.g.learning_rate=1e-2etc.LoadPrimarily includes all I/O operations that are needed to provide a Trial with the required data, as well as initializations of the model class, with a certain hyperparameter setting.TrainThe training procedure of a model, which computes model parameters to solve a classification or regression problem. Generally training is repeated for a fixed number of epochs.ValidateThe trained model has to be validated on a given dataset. Validation captures the performance of a hyperparameter setting of a certain model. The performance of this model on the validation set is later used to find the best hyperparameter setting.Result CollectionThe collection of models, classifcation/regression results or other metrics for the problem at hand of each trial. After running of all trials results have to be consolidated for a compairison. Depending on the framework, this step might be a continues process that end once all trails are compleat or a process that is triggered after the framework under test observed the compleation of all trails. However, for this benchmark we allways measure the result collection as the time between the last compleated trail and the identification of the best performing hyperparameter set.TestThe final evaluation of the model which performed the best of all trials on the validation set. The test results are thus the final results for the model with the best hyperparameter setting.Metric CollectionDescribes the collection of all gathered Metrics, which are not used by the HPO framework (Latencies, CPU Resources, etc.). This step runs outside of the HPO Framework.Un-DeployThe clean-up procedure to undeploy all components of the HPO Framework that were deployed in theDeploystep.The docker container stub is locatedhere.CollaborationMake sure to fork the repository. For informations on how to fork readhere. When you start working on your code, in your forked repositry, make sure to create a new branch withcollaborator_name/feature_name. When performing your work assure, that you keep yourmainbranch of your forked repository up to date with themainbrain of this repository as it is a direct copy. How to sync yourmainbranch with the one of this repository can be readhere. Once your work is done submit a pull request to the original repository (this one). A guide can be foundhere. For the Pull Request regular rules apply. Provide a short description of the feature you are adding and make sure your code is in a good shape (decent amount of documentation, no extremely long script files). If you use Python it is recommended to useFlake8.System ComponentsAdresses all components that perform necessary task in an HPO. E.g. Scheduling, I/O Operations etc.
A component is not necessarily one Object within the framework, but it can be a collection of multiple objects performing similar elemental tasks.
A fair compairison between HPO Frameworks would map all HPO Frameworks into these components and measure their performed tasks against each other.