Package: mlos-bench
mlos-benchThisdirectorycontains the code for themlos-benchexperiment runner package.It makes use of themlos-corepackage for its optimizer.It's available forpip installvia the pypi repository atmlos-bench.Table of Contentsmlos-benchTable of ContentsDescriptionFeaturesQuickstartInstall and activate the conda environmentMake sure that you have Azure CLI tool installed and workingGenerate access tokens to interact with Azure resourcesCreate a JSON config with DB credentials OptionalCreate a top-level configuration file for your MLOS setupCreate another config file for the parameters specific to your experimentImportance of the Experiment ID configRun the benchmarkOptimizationResuming interrupted experimentsAnalyzing ResultsDescriptionmlos-benchis an end-to-end benchmarking service that can be independently launched for experimentation but is also integrated withmlos-coreas its optimizer for OS tuning.
Given a user-provided VM configuration,mlos-benchprovisions a configured environment and remotely executes benchmarks on the cloud.
Experiment results (benchmark results & telemetry) are stored as input to themlos-coreoptimization engine loop to evaluate proposed configuration parameters and produce new results.FeaturesWith aJSON5config fileand command line parameters as input,mlos-benchstreamlines workload performance measurement by automating the following benchmarking steps:Set up & clean up benchmark and application configurationEase of use:Mlos-bench abstracts away controls for managing VMs in Azure, e.g., setup, teardown, stop, deprovision, and reboot. Get visibility into VM status through Azure Portal, ensuring that a VM is provisioned & running before issuing commands to it.Versatility:Mlos-bench provides a common interface to control a collection of environments (application, OS, VM), regardless of where or which cloud they come from. This allows changes to easily propagate to all environment layers when a new set of kernel parameters are applied.Efficiency:In adapting an environment to new parameters, mlos-bench optimizes for low re-configuration costs during optimization. For example, considering that not all OS kernel parameter adjustments require a full reboot, as some can be changed during run-time.Run benchmarks in the provisioned environment & standardize results for the optimizerThrough Azure File Share, access docker scripts to run benchmarks & store results as input for optimization. For example, execute Redis benchmark uploaded to the file share, running a benchmark docker container with specified parameters. The file share is mounted to VMs via remote execution, instead of ARM templates.Configurable:Specify a python script in the initial config to post-process & standardize benchmark results. An example post-processing script for Redis benchmarks is included.Local & remote benchmark execution:Benchmarks can be run both locally in Hyper-V and remotely on Azure. Local execution allows better accuracy, while Azure runs are required to estimate the benchmark noise and understand the VM behavior when using cloud storage.Cloud agnostic:Mlos-bench can remotely execute benchmarks on other clouds, outside of Azure - e.g., controls for EC2 instances and ability to provision environments on AWS with Terraform.Persistence:Storage integration is available to persist experiment parameters and track results for re-use, either for analysis during & after trials, or warm-starting future experiments.QuickstartTo get started, we can adapt an example configuration to test out runningmlos-bench.
For these instructions, we will be using Azure for our resources.1. Install and activate the conda environmentFrom here onwards we assume we are in the project root directory.
Ensure you have a conda environment (mlos) set up for executingmlos_bench.
Create and activate the environment with:condaenvcreate-fconda-envs/mlos.yml
condaactivatemlosNote: if you are running inside the devcontainer, this should be done automatically.2. Make sure that you have Azure CLI tool installed and workingInstallation instructions foraz(Azure CLI)can be found here.Note:azcomes preinstalled inside the devcontainer.If necessary, login to Azure and set your default subscription:# If using az cli for the first time, a login will be required:azlogin# Make sure to set your default subscription, RG, and Storage Account for these experiments.# For instance:azaccountset--subscription"My Subscription Name"azconfigsetdefaults.group=MyRG--local
azconfigsetstorage.account=MyStorageAccount--local
azaccountset--subscription"..."3. Generate access tokens to interact with Azure resourcesA script at./scripts/generate-azure-credentials-configproduces a JSON config snippet with necessary Azure credentials../scripts/generate-azure-credentials-config>./global_config_azure.jsoncThis data produced in theglobal_config_azure.jsoncfile is in the format that can be used by our framework.{"subscription":"some-guid","tenant":"some-other-guid","storageAccountKey":"some-base-64-encoded-key",}Note: On Linux, this script also requiresjqto also be installed (comes preinstalled in the devcontainer).4. Create a JSON config with DB credentials (Optional)If you plan to store the information about experiments and benchmarks in a (remote) database like PostgreSQL or MySQL, create a JSON/JSONC file with the DB hostname and password.
Seemysql.jsoncorpostgresql.jsoncconfiguration files for examples with a more complete list of DB parameters supported by underlying theSqlAlchemylibrary.
Save your config in./global_config_storage.jsoncfile.
It should look like this:{"host":"mysql-db.mysql.database.azure.com","password":"database_password"}Any parameter that is not specified in./global_config_storage.jsonwill be taken from the corresponding DB's config file, e.g.,postgresql.jsonc.For database like SQLite or DuckDB, there is no need for an additional config file.
The data will be stored locally in a file, e.g.,./mlos_bench.duckdb.
Seesqlite.jsoncorduckdb.jsoncfor more details.Note: if no storage is specified, a basic sqlite config will be used by default.5. Create a top-level configuration file for your MLOS setupWe provide a few examples of such files in./mlos_bench/config/cli/.
For example,azure-redis-opt.jsoncis a configuration for optimizing Redis VM on Azure and saving the results in a local SQLite database.
Likewise,azure-redis-bench.jsoncis a setup to run a single Redis benchmark (and, again, save the results in SQLite).CLI configs like those are meant to connect several MLOS components together, namely:Benchmarking environment (configured inenvironments/root/root-azure-redis.jsonc);Optimization engine (optimizers/mlos_core_flaml.jsonc);Storage for experimental data (storage/sqlite.jsonc);They also refer to other configs, e.g.Reusable config snippets in"config_path"section, andAdditional config files containing sensitive data like DB passwords and Azure credentials.Make sure that the files./global_config_azure.jsoncand./global_config_storage.jsonyou created in the previous steps are included in the"globals"section of your CLI config.For the purpose of this tutorial, we will assume that we reuse the existingazure-redis-bench.jsoncandazure-redis-opt.jsoncconfigurations without any changes.
In a more realistic scenario, however, you might need to change and/or create new config files for some parts of your benchmarking environment.
We'll give more details on that below.5. Create another config file for the parameters specific to your experimentCopy one of our examples, e.g.,experiment_RedisBench.jsoncand name it after your experiment, e.g.experiment_MyBenchmark.jsonc.In that file, you can specify any parameters that occur in your other configs, namely in"const_args"section of the Environment configs, or in"config"sections of your Service, Storage, or Optimizer configurations.A general rule is that the parameters from the global configs like./global_config_azure.jsoncorexperiment_MyAppBench.jsoncoverride the corresponding parameters in other configurations.
That allows us to propagate the values of the parameters that are specific to the experiment into other components of the framework and keep the majority of the config files in our library immutable and reusable.Importance of the Experiment ID configAn important part of this file is the value ofexperiment_idwhich controls the storage and retrieval of trial data.
Should the experiment be interrupted, theexperiment_idwill be used to resume the experiment from the last completed trial, reloading the optimizer with data from the previously completed trial data.As such this value should be unique for each experiment.
Be sure to change it whenever"incompatible"changes are made to the experiment configuration or scripts.
Unfortunately, determining what constitutes and"incompatible"change for any given system is not always possible, somlos_benchlargely leaves this up to the user.6. Run the benchmarkNow we can run our configuration withmlos_bench:mlos_bench--config"./mlos_bench/mlos_bench/config/cli/azure-redis-bench.jsonc"--globals"experiment_MyBenchmark.jsonc"This should run a single trial with the given tunable values (loaded from one or more files in the"tunable_values"), write the results to the log and keep the environment running (as directed by the"teardown": falseconfiguration parameter in the CLI config).Note that using the--globalscommand line option is the same as addingexperiment_MyBenchmark.jsoncto the"globals"section of the CLI config.
Same applies to all other CLI parameters - e.g., you can change the log level by adding--log_level INFOto the command line.Also, note that you don't have to provide full path to theexperiment_MyBenchmark.jsoncfile - the application will look for it in the paths specified in the"config_path"section of the CLI config.OptimizationSearching for an optimal set of tunable parameters is very similar to running a single benchmark.
All we have to do is specifying theOptimizerin the top-level configuration, like in ourazure-redis-opt.jsoncexample.mlos_bench--config"./mlos_bench/mlos_bench/config/cli/azure-redis-opt.jsonc"--globals"experiment_MyBenchmark.jsonc"--max_iterations10Note that again we use the command line option--max_iterationsto override the default value frommlos_core_flaml.jsonc.We don't have to specify the"tunable_values"for the optimization: the optimizer will suggest new values on each iteration and the framework will feed this data into the benchmarking environment.Resuming interrupted experimentsExperiments sometimes get interrupted, e.g., due to errors in automation scripts or other failures in the system.To resume an interrupted experiment, simply run the same command as before.As mentioned above in theimportance of theexperiment_idconfigsection, theexperiment_idis used to resume interrupted experiments, reloading prior trial data for thatexperiment_id.Analyzing ResultsThe results of the experiment are stored in the database as specified in experiment configs (see above).After running the experiment, you can use themlos-vizpackage to analyze the results in a Jupyter notebook, for instance.
See thesqlite-autotuningrepository for a full example.Themlos-vizpackage uses theExperimentDataandTrialDatamlos_bench.storageAPIsto load the data from the database and visualize it.For example:frommlos_bench.storageimportfrom_config# Specify the experiment_id used for your experiment.experiment_id="YourExperimentId"trial_id=1# Specify the path to your storage config file.storage=from_config(config_file="storage/sqlite.jsonc")# Access one of the experiments' data:experiment_data=storage.experiments[experiment_id]# Full experiment results are accessible in a data frame:results_data_frame=experiment_data.results# Individual trial results are accessible via the trials dictionary:trial_data=experiment_data.trials[trial_id]# Tunables used for the trial are accessible via the config property:trial_config=trial_data.configSee Also:https://microsoft.github.io/MLOSfor full API documentation.