Package: baytune
An open source project from Data to AI Lab at MIT.A simple, extensible backend for developing auto-tuning systems.License:MITDevelopment Status:Pre-AlphaDocumentation:https://mlbazaar.github.io/BTBHomepage:https://github.com/MLBazaar/BTBOverviewBTB ("Bayesian Tuning and Bandits") is a simple, extensible backend for developing auto-tuning
systems such as AutoML systems. It provides an easy-to-use interface fortuningmodels andselectingbetween models.It is currently being used in several AutoML systems:ATM, a distributed, multi-tenant AutoML system for
classifier tuningMIT's systemfor the DARPAData-driven discovery of models(D3M) programAutoBazaar, a flexible, general-purpose
AutoML systemTry it out now!If you want to quickly discoverBTB, simply click the button below and follow the tutorials!InstallRequirementsBTBhas been developed and tested onPython 3.6, 3.7 and 3.8Also, although it is not strictly required, the usage of avirtualenvis highly recommended in order to avoid
interfering with other software installed in the system whereBTBis run.Install with pipThe easiest and recommended way to installBTBis usingpip:pipinstallbaytuneThis will pull and install the latest stable release fromPyPi.If you want to install from source or contribute to the project please read theContributing Guide.QuickstartIn this short tutorial we will guide you through the necessary steps to get started using BTB
toselectbetween models andtunea model to solve a Machine Learning problem.In particular, in this example we will be usingBTBSessionto perform solve theWineclassification problem
by selecting between theDecisionTreeClassifierand theSGDClassifiermodels fromscikit-learnwhile also searching for their besthyperparameterconfiguration.Prepare a scoring functionThe first step in order to use theBTBSessionclass is to develop ascoringfunction.This is a Python function that, given a model name and ahyperparameterconfiguration,
evaluates the performance of the model on your data and returns a score.fromsklearn.datasetsimportload_winefromsklearn.linear_modelimportSGDClassifierfromsklearn.metricsimportf1_score,make_scorerfromsklearn.model_selectionimportcross_val_scorefromsklearn.treeimportDecisionTreeClassifierdataset=load_wine()models={'DTC':DecisionTreeClassifier,'SGDC':SGDClassifier,}defscoring_function(model_name,hyperparameter_values):model_class=models[model_name]model_instance=model_class(**hyperparameter_values)scores=cross_val_score(estimator=model_instance,X=dataset.data,y=dataset.target,scoring=make_scorer(f1_score,average='macro'))returnscores.mean()Define the tunable hyperparametersThe second step is to define thehyperparametersthat we want totunefor each model asTunables.frombtb.tuningimportTunablefrombtb.tuningimporthyperparamsashptunables={'DTC':Tunable({'max_depth':hp.IntHyperParam(min=3,max=200),'min_samples_split':hp.FloatHyperParam(min=0.01,max=1)}),'SGDC':Tunable({'max_iter':hp.IntHyperParam(min=1,max=5000,default=1000),'tol':hp.FloatHyperParam(min=1e-3,max=1,default=1e-3),})}Start the searching processOnce you have defined ascoringfunction and the tunablehyperparametersspecification of your
models, you can start the searching for the best model andhyperparameterconfiguration by using
thebtb.BTBSession.All you need to do is create an instance passing the tunablehyperparametersscpecification
and the scoring function.frombtbimportBTBSessionsession=BTBSession(tunables=tunables,scorer=scoring_function)And then call therunmethod indicating how many tunable iterations you want theBTBSessionto
perform:best_proposal=session.run(20)The result will be a dictionary indicating the name of the best model that could be found
and thehyperparameterconfiguration that was used:{
    'id': '826aedc2eff31635444e8104f0f3da43',
    'name': 'DTC',
    'config': {
        'max_depth': 21,
        'min_samples_split': 0.044010284821858835
    },
    'score': 0.907229308339589
}How does BTB perform?We have a comprehensivebenchmarking frameworkthat we use to evaluate the performance of ourTuners. For every release, we perform benchmarking
against 100's of challenges, comparing tuners against each other in terms of number of wins.
We present the latest leaderboard from latest release below:Number of Wins on latest Versiontunerwith tieswithout tiesAx.optimize22032BTB.GCPEiTuner1392BTB.GCPTuner25290BTB.GPEiTuner20816BTB.GPTuner21324BTB.UniformTuner1771HyperOpt.tpe1866SMAC.HB4AC1804SMAC.SMAC4HPO_EI22031SMAC.SMAC4HPO_LCB20516SMAC.SMAC4HPO_PI22135Detailed results from which this summary emerged are availablehere.If you want to compare your own tuner, follow the steps in our benchmarking frameworkhere.If you have a proposal for tuner that we should include in our benchmarking get in touch
with us atdailabmit@gmail.com.More tutorialsTo justtunehyperparameters- see ourtuningtutorialhereanddocumentation here.To see thetypes ofhyperparameterswe support
see ourdocumentation here.You can read aboutour benchmarking framework here.See ourtutorial onselectionhereanddocumentation here.For more details aboutBTBand all its possibilities and features, please check theproject documentation site!Also do not forget to have a look at thenotebook tutorials.Citing BTBIf you useBTB, please consider citing the followingpaper:@article{smith2019mlbazaar,
  author = {Smith, Micah J. and Sala, Carles and Kanter, James Max and Veeramachaneni, Kalyan},
  title = {The Machine Learning Bazaar: Harnessing the ML Ecosystem for Effective System Development},
  journal = {arXiv e-prints},
  year = {2019},
  eid = {arXiv:1905.08942},
  pages = {arxiv:1904.09535},
  archivePrefix = {arXiv},
  eprint = {1905.08942},
}