Package: mlautomator
ML AutomatorAuthor: Kevin VecmanisMachine Learning Automator (ML Automator) is an automation project that integratesSequential Model Based Optimization(SMBO) with the main learning algorithms from Python's Sci-kit Learn library to generate a really fast, automated tool for tuning machine learning algorithms.MLAutomatorleverages a library called Hyperopt to accomplish this.Read more about Hyperopt hereWhat is SMBO?SMBO is a form of hyperparameter tuning, like grid search and randomized search.  In contrast to grid and randomized search, however, SMBO usedBayesian Optimizationto build a probability model, through trial and error, that is able to better predict what hyperparameters might produce a better model.  The "sequential" just means that multiple trials are run, one after another, each time testing better hyper parameters by applying bayesion reasoning and updating the existing probability model.The trade-off here is that SMBO models spend more time between each iteration "selecting" the next choice of hyperparameters - but this is accepted because the extra time taken to choose the next hyperparameters is typicallysignigicantlyless than each training iteration.  In other words, SMBO results in:Reduced time tuning hyperparameters compared to grid and random search methods.Better scores on the testing set.Installation:Installation is easy - of coursepipcan be swapped forpip3orpipenv(in a virtual environment)pip install mlautomatorKey features:Optimizes across data pre-processing and feature selectionin additionto hyperparameters.Fast, intelligent scan of parameter space usingHyperopt.Optimized parameter search permits scanning a larger cross section of algorithms in the same period of time.An exceptional spot-checking algorithm.UsageMLAutomatoraccepts a training dataset X, and a target Y.  The user can define their own functions for how these datasets are produced.  Note that MLAutomator is designed to be a highly optimized spot-checking algorithm - you should take care to make sure your data is free from errors and any missing values have been dealt with.MLAutomatorwill find ways of transforming and pre-processing your data to produce a superior model.  Feel free to make your own transformations before passing the data toMLAutomator.Optional data utilitiesI'm building a suite of data utility functions which can prepare most classification and regression datasets.  These, however, are optional -MLAutomatoronly requiresX and Yinputs in the form of a numpy ndarray.fromdata.utilitiesimportclf_prepx,y=clf_prep('pima-indians-diabetes.csv')Once you have training and target data, this is the main call to use MLAutomator...Classification Example: 2-classfrommlautomator.mlautomatorimportMLAutomatorautomator=MLAutomator(x,y,iterations=25)automator.find_best_algorithm()automator.print_best_space()MLAutomatorcan typically find a ~ 98th percentile solution in a fraction of the time ofGridsearch or Randomized search.  Here it did a comprehensive scan across all hyperparameters for 6 common machine learning algorithms and produced exceptional model performance for the classic Pima Indians Diabetes dataset.Best Algorithm Configuration:
    Best algorithm: Logistic Regression
    Best accuracy : 77.73239917976761%
    C : 0.02341
    k_best : 6
    penalty : l2
    scaler : RobustScaler(
        copy=True, 
        quantile_range=(25.0, 75.0), 
        with_centering=True,
        with_scaling=True)
    solver : lbfgs
    Found best solution on iteration 132 of 150
    Validation used: 10-fold cross-validationClassification Example: Multi-classHere are the results from the classic iris dataset, a multi-class classification problem with three classesfromdata.utilitiesimportfrom_sklearnfrommlautomator.mlautomatorimportMLAutomatorx,y=from_sklearn('iris')automator=MLAutomator(x,y,iterations=30,algo_type='classifier',score_metric='accuracy')automator.find_best_algorithm()automator.print_best_space()Best Algorithm Configuration:
    Best algorithm: Bag of Support Vector Machine Classifiers
    Best accuracy : 96.67%
    C : 0.7064
    degree : 2
    gamma : auto
    k_best : 2
    kernel : rbf
    n_estimators : 9
    probability : True
    scaler : None
    Found best solution on iteration 3 of 30
    Validation used: 10-fold cross-validationRegression ExampleML Automator supports regression problems as well. In this example we call the Boston Housing dataset fromsklearn.datasetsusing one of our utility functions.fromdata.utilitiesimportfrom_sklearnx,y=from_sklearn('boston')frommlautomator.mlautomatorimportMLAutomatorautomator=MLAutomator(x,y,iterations=30,algo_type='regressor',score_metric='neg_mean_squared_error')automator.find_best_algorithm()automator.print_best_space()Best Algorithm Configuration:
    Best algorithm: K-Neighbor Regressor
    Best neg_mean_squared_error : 10.41395782834094
    algorithm : kd_tree
    k_best : 11
    n_neighbors : 2
    scaler : StandardScaler(copy=True, with_mean=True, with_std=True)
    weights : distance
    Found best solution on iteration 24 of 30
    Validation used: 10-fold cross-validationModel PersistenceML Automator allows you save fit, save, and load the optimal pipeline discovered by thefind_best_algorithm()method.  A complete workflow would look something like this:fromdata.utilitiesimportclf_prepfrommlautomator.mlautomatorimportMLAutomatorx,y=clf_prep('pima-indians-diabetes.csv')automator=MLAutomator(x,y,iterations=30,algo_type='classifier',score_metric='accuracy')automator.find_best_algorithm()automator.fit_best_pipeline()automator.save_best_pipeline('Path/to/your/directory')# some time later....automator.load_best_pipeline('Path/to/your/directory')Note that MLAutomator is storing the entiretransform/feature selection/modelpipeline for you so that none of the prerequisite processing needs to be done when you need to make predictions on out-of-sample data.Existing Algorithm SupportMLAutomatorcurrently supports the following algorithms:Classification:XGBoost ClassifierRandom Forest ClassifierSupport Vector MachinesNaive Bayes ClassifierStochastic Gradient Descent Classification (SGD)K-Nearest Neighbors ClassificationLogistic RegressionRegression:XGBoost RegressorRandom Forest RegressorSupport Vector Machine RegressionSGD RegressionK-Nearest Neighbors RegressionUnless otherwise declared using thespecific_algosargument, MLAutomator will scan all algorithms to find the best performer.