Package: gaplearn
GapLearnGapLearn bridges gaps between other machine learning and deep learning tools. All models can be passed to the functions below regardless of the framework they were built upon (scikit-learn, tensorflow, xgboost, or even good ole numpy).My first package objective is to bring transparency to the often black-boxy model training process by making by making robust post-mortem analysis of the hyperparameter and feature selection processes possible. The functions below also further automate some processes while still giving the user full control of the results.Many features are on their way. Unit tests and full documentation will be added shortly.The source code is available onGitHubInstallationpipinstallgaplearnSee the latest version onPyPISubmodulescvThecvsubmodule will have these classes (sfs has been released):SFSDescription:This is a sequential feature selector that enables you to perform backwards elimination with any model (not just a linear regression).At each step, the feature that has the lowest permutation importance is selected for removal. The permutation importance is measured as the decrease in accuracy by default, but the user can pass any custom scoring function.Improvements on their way:Add forward selection and all subsets testingAdd more built-in scoring functions to for assessing feature permutation importanceCreate custom permutation scoring method to removeeli5as a dependencyMethodsbackwards_elimination(X, y, model, params = {}, fit_function = None, predict_function = None, score_function = None, score_name = 'score', cols = [], verbose = 0)Run the backwards eliminationParams:X: (DataFrame or matrix) with independent variablesy: (iterable) dependent corresponding variable valuesparams: (dict) parameter set for modelmodel: a model architecture to be trained and evaluatedfit_function: (function) the function that will be used to train the model; the function must accept the parametersmodel,X, andy; if this value is not set,backwards_eliminationwill attempt to use your model'sfitmethodpredict_function: (function) the function that will be used to make predictions with the model; the function must accept the parametersmodel, andX; if this value is not set,backwards_eliminationwill attempt to use your model'spredictmethodscore_function: (function) the function that will be used to score the model and determine the feature permutation importance; the function must accept the parametersyandpreds; if this value is not set, the accuracy will be usedscore_name: (str) name of the score calculated by thescore_function; 'score' by defaultcols: (list) the names of the columns in the matrix or DataFrameverbose: (0, 1, or 2) determines the amount of printingget_summary_be()Get a summary of the results by step; this can be used to more robustly and holistically determine which feature set is ideal for your problemget_results_be()Get the results for each observation within each step; this can be used to more robustly and holistically determine which feature set is ideal for your problemget_features_be()Get all the features used in the sequential feature selectionget_set_by_score_be(min_score = None, num_steps = 'all')Get the n feature sets (n is determined bynum_steps) that achieve a score greather than or equal tomin_scoreParams:min_score: (float or int) the minumum score a feature set must achieve to be returnednum_steps: (str or int) the number of feature sets to return; 'all' is the only valid str optionget_set_by_features_be(num_features, max_features = None)Get the all feature sets that havenum_featuresfeatures; ifmax_featuresis set, feature sets with betweennum_featuresandmax_featuresfeatures will be returnedParams:num_features: (int) the number of features the returned feature set should have; functions as a minimum ifmax_featuresis setmax_features: (int) the maximum number of features the returned feature sets should haveExample 1:#### Perform a backwards elimination with sci-kit learn's random forest model ####importpandasaspdfromgaplearn.cvimportSFSX=pd.read_csv('X_classification.csv')y=pd.read_csv('y_classification.csv')fs=SFS()print('The backwards elimination has been run:{}'.format(fs.be_complete))# prints Falsefromsklearn.ensembleimportRandomForestClassifierrfc=RandomForestClassifier()# Run the backwards eliminationfs.backwards_elimination(X,y,model=rfc,params={'n_jobs':-1})# Get the step-by-step summarysummary=fs.get_summary_be()# Alternatively, `summary = fs.summary_be`# Get the predictions and true values for each observationresults=fs.get_results_be()# Alternatively, `results = fs.results_be# Get the features used in the analysisfeatures=fs.features_be# Alternatives, `sorted(list(results['feature to remove']))# Identify which feature set can achieve at least 85% accuracy with the smallest number of featuresat_least_85=fs.get_set_by_score_be(min_score=.85,num_steps=1)# Identify the best model with only 4 featuresfeatures_4=fs.get_set_by_features_be(num_features=4)Example 2:#### Perform a more complex backwards elimination with sci-kit learn's naive bayes model ####importpandasaspdfromgaplearn.cvimportSFSfromsklearn.linear_modelimportSGDRegressormodel_sgd=SGDRegressor(loss='modified_huber',penalty='elasticnet')X=pd.read_csv('X_regression.csv')y=pd.read_csv('y_regression.csv')fs=SFS()# Define a score_functiondefmse(y,preds):score=sum([(preds[i]-y[i])**2foriinrange(y.shape[0])])/y.shape[0]returnscore# Define a predict_functiondefarbitrary_prediction(model,X):preds=model.predict(X)+1# arbitrarily deciding to add 1 to the prediction (realistically, this would be a wrapper for model that don't have a `fit` method)returnpreds# Define a predict_functiondefpredict_w_proba(model,X):preds=[1ifx[1]>0.6else0forxinmodel.predict_proba(X)]returnpreds# Run the backwards eliminationfs.backwards_elimination(X,y,model=model_sgd,predict_function=predict_w_proba,score_function=mse)# Get the step-by-step summarysummary=fs.get_summary_be()# Alternatively, `summary = fs.summary_be`# Get the predictions and true values for each observationresults=fs.get_results_be()# Alternatively, `results = fs.results_be`# Get the features used in the analysisfeatures=fs.features_be# Alternatively, `sorted(list(results['feature to remove']))`# Identify which two feature sets can achieve at least 85% accuracy with the smallest number of featuresat_least_85=fs.get_set_by_score_be(min_score=.85,num_steps=2)# Identify the best models with 3-5 featuresfeatures_3_5=fs.get_set_by_features_be(num_features=3,max_features=5)SearchClusterDescription:This is a hyperparameter grid/random search for clustering algorithmsUnlike other grid/random search algorithms, this one enables you to get the observation-by-observation results from each parameter set so that you can do deep post-mortem analysis of the grid/random search.Documentation:Coming soon. See the documentation for details, in the meantime:https://github.com/awhedon/gaplearnMethodssearch_cluster(model, params, X, cols = None, fit_function = None, label_function = None, centroid_function = None, score_function = None, metric_name = 'score', random = None, centroid = False, verbose = 0)Run the grid/random hyperparameter searchParams:model: a model architecture to be trained and evaluatedparams: (dict) the hypterparameter grid (ex.: {'C': [0.1, 0.5, 0.9], ...})X: (DataFrame or matrix) features to be used for clusteringcols: (list) the names of the columns in the matrix or DataFramefit_function: (function) the function that will be used to train the model; the function must accept the parametersmodel,params, andX; if this value is not set,search_clusterwill attempt to use your model'sfitmethodlabel_function: (function) the function that will use the trained model to attach labels to the unseen data; the function must accept the parametermodel; if this value is not set,search_clusterwill attempt to extract your model'slabels_attribute valuecentroid_function: (function) the function that will extract the cluster centroids from the trained model; the function must accept the parametermodel; if this value is not set,search_clusterwill attempt to extract your model'scluster_centers_attribute valuescore_function: (function) the function that will score the results from the trained model; the function must accept the parametersXandlabels; if this value is not set,search_clusterwill calculate the silhouette scoremetric_name: (str) the name of the evaluation metric; 'score' is the defaultrandom: (int or float) if int, the maximum number of parameter sets to use; if float, the percentage of parameter sets to use (.7 means that 70% of the available parameter sets will be used for testing); default is None, which means that all parameter sets will be usedcentroid: (bool) if the model is centroid-based, you can set this to True and define acentroid_function(or leave it as None if the model has acluster_centers_attribute) so that the cluster centers will be extractedverbose: (0 or 1) determines the amount of printingget_best_model(return_scores = False)get_best_models(n = 1, return_scores= False)get_best_params()get_labels()get_param_results()Example 1:fromgaplearn.cvimportSearchClusterimportpandasaspdfromsklearnimportKMeanssc=SearchCluster()X=pd.read_csv('X_clustering.csv')params={'':,'':[],'':[]}# Run the hyperparameter grid searchsc.search_cluster(model=KMeans,params=params,X=X,centroid=True)# Get the best modelbest_model=sc.get_best_model()# Alternatively, `best_model = sc.best_model`# Get the five best models and their performancebest_models_5,best_models_5_scores=sc.get_best_models(n=5,return_scores=True)# Get the model params that performed bestbest_params=sc.get_best_params()# Alternatively, `best_params = sc.best_params`# Get a summary of the results for each parameter setparam_results=sc.get_param_results()# Alternatively, `param_results = sc.param_results`# Get the labels for each observation for each parameter set's k-fold validation to robustly analyze the differences in performancelabels=sc.get_labels()# Alternatively, `labels = sc.labels_df`Example 2:fromgaplearn.cvimportSearchClusterimportpandasaspdimportrandomfrommy_moduleimportModelClasssc=SearchCluster()X=pd.read_csv('X_clustering.csv')params={'param1':['value1','value2','value3'],'param2':['value1','value2','value3'],'param3':['value1','value2','value3']}deffit_function(model,params,X):model(**params).fit_my_model(X)returnmodeldefscore_function(X,labels):score=random.random()returnscore# Run the hyperparameter random search, testing 50% of the parameter setssc.search_cluster(model=KMeans,params=params,X=X,fit_function=fit_function,score_function=score_function,random=0.5)# Get the best model and resultsbest_model,best_model_scores=sc.get_best_model(return_scores=True)# Get the five best models and their performancebest_models_5,best_models_5_scores=sc.get_best_models(n=5,return_scores=True)# Get the model params that performed bestbest_params=sc.get_best_params()# Alternatively, `best_params = sc.best_params`# Get a summary of the results for each parameter setparam_results=sc.get_param_results()# Alternatively, `param_results = sc.param_results`# Get the labels for each observation for each parameter set's k-fold validation to robustly analyze the differences in performancelabels=sc.get_labels()# Alternatively, `labels = sc.labels_df`Search (in development)Description:This is a hyperparameter grid/random search for both regression algorithms and classification algorithmsUnlike other grid/random search algorithms, this one enables you to get the observation-by-observation results from each parameter set so that you can do deep post-mortem analysis of the grid/random search.data_engThedata_engsubmodule will have these classes:DistributedSQL (in development)Description:This enables users to chunk multi-parameter sql queries and process them on multiple threads.