Package: powerlift
PowerliftAdvancing the state of machine learning?With 5-10 datasets? Wake me up when I'm dead.Powerlift is all about testing machine learning techniques across many, many datasets. So many, that we had run into design of experiment concerns. So many, that we had to develop a package for it.Yes, we run this for InterpretML on as many docker containers we can run in parallel on. Why wait days for benchmark evalations when you can wait for minutes? Rhetorical question, please don't hurt me.deftrials(task):iftask.problem=="binary"andtask.scalar_measure("n_rows")<=10000:return["rf","svm"]return[]defbenchmark(trial):fromsklearn.ensembleimportRandomForestClassifierfromsklearn.svmimportLinearSVCfromsklearn.calibrationimportCalibratedClassifierCVfromsklearn.model_selectionimporttrain_test_splitfromsklearn.metricsimportroc_auc_scorefromsklearn.pipelineimportPipelinefromsklearn.preprocessingimportOneHotEncoder,FunctionTransformerfromsklearn.composeimportColumnTransformerfromsklearn.imputeimportSimpleImputeriftrial.task.problem=="binary"andtrial.task.origin=="openml":X,y,meta=trial.task.data(["X","y","meta"])# Holdout splitX_tr,X_te,y_tr,y_te=train_test_split(X,y,test_size=0.3)# Build preprocessoris_cat=meta["categorical_mask"]cat_cols=[idxforidxinrange(X.shape[1])ifis_cat[idx]]num_cols=[idxforidxinrange(X.shape[1])ifnotis_cat[idx]]cat_ohe_step=("ohe",OneHotEncoder(sparse=True,handle_unknown="ignore"))cat_pipe=Pipeline([cat_ohe_step])num_pipe=Pipeline([("identity",FunctionTransformer())])transformers=[("cat",cat_pipe,cat_cols),("num",num_pipe,num_cols)]ct=Pipeline([("ct",ColumnTransformer(transformers=transformers)),("missing",SimpleImputer(add_indicator=True,strategy="most_frequent"),),])# Connect preprocessor with target learneriftrial.method.name=="svm":clf=Pipeline([("ct",ct),("est",CalibratedClassifierCV(LinearSVC()))])else:clf=Pipeline([("ct",ct),("est",RandomForestClassifier())])# Trainclf.fit(X_tr,y_tr)# Predictpredictions=clf.predict_proba(X_te)[:,1]# Scoreauc=roc_auc_score(y_te,predictions)trial.log("auc",auc)# Create experiment within database.frompowerlift.benchimportExperimentexperiment=Experiment("postgresql://localhost/powerlift",name="SVM vs RF")# Only run this once for the database (downloads PMLB and OpenML CC18 datasets).frompowerlift.benchimportpopulate_with_datasetspopulate_with_datasets(experiment.store,cache_dir="~/.powerlift")# Run experimentexecutor=experiment.run(benchmark,trials,timeout=10)executor.join()This can also be run on Azure Container Instances where needed.# Run experiment (but in ACI).frompowerlift.executorsimportAzureContainerInstanceazure_tenant_id=os.getenv("AZURE_TENANT_ID")azure_client_id=os.getenv("AZURE_CLIENT_ID")azure_client_secret=os.getenv("AZURE_CLIENT_SECRET")subscription_id=os.getenv("AZURE_SUBSCRIPTION_ID")resource_group=os.getenv("AZURE_RESOURCE_GROUP")store=Store(os.getenv("AZURE_DB_URL"))executor=AzureContainerInstance(store,azure_tenant_id,azure_client_id,azure_client_secret,subscription_id,resource_group,n_running_containers=5,num_cores=1,mem_size_gb=2,raise_exception=True,)experiment=Experiment(store,"SVM vs RF")executor=experiment.run(benchmark,trials,timeout=10,executor=executor)executor.join()Installpip install powerliftThat's it, go get 'em boss.