Package: orpheus-ml
OrpheusWhat is Orpheus?Orpheusstands forOptimized Robust Pipelines for Heuristic Ensemble Utilization and Selection.It provides a tool for data scientists and machine learning engineers to automate pipeline construction and optimization, as well as experiment with various combinations of preprocessing techniques and estimators. Orpheus is build on top of thescikit-learnlibrary and is compatible with all scikit-learn estimators.It is a Python package designed to automate the process of building and optimizing machine learning pipelines. These pipelines are different from the conventional Pipeline class from Scikit-Learn, in the sense that a pipeline can contain multiple estimators instead of just one. This class inherits from the Scikit-Learn Pipeline class and is calledMultiEstimatorPipeline.Some common use-cases for Orpheus include:Building and optimizing pipelines for regression and classification problems.Preprocessing data using a variety of techniques such as scaling, feature adding, and feature selection.Combining multiple estimators into a single pipeline.Evolving pipelines through stack generalization.Evaluating the performance of pipelines.Explanation of featuresSupport for custom metricsSupport for time-seriesSupport for PyTorch modelsHow to Use OrpheusAll steps can be controlled through a configuration file in YAML format, which is created when you first run the program with an instance of theComponentServiceorPipelineOrchestratorclass. You can edit this file to change the settings of all the preprocessing components. Detailed explanations of the component settings are provided within the configuration file itself.The preprocessing components are performed in the following order:Scalingcomponent: Identifies and applies the best scaler for the data.Feature Addingcomponent: Adds recommended features to the data.Feature Removingcomponent: Implements various algorithms to remove poorly performing or redundant features.HyperTunercomponent: Performs hyperparameter tuning through a three-round process, storing trained models and their performance. Each HyperTuner instance represents a single fold, acquired by the splits of an object which inherits fromBaseCrossValidatorclass in Scikit-Learn (eg .TimeSeriesSplit, KFold, ShuffleSplitetc.)In addition to the configuration file, you can control the enabled/disabled status of components using the parameters in theComponentService.initializemethod.MultiEstimatorPipelineTheMultiEstimatorPipelineclass is a scikit-learn pipeline with additional functionality, the main one being the ability to add multiple estimators and make combined predictions with them. Estimators in the pipeline can be accessed by theestimatorsattribute, which is a list where the estimators are indexed by their score. The better the score, the higher the index of the estimator in the list.The scores can be updated and can be used to determine the weights of the estimators when making predictions. This is done through thescoremethod. How estimators are weighted scorewise, can be checked by theget_weightsmethod.Pipelines can be saved to disk and loaded again using thesaveandloadmethods.Common ParametersMost classes, including the components, share a common set of parameters:metric/scoring: A callable that takes twopd.Seriesobjects and returns afloat. This is the metric that will be optimized during the pipeline execution. Examples includesklearn.metrics.mean_squared_errorandsklearn.metrics.accuracy_score. Also, custom metricfunctions can be used. In this case, they need to be registered through thePipelineOrchestrator.register_metricstatic method.config_path: Astrrepresenting the path to the configuration file of the components. This file specifies the hyperparameters and other settings for each component in the pipeline.maximize_scoring: Aboolindicating whether to maximize or minimize themetric/scoring. IfTrue, the pipeline will try to maximize the metric. IfFalse, the pipeline will try to minimize the metric.verbose: Anintrepresenting the verbosity level. The higher the value, the more information will be printed to the console during the pipeline execution. The possible values are:0: No information will be printed to the console.1Only warnings, errors and critical messages will be printed to the console.2: Only important informative messages and errors will be printed to the console.3:All messages, including errors, will be printed to the console.InPipelineOrchestrator, iflog_file_pathis set, logging to this file will be done instead of printing to the console.ServicesComponentServiceComponentServiceis the service class which binds all preprocessing-and training components together. It is responsible for all the preprocessing and training of the data. It also provides the ability to generate pipelines for the best base models and stacked models, found by the hyperparameter tuning process. These pipelines include the preprocessing steps and estimators. Before the scaling process, binary features as a default are excluded fromScalingandFeatureAddingcomponents. This is done to prevent the scaling and adding of features based on binary features, which is generally undesirable.In addition, the parametersordinal_featuresandcategorical_featurescan be used to specify ordinal and categorical features. These features also will be excluded from theScalingandFeatureAddingprocess.
Theordinal_featuresparameter takes a dict as value, where the key is the columnname and the value a list with values in the column from low to high. Thecategorical_featuresparameter takes a list with columnnames as its value.Theestimator_listparameter allows you to provide your own list of uninitialized estimators.
By default, this is set to None, and all scikit-learn estimators will be used, determined by the type of your data (classification or regression).
If you wish to use only your custom estimators and exclude the default scikit-learn estimators, setuse_sklearn_estimators_aside_estimator_listtoFalse.
Alternatively, estimators from other libraries with scikit-learn compatible interfaces can be added toestimator_list,
such asxgboostandlightgbm.Last named parameters are available in both theComponentServiceandPipelineOrchestratorclasses.Basic usage of the ComponentService class:importpandasaspdfromsklearn.model_selectionimportShuffleSplit,train_test_splitfromsklearn.datasetsimportmake_regressionfromorpheusimportComponentService,PipelineEvolverService,MultiEstimatorPipelineconfig_path="./configurations.yaml"# create a cross validation object. replace with your own cv objectcv_obj=ShuffleSplit(n_splits=3)# create a synthetic dataset. replace with your own dataX,y=make_regression(n_samples=1000,n_features=5,random_state=42,)X=pd.DataFrame(X)X.columns=[f"feature_{N}"forNinrange(1,X.shape[1]+1)]y=pd.Series(y)# split into train and testX_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.2,random_state=42,shuffle=True)if__name__=="__main__":# initialize the compomnentservice.# at first runtime, program will create a config file if it doesn't exist yet.# you can edit this file to change the settings of all the preprocessing components# before running the program again.component_service=ComponentService(X_train,X_test,y_train,y_test,config_path=config_path,cv_obj=cv_obj,n_jobs=-1,)# kick off the preprocessing and training process.# settings per component are read from the config file and applied# to the preprocessing and training process when running this method.component_service.initialize(scale=True,add_features=True,remove_features=True,)# generate fitted pipelines for best base models and stacked models,# found by the hyperparameter tuning process.# these include the preprocessing steps and estimators.pipe_base:MultiEstimatorPipeline=component_service.generate_pipeline_for_base_models(top_n_per_tuner=5)pipe_stacked:MultiEstimatorPipeline=component_service.generate_pipeline_for_stacked_models(top_n_per_tuner_range=[3,5])# evolve the pipelines through stack generalizationevolver=PipelineEvolverService(pipe_stacked)evolved_pipe_hv=evolver.evolve_voting(n_jobs=4,voting="hard")evolved_pipe_hv.fit(X_train,y_train)print(evolved_pipe_hv.score(X_test,y_test))evolved_pipe_sv=evolver.evolve_voting(n_jobs=4,voting="soft")evolved_pipe_sv.fit(X_train,y_train)print(evolved_pipe_sv.score(X_test,y_test))PipelineOrchestratorFor a simpler and more high-level user interface, you can utilize thePipelineOrchestratorclass.This class provides full and easy control over the entire signalflow, from the preprocessing components to model validation (eg.ComponentServiceis being used under the hood). It assumes a heuristic approach where the dataset is split into 3 partitions: The train, test and validationsets. This to ensure the quality of the models afterwards.The trainset will be assigned the folds by the Scikit-Learn cross-validation object and should generally be the largest dataset.The second dataset, in this context called the testset, will be used to evaluate the models from the earlier training process. During this process, 3 generations of models will be created. You can change this by setting thegenerationsparameter in thePipelineOrchestrator.build()method.The three generations are:Generation 1: Base:These are the top-performing base models discovered through the hyperparameter tuning process in the HyperTuner component.
Each instantiated HyperTuner object serves as a "tuner" and also represents a single cross-validation fold.
The number of models per tuner is determined by thetop_n_per_tunerparameter in the PipelineOrchestrator.build() method.Generation 2: Stacked:These meta-models are formed by combining the base models from generation 1 using various ensemble methods, such as voting, stacking, and averaging.Generation 3: Evolved:This is a single meta-model created by ensembling the models from generation 2.After utilizing thePipelineOrchestrator.build()method, models in the created pipelines can be validated by thePipelineOrchestrator.fortify()method. Here, stresstests will be executed on the models in all pipeline generations. Models which do not pass the stresstests, will be removed from their pipeline. For this process, the validationset will be used.Hierarchy diagramThis diagram provides a visual overview of how different components and services interact within the Orpheus framework:FlowchartHere is a concrete example what parts of the complete training process are automated by Orpheus:Basic usage of the PipelineOrchestrator class:importosimportpandasaspdfromsklearn.model_selectionimportShuffleSplit,train_test_splitfromsklearn.datasetsimportmake_regressionfromsklearn.metricsimportr2_scorefromorpheusimportPipelineOrchestratorconfig_path="./configurations.yaml"# create a cross-validation object. Replace with your own cv objectcv_obj=ShuffleSplit(n_splits=4)# create a synthetic dataset. Replace with your own dataX,y=make_regression(n_samples=1000,n_features=5,random_state=42,)X=pd.DataFrame(X)X.columns=[f"feature_{N}"forNinrange(1,X.shape[1]+1)]y=pd.Series(y)# split into train and testX_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.2,random_state=42,shuffle=True)if__name__=="__main__":orchestrator=PipelineOrchestrator(X_train,y_train,metric=r2_score,config_path=config_path,cv_obj=cv_obj,verbose=3,n_jobs=max(1,int(os.cpu_count()/2)),shuffle=True,test_size=0.1,validation_size=0.1,)(orchestrator.pre_optimize(max_splits=4).build(scale=False,add_features=False,remove_features=False,).fortify(optimize_n_jobs=True,threshold_score=0.90,plot_explaining=True,))# make predictionspred_base=orchestrator.pipelines["base"].predict(X_test)pred_stacked=orchestrator.pipelines["stacked"].predict(X_test)pred_evolved=orchestrator.pipelines["evolved"].predict(X_test)# get an overview of the feature importancesexplained_features=orchestrator.get_explained_features()# save the pipelines to disk for later useorchestrator.pipelines["base"].save("base_pipeline")orchestrator.pipelines["stacked"].save("stacked_pipeline")orchestrator.pipelines["evolved"].save("evolved_pipeline")Because of its simpler interface, general advice is to use the PipelineOrchestrator class for all actions, unless you have a specific reason not to, like for example, if you want more fine-grained control.Explanation of featuresFeatures can be explained through LIME (Local Interpretable Model-agnostic Explanations). Explanations are done on a per-sample basis.
This is done by thePipelineOrchestrator.fortify()method. Theplot_explainingparameter controls whether the explanations are plotted.
Setting theplot_explainingparameter toTruewill plot the explanations for the best base model, the best stacked model, and the evolved model.Custom metricsCustom metrics can be registered through thePipelineOrchestrator.register_metricstatic method. This method takes a callable as its only parameter. The callable should take twopd.Seriesobjects as its parameters and return afloat. The firstpd.Seriesobject represents the true values, while the secondpd.Seriesobject represents the predicted values.PipelineOrchestratorProxyMetadata about eachPipelineOrchestratorrun can be stored in a (sqlite) database using thePipelineOrchestratorProxyclass. This class takes anPipelineOrchestratoras argument in its constructor and adds the ability to store metadata about each run in a database.
This adds interesting new functionality, such as the ability to analyse the metadata from the database and find out which configurations of the components work best for a specific dataset.
The idea of this experimental class is to use a surrogate model to find the best configuration of the components for a specific dataset. This is done by training the surrogate model on the configurationdata from the database, where the scores are used as the target.
With the new proposed configurations, more and more iterations are produced, which deliver new data to train the surrogate model.
Using this technique, the idea is that the surrogate model should eventually converge to the best configuration for the dataset.TipsIf overfitting is a problem when using a classifier, consider adjusting the following settings in the YAML configurationfile for the HyperTuner component:TheR2_weightscan be adjusted to prioritize regularization. A starting point may be{"best_mean": 0.9, "lowest_stdev": 0.3, "amount_of_unique_vals": 0.3}. It is important to understand these weights are applied on a per-estimatorpopulation basis during the round 2 process. For instance, if theRandomForestClassifierestimatorpopulation (meaning ALL trained instances ofRandomForestClassifierduring round 2) had the highest meanaccuracyscore of 0.85 in round 2, compared to all other trained estimator-populations, and "bestmean" has the highest weight, there will be a significant chance that _RandomForestClassifierwill be the estimator advancing to round 3.penalty_to_score_if_overfitting: Increase the value to1.0to impose a heavy penalty on overfitting.If you encounter memory or performance issues due to a large dataset, consider utilizing therandom_subsetparameter in the YAML configurationfile. This parameter, available in theScaling,FeatureRemoving, andHyperTunercomponents, extracts a random subset of the data. Note that the indices may vary with each fitting iteration, the sole exception being theFeatureRemovingcomponent.If the program keeps on hanging, use thelog_cpu_memory_usageparameter in the constructor ofPipelineOrchestratorto keep track of memory and cpu-usage. If the hanging occurs inPipelineOrchestrator.build(), try thetimeout_durationparameter.