Package: meta-learning-framework
Meta Learning FrameworkA framework to apply machine learning on how to combine models (learn to ensemble). It implements a machine learning classifier at theinstance level, choosing a subset of models to be employed, instead of using a simple bagging method (callednaive ensemblein this work).Table of contentsIntroductionDependenciesInstallationExamplesClassification - BinaryClassification - ScoreRegressionForecastingPerformanceIntroductionIt contains three main classes:MetaLearningModel,BaseModelandMetaClassifier. BaseModel and MetaClassifier are abstract classes to be used asparents. They contain attributes and methods that a base model (classifier or regressor) and the meta classifiermusthave.MetaLearningModelis the class that does all the work. The fit method will use a cross-validation (or a simple train_test_split validation, depends on the cv param) to create a training set to the meta classifier. In this process, it train every base model and predict a batch of instances, comparing the output with the true values.Depending on the task and mode (combineranderror_measurefuction as well), it will select the best base model(s) for each instances in the batch. It can be a multi-class or even a multi-label task. The meta model training set will be composed ofinstances(as they are in the original problem) andtargets(arrays of zeros and ones), indicating which base model(s) were selected or not for each instance (they have number_of_base_models length).In the prediction step, first theMetaLearningModelwill predict the instance choosing which base models are going to be used. Then,only those selected bse modelsare going to predict the given instance. Finally, their outputs is combined using thecombinerfuction.You can see more about this meta learning approach on this paper:https://link.springer.com/chapter/10.1007/978-3-030-61380-8_29.Dependenciesnumpysklearnpandastqdmsktime(optional - test code)dtw-python(optional - test code)tensorflow(optional - test code)InstallationSimply run:pipinstallmeta-learning-framework==0.0.1You can also clone the repository and run:cdmeta-learning-framework/
pipinstall.ExamplesThis section presents how to execute some test codes for you to better understand how this framework is supposed to work.Classification - Binary ModeThis example usessktimeframework for time series classification. Binary mode indicates that when creating the meta classifier training set, base models that correctly predict instances' class will be selected (soft selection). Notice that it can imply a multi-label classification task.Run the following commands:cdtests/
python3tsc_classification_example.pysktime_dataset_namebinaryClassification - Score ModeThis example usessktimeframework for time series classification as well. The difference to the previous one is the mode. Now, the base model that outputs the best score distribution will be choosen. It implies only a multi-class classification task.Run the following commands:cdtests/
python3tsc_classification_example.pysktime_dataset_namescoreRegression (Score mode only)This example uses sklearn's regressiondatasets. There is a Random Forest classifier which learns to choose the base regressor that will output a prediction with the smallest error possible between all regressors (multi-class task).Run the following commands:cdtests/
python3sklearn_regression_example.pyForecastingThis example is the most complex. It is a regression task, but since is a forecasting,you can not use cross-validation directly. So it uses a generator of train/test split indexes, regarding instances order in time. Also, it is a multi output regression task, since it tries to predict two future values.Run the following commands:cdtests/
python3forecasting_example.pyPerformanceClassification - Both binary and scoreRunning the scriptsktime_classification_time.pythese are the accuracy for a couple of datasets:Sktime DatasetNaive Ensemble - AccuracyBinary - AccuracyScore - AccuracyBinary - models usedHam.63.62.560.66PowerCons.95.99.812.00MoteStrain.94.94.810.00MedicalImages.74.75.570.00DistalPhalanxTW.67.65.580.32InsectWingbeatSound.61.610.00Simple regression taskRunning the scriptsklearn_regression_example.pyusingfetch_california_housingdataset the following results were achieved:ModelMAER2Meta Model0.46780.6927Naive Ensemble0.66810.4622Individual - Linear0.54140.6047Individual - SVR0.8743-0.0284Individual - 3NN0.83840.1015Individual - AdaBoost0.80830.3669Forecasting taskRunning the scriptforecasting_example.pyusing a white noise time series:ModelMean Absolute Sum ErrorMean Absolute Weighted ErrorMeta Model0.25540.1527Naive Ensemble0.30230.1720Individual - Linear0.25160.1510Individual - 3NN0.41760.2228Individual - Random Forest0.26980.1589