Package: combo
Deployment & Documentation & StatsBuild Status & Coverage & Maintainability & Licensecombois a comprehensive Python toolbox forcombining machine learning (ML) models and scores.Model combinationcan be considered as a subtask ofensemble learning,
and has been widely used in real-world tasks and data science competitions like Kaggle[3].combohas been used/introduced in various research works since its inception[7][11].combolibrary supports the combination of models and score from
key ML libraries such asscikit-learn,xgboost, andLightGBM,
for crucial tasks including classification, clustering, anomaly detection.
See figure below for some representative combination approaches.combois featured for:Unified APIs, detailed documentation, and interactive examplesacross various algorithms.Advanced and latest models, such as Stacking/DCS/DES/EAC/LSCP.Comprehensive coveragefor classification, clustering, anomaly detection, and raw score.Optimized performance with JIT and parallelizationwhen possible, usingnumbaandjoblib.API Demo:fromcombo.models.classifier_stackingimportStacking# initialize a group of base classifiersclassifiers=[DecisionTreeClassifier(),LogisticRegression(),KNeighborsClassifier(),RandomForestClassifier(),GradientBoostingClassifier()]clf=Stacking(base_estimators=classifiers)# initialize a Stacking modelclf.fit(X_train,y_train)# fit the model# predict on unseen datay_test_labels=clf.predict(X_test)# label predictiony_test_proba=clf.predict_proba(X_test)# probability predictionCiting combo:combo paperis published inAAAI 2020(demo track).
If you use combo in a scientific publication, we would appreciate citations to the following paper:@inproceedings{zhao2020combo,
  title={Combining Machine Learning Models and Scores using combo library},
  author={Zhao, Yue and Wang, Xuejian and Cheng, Cheng and Ding, Xueying},
  booktitle={Thirty-Fourth AAAI Conference on Artificial Intelligence},
  month = {Feb},
  year={2020},
  address = {New York, USA}
}or:Zhao, Y., Wang, X., Cheng, C. and Ding, X., 2020. Combining Machine Learning Models and Scores using combo library. Thirty-Fourth AAAI Conference on Artificial Intelligence.Key Links and Resources:awesome-ensemble-learning(ensemble learning related books, papers, and more)View the latest codes on GithubView the documentation & APIView all examplesView the demo video for AAAI 2020Execute Interactive Jupyter NotebooksTable of Contents:InstallationAPI Cheatsheet & ReferenceImplemented AlgorithmsExample 1: Classifier Combination with Stacking/DCS/DESExample 2: Simple Classifier CombinationExample 3: Clustering CombinationExample 4: Outlier Detector CombinationDevelopment StatusInclusion CriteriaInstallationIt is recommended to usepipfor installation. Please make surethe latest versionis installed, as combo is updated frequently:pipinstallcombo# normal installpipinstall--upgradecombo# or update if neededpipinstall--precombo# or include pre-release version for new featuresAlternatively, you could clone and run setup.py file:gitclonehttps://github.com/yzhao062/combo.gitcdcombopipinstall.Required Dependencies:Python 3.5, 3.6, or 3.7joblibmatplotlib (optional for running examples)numpy>=1.13numba>=0.35pyodscipy>=0.19.1scikit_learn>=0.20Note on Python 2:
The maintenance of Python 2.7 will be stopped by January 1, 2020 (seeofficial announcement).
To be consistent with the Python change and combo’s dependent libraries, e.g., scikit-learn,combo only supports Python 3.5+and we encourage you to use
Python 3.5 or newer for the latest functions and bug fixes. More information can
be found atMoving to require Python 3.API Cheatsheet & ReferenceFull API Reference: (https://pycombo.readthedocs.io/en/latest/api.html).
The following APIs are consistent for most of the models
(API Cheatsheet:https://pycombo.readthedocs.io/en/latest/api_cc.html).fit(X, y): Fit estimator. y is optional for unsupervised methods.predict(X): Predict on a particular sample once the estimator is fitted.predict_proba(X): Predict the probability of a sample belonging to each class once the estimator is fitted.fit_predict(X, y): Fit estimator and predict on X. y is optional for unsupervised methods.For raw score combination (after the score matrix is generated),
use individual methods from“score_comb.py”directly.
Raw score combination API: (https://pycombo.readthedocs.io/en/latest/api.html#score-combination).Implemented Algorithmscombogroups combination frameworks by tasks. General purpose methods are
fundamental ones which can be applied to various tasks.TaskAlgorithmYearRefGeneral PurposeAverage & Weighted Average: average across all scores/prediction results, maybe with weightsN/A[13]General PurposeMaximization: simple combination by taking the maximum scoresN/A[13]General PurposeMedian: take the median value across all scores/prediction resultsN/A[13]General PurposeMajority Vote & Weighted Majority VoteN/A[13]ClassificationSimpleClassifierAggregator: combining classifiers by general purpose methods aboveN/AN/AClassificationDCS: Dynamic Classifier Selection (Combination of multiple classifiers using local accuracy estimates)1997[8]ClassificationDES: Dynamic Ensemble Selection (From dynamic classifier selection to dynamic ensemble selection)2008[5]ClassificationStacking (meta ensembling): use a meta learner to learn the base classifier resultsN/A[4]ClusteringClusterer Ensemble: combine the results of multiple clustering results by relabeling2006[12]ClusteringCombining multiple clusterings using evidence accumulation (EAC)2002[6]Anomaly DetectionSimpleDetectorCombination: combining outlier detectors by general purpose methods aboveN/A[2]Anomaly DetectionAverage of Maximum (AOM): divide base detectors into subgroups to take the maximum, and then average2015[1]Anomaly DetectionMaximum of Average (MOA): divide base detectors into subgroups to take the average, and then maximize2015[1]Anomaly DetectionXGBOD: a semi-supervised combination framework for outlier detection2018[9]Anomaly DetectionLocally Selective Combination (LSCP)2019[10]The comparison among selected implemented modelsis made available below
(Figure,compare_selected_classifiers.py,Interactive Jupyter Notebooks).
For Jupyter Notebooks, please navigate to“/notebooks/compare_selected_classifiers.ipynb”.All implemented modesare associated with examples, check“combo examples”for more information.Example of Stacking/DCS/DES“examples/classifier_stacking_example.py”demonstrates the basic API of stacking (meta ensembling).“examples/classifier_dcs_la_example.py”demonstrates the basic API of Dynamic Classifier Selection by Local Accuracy.“examples/classifier_des_la_example.py”demonstrates the basic API of Dynamic Ensemble Selection by Local Accuracy.It is notedthe basic API is consistent across all these models.Initialize a group of classifiers as base estimators# initialize a group of classifiersclassifiers=[DecisionTreeClassifier(random_state=random_state),LogisticRegression(random_state=random_state),KNeighborsClassifier(),RandomForestClassifier(random_state=random_state),GradientBoostingClassifier(random_state=random_state)]Initialize, fit, predict, and evaluate with Stackingfromcombo.models.classifier_stackingimportStackingclf=Stacking(base_estimators=classifiers,n_folds=4,shuffle_data=False,keep_original=True,use_proba=False,random_state=random_state)clf.fit(X_train,y_train)y_test_predict=clf.predict(X_test)evaluate_print('Stacking | ',y_test,y_test_predict)See a sample output of classifier_stacking_example.pyDecisionTree|Accuracy:0.9386,ROC:0.9383,F1:0.9521LogisticRegression|Accuracy:0.9649,ROC:0.9615,F1:0.973KNeighbors|Accuracy:0.9561,ROC:0.9519,F1:0.9662GradientBoosting|Accuracy:0.9605,ROC:0.9524,F1:0.9699RandomForest|Accuracy:0.9605,ROC:0.961,F1:0.9693Stacking|Accuracy:0.9868,ROC:0.9841,F1:0.9899Example of Classifier Combination“examples/classifier_comb_example.py”demonstrates the basic API of predicting with multiple classifiers.It is noted that the API across all other algorithms are consistent/similar.Initialize a group of classifiers as base estimators# initialize a group of classifiersclassifiers=[DecisionTreeClassifier(random_state=random_state),LogisticRegression(random_state=random_state),KNeighborsClassifier(),RandomForestClassifier(random_state=random_state),GradientBoostingClassifier(random_state=random_state)]Initialize, fit, predict, and evaluate with a simple aggregator (average)fromcombo.models.classifier_combimportSimpleClassifierAggregatorclf=SimpleClassifierAggregator(classifiers,method='average')clf.fit(X_train,y_train)y_test_predicted=clf.predict(X_test)evaluate_print('Combination by avg   |',y_test,y_test_predicted)See a sample output of classifier_comb_example.pyDecisionTree|Accuracy:0.9386,ROC:0.9383,F1:0.9521LogisticRegression|Accuracy:0.9649,ROC:0.9615,F1:0.973KNeighbors|Accuracy:0.9561,ROC:0.9519,F1:0.9662GradientBoosting|Accuracy:0.9605,ROC:0.9524,F1:0.9699RandomForest|Accuracy:0.9605,ROC:0.961,F1:0.9693Combinationbyavg|Accuracy:0.9693,ROC:0.9677,F1:0.9763Combinationbyw_avg|Accuracy:0.9781,ROC:0.9716,F1:0.9833Combinationbymax|Accuracy:0.9518,ROC:0.9312,F1:0.9642Combinationbyw_vote|Accuracy:0.9649,ROC:0.9644,F1:0.9728Combinationbymedian|Accuracy:0.9693,ROC:0.9677,F1:0.9763Example of Clustering Combination“examples/cluster_comb_example.py”demonstrates the basic API of combining multiple base clustering estimators.“examples/cluster_eac_example.py”demonstrates the basic API of Combining multiple clusterings using evidence accumulation (EAC).Initialize a group of clustering methods as base estimators# Initialize a set of estimatorsestimators=[KMeans(n_clusters=n_clusters),MiniBatchKMeans(n_clusters=n_clusters),AgglomerativeClustering(n_clusters=n_clusters)]Initialize a Clusterer Ensemble class and fit the modelfromcombo.models.cluster_combimportClustererEnsemble# combine by Clusterer Ensembleclf=ClustererEnsemble(estimators,n_clusters=n_clusters)clf.fit(X)Get the aligned results# generate the labels on Xaligned_labels=clf.aligned_labels_predicted_labels=clf.labels_Example of Outlier Detector Combination“examples/detector_comb_example.py”demonstrates the basic API of combining multiple base outlier detectors.Initialize a group of outlier detection methods as base estimators# Initialize a set of estimatorsdetectors=[KNN(),LOF(),OCSVM()]Initialize a simple averaging aggregator, fit the model, and make
the prediction.fromcombo.models.detectorcombinationimportSimpleDetectorAggregatorclf=SimpleDetectorAggregator(base_estimators=detectors)clf_name='Aggregation by Averaging'clf.fit(X_train)y_train_pred=clf.labels_# binary labels (0: inliers, 1: outliers)y_train_scores=clf.decision_scores_# raw outlier scores# get the prediction on the test datay_test_pred=clf.predict(X_test)# outlier labels (0 or 1)y_test_scores=clf.decision_function(X_test)# outlier scoresEvaluate the prediction using ROC and Precision @ Rank n.# evaluate and print the resultsprint("\nOn Training Data:")evaluate_print(clf_name,y_train,y_train_scores)print("\nOn Test Data:")evaluate_print(clf_name,y_test,y_test_scores)See sample outputs on both training and test data.OnTrainingData:AggregationbyAveragingROC:0.9994,precision@rankn:0.95OnTestData:AggregationbyAveragingROC:1.0,precision@rankn:1.0Development Statuscombois currentlyunder developmentas of Feb, 2020. A concrete plan has
been laid out and will be implemented in the next few months.Similar to other libraries built by us, e.g., Python Outlier Detection Toolbox
(pyod),combois also targeted to be published inJournal of Machine Learning Research (JMLR),open-source software track. A demo paper has been presented inAAAI 2020for progress update.Watch & Starto get the latest update! Also feel free to send me an email (zhaoy@cmu.edu)
for suggestions and ideas.Inclusion CriteriaSimilarly to scikit-learn, We mainly consider well-established algorithms for inclusion.
A rule of thumb is at least two years since publication, 50+ citations, and usefulness.However, we encourage the author(s) of newly proposed models to share and add your implementation into combo
for boosting ML accessibility and reproducibility.
This exception only applies if you could commit to the maintenance of your model for at least two year period.Reference[1](1,2)Aggarwal, C.C. and Sathe, S., 2015. Theoretical foundations and algorithms for outlier ensembles.ACM SIGKDD Explorations Newsletter, 17(1), pp.24-47.[2]Aggarwal, C.C. and Sathe, S., 2017. Outlier ensembles: An introduction. Springer.[3]Bell, R.M. and Koren, Y., 2007. Lessons from the Netflix prize challenge.SIGKDD Explorations, 9(2), pp.75-79.[4]Gorman, B. (2016). A Kaggler’s Guide to Model Stacking in Practice. [online] The Official Blog of Kaggle.com. Available at:http://blog.kaggle.com/2016/12/27/a-kagglers-guide-to-model-stacking-in-practice[Accessed 26 Jul. 2019].[5]Ko, A.H., Sabourin, R. and Britto Jr, A.S., 2008. From dynamic classifier selection to dynamic ensemble selection.Pattern recognition, 41(5), pp.1718-1731.[6]Fred, A. L. N., & Jain, A. K. (2005). Combining multiple clusterings using evidence accumulation.IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(6), 835–850.https://doi.org/10.1109/TPAMI.2005.113[7]Raschka, S., Patterson, J. and Nolet, C., 2020. Machine Learning in Python: Main developments and technology trends in data science, machine learning, and artificial intelligence. arXiv preprint arXiv:2002.04803.[8]Woods, K., Kegelmeyer, W.P. and Bowyer, K., 1997. Combination of multiple classifiers using local accuracy estimates.IEEE transactions on pattern analysis and machine intelligence, 19(4), pp.405-410.[9]Zhao, Y. and Hryniewicki, M.K. XGBOD: Improving Supervised Outlier Detection with Unsupervised Representation Learning.IEEE International Joint Conference on Neural Networks, 2018.[10]Zhao, Y., Nasrullah, Z., Hryniewicki, M.K. and Li, Z., 2019, May. LSCP: Locally selective combination in parallel outlier ensembles. InProceedings of the 2019 SIAM International Conference on Data Mining (SDM), pp. 585-593. Society for Industrial and Applied Mathematics.[11]Zhao, Y., Nasrullah, Z. and Li, Z., 2019. PyOD: A Python Toolbox for Scalable Outlier Detection.Journal of Machine Learning Research, 20, pp.1-7.[12]Zhou, Z.H. and Tang, W., 2006. Clusterer ensemble.Knowledge-Based Systems, 19(1), pp.77-83.[13](1,2,3,4)Zhou, Z.H., 2012. Ensemble methods: foundations and algorithms. Chapman and Hall/CRC.