Package: mlimputer
MLimputer - Null Imputation Framework for Supervised Machine LearningFramework ContextualizationTheMLimputerproject constitutes an complete and integrated pipeline to automate the handling of missing values in datasets through regression prediction and aims at reducing bias and increase the precision of imputation results when compared to more classic imputation methods.
This package provides multiple algorithm options to impute your data, in which every observed data column with existing missing values is fitted with a robust preprocessing approach and subsequently predicted.The architecture design includes three main sections, these being: missing data analysis, data preprocessing and supervised model imputation which are organized in a customizable pipeline structure.This project aims at providing the following application capabilities:General applicability on tabular datasets: The developed imputation procedures are applicable on any data table associated with any Supervised ML scopes, based on missing data columns to be imputed.Robustness and improvement of predictive results: The application of the MLimputer preprocessing aims at improve the predictive performance through customization and optimization of existing missing values imputation in the dataset input columns.Main Development ToolsMajor frameworks used to built this project:PandasSklearnCatBoostWhere to get itBinary installer for the latest released version is available at the Python Package Index(PyPI).The source code is currently hosted on GitHub at:https://github.com/TsLu1s/MLimputerInstallationTo install this package from Pypi repository run the following command:pip install mlimputerUsage ExamplesThe first needed step after importing the package is to load a dataset (split it) and define your choosen imputation model.
The imputation model options for handling the missing data in your dataset are the following:RandomForestExtraTreesGBRKNNXGBoostLightgbmCatboostAfter creating aMLimputerobject with your imputation selected model, you can then fit the missing data through thefit_imputmethod. From there you can impute the future datasets withtransform_imput(validate, test ...) with the same data properties. Note, as it shows in the example bellow, you can also customize your model imputer parameters by changing it's configurations and then, implementing them in theimputer_configsparameter.Through thecross_validationfunction you can also compare the predictive performance evalution of multiple imputations, allowing you to validate which imputation model fits better your future predictions.Importante Notes:The actual version of this package does not incorporate the imputing of categorical values, just the automatic handling of numeric missing values is implemented.frommlimputer.imputationimportMLimputerimportmlimputer.model_selectionasmsimportmlimputer.parametersasparamsimportpandasaspdimportnumpyasnpfromsklearn.model_selectionimporttrain_test_splitimportwarningswarnings.filterwarnings("ignore",category=Warning)#-> For a clean consoledata=pd.read_csv('csv_directory_path')# Dataframe Loading Exampletrain,test=train_test_split(data,train_size=0.8)train,test=train.reset_index(drop=True),test.reset_index(drop=True)# <- Required# All model imputation options ->  "RandomForest","ExtraTrees","GBR","KNN","XGBoost","Lightgbm","Catboost"# Customizing Hyperparameters Examplehparameters=params.imputer_parameters()print(hparameters)hparameters["KNN"]["n_neighbors"]=5hparameters["RandomForest"]["n_estimators"]=30# Imputation Example 1 : KNNmli_knn=MLimputer(imput_model="KNN",imputer_configs=hparameters)mli_knn.fit_imput(X=train)train_knn=mli_knn.transform_imput(X=train)test_knn=mli_knn.transform_imput(X=test)# Imputation Example 2 : RandomForestmli_rf=MLimputer(imput_model="RandomForest",imputer_configs=hparameters)mli_rf.fit_imput(X=train)train_rf=mli_rf.transform_imput(X=train)test_rf=mli_rf.transform_imput(X=test)#(...)## Performance Evaluation Regression - Imputation CrossValidation Examplefromsklearn.linear_modelimportLinearRegressionfromsklearn.ensembleimportRandomForestRegressorfromcatboostimportCatBoostRegressorleaderboard_rf_imp=ms.cross_validation(X=train_rf,target="Target_Name_Col",test_size=0.2,n_splits=3,models=[LinearRegression(),RandomForestRegressor(),CatBoostRegressor()])## Export Imputation Metadata# Imputation Metadataimportpickleoutput=open("imputer_rf.pkl",'wb')pickle.dump(mli_rf,output)LicenseDistributed under the MIT License. SeeLICENSEfor more information.ContactLuis Santos -LinkedIn