Package: autoimpute
AutoimputeAutoimputeis a Python package for analysis and implementation ofImputation Methods!View our websiteto exploreAutoimputein more detail. New tutorials coming soon!Check out our docsto get the developer guide toAutoimpute.Conference TalksWe presentedAutoimputeat a couple of PyData conferences!PyData NYC: New and Upcoming slot in November 2019PyData LA: Main talk slot in December 2019. See the videohere!Notes on Developmentv 0.13+ has upgraded to support pymc, the next generation of the pymc3 libraryv 0.12+ contains the newMiceImputer. Thanks to@gjdvfor the help on this issue!We're looking to collaborate and happy to work with those interested!If you'd like to get involved, feel free to reach out! Our info is on theAuthorspage.Creators and MaintainersJoseph Kearney –@kearnzShahid Barkat -@shabarkaArnab Bose (Advisor) -@bosearnabSee theAuthorspage to get in touch!InstallationAutoimputeisregistered with PyPI!Download withpip install autoimpute.Ifpipcached an older version, trypip install --no-cache-dir --upgrade autoimpute.If you want to work with the development branch, use the script below:Developmentgitclone-bdev--single-branchhttps://github.com/kearnz/autoimpute.gitcdautoimpute
pythonsetup.pyinstallMotivationMost machine learning algorithms expect clean and complete datasets, but real-world data is messy and missing. Unfortunately, handling missing data is quite complex, so programming languages generally punt this responsibility to the end user. By default, R drops all records with missing data - a method that is easy to implement but often problematic in practice. For richer imputation strategies, R has multiple packages to deal with missing data (MICE,Amelia,TSImpute, etc.). Python users are not as fortunate. Python'sscikit-learnthrows a runtime error when an end user deploys models on datasets with missing records, and few third-party packages exist to handle imputation end-to-end.Therefore, this package aids the Python user by providing more clarity to the imputation process, making imputation methods more accessible, and measuring the impact imputation methods have in supervised regression and classification. In doing so, this package brings missing data imputation methods to the Python world and makes them work nicely in Python machine learning projects (and specifically ones that utilizescikit-learn). Lastly, this package provides its own implementation of supervised machine learning methods that extend bothscikit-learnandstatsmodelsto mutiply imputed datasets.Main FeaturesUtility functions to examine patterns in missing data and decide on relevant features for imputationMissingness classifier and automatic missing data test set generatorNative handling for categorical variables (as predictors and targets of imputation)Single and multiple imputation classes forpandasDataFramesCustom visualization support for utility functions and imputation methodsAnalysis methods and pooled parameter inference using multiply imputed datasetsNumerous imputation methods, as specified in the table below:Imputation Methods SupportedUnivariateMultivariateTime Series / InterpolationMeanLinear RegressionLinearMedianBinomial Logistic RegressionQuadraticModeMultinomial Logistic RegressionCubicRandomStochastic RegressionPolynomialNormBayesian Linear RegressionSplineCategoricalBayesian Binary Logistic RegressionTime-weightedPredictive Mean MatchingNext Obs Carried BackwardLocal Residual DrawsLast Obs Carried ForwardTodoAdditional cross-sectional methods, including random forest, KNN, EM, and maximum likelihoodAdditional time-series methods, including EWMA, ARIMA, Kalman filters, and state-space modelsExtended support for visualization of missing data patterns, imputation methods, and analysis modelsAdditional support for analysis metrics and analyis models after multiple imputationMultiprocessing and GPU support for larger datasets, as well as integration withdaskDataFramesExample UsageAutoimpute is designed to be user friendly and flexible. When performing imputation, Autoimpute fits directly intoscikit-learnmachine learning projects. Imputers inherit from sklearn'sBaseEstimatorandTransformerMixinand implementfitandtransformmethods, making them valid Transformers in an sklearn pipeline.Right now, there are threeImputerclasses we'll work with:fromautoimpute.imputationsimportSingleImputer,MultipleImputer,MiceImputersi=SingleImputer()# pass through data oncemi=MultipleImputer()# pass through data multiple timesmice=MiceImputer()# pass through data multiple times and iteratively optimize imputations in each columnWhich to use, and When?There are tradeoffs between the three imputersWe won't get into the specifics regarding why, but here are a couple points to keep in mind:Execution time (best to worst):SingleImputer,MultipleImputer,MiceImputerImputation quality (best to worst):MiceImputer,MultipleImputer,SingleImputerThis shouldn't come as a surprise. TheMiceImputerdoes the most work, while theSingleImputerdoes the leastThe example below use theMiceImputer, but you can swap in theMultipleImputerorSingleImputeras wellImputations can be as simple as:# simple example using default instance of MiceImputerimp=MiceImputer()# fit transform returns a generator by default, calculating each imputation method lazilyimp.fit_transform(data)Or quite complex, such as:# create a complex instance of the MiceImputer# Here, we specify strategies by column and predictors for each column# We also specify what additional arguments any `pmm` strategies should takeimp=MiceImputer(n=10,strategy={"salary":"pmm","gender":"bayesian binary logistic","age":"norm"},predictors={"salary":"all","gender":["salary","education","weight"]},imp_kwgs={"pmm":{"fill_value":"random"}},visit="left-to-right",return_list=True)# Because we set return_list=True, imputations are done all at once, not evaluated lazily.# This will return M*N, where M is the number of imputations and N is the size of original dataframe.imp.fit_transform(data)Autoimpute also extends supervised machine learning methods fromscikit-learnandstatsmodelsto apply them to multiply imputed datasets (using theMiceImputerunder the hood). Right now, Autoimpute supports linear regression and binary logistic regression. Additional supervised methods are currently under development.As with Imputers, Autoimpute's analysis methods can be simple or complex:fromautoimpute.analysisimportMiLinearRegression# By default, use statsmodels OLS and MiceImputer()simple_lm=MiLinearRegression()# fit the model on each multiply imputed dataset and pool parameterssimple_lm.fit(X_train,y_train)# get summary of fit, which includes pooled parameters under Rubin's rules# also provides diagnostics related to analysis after multiple imputationsimple_lm.summary()# make predictions on a new dataset using pooled parameterspredictions=simple_lm.predict(X_test)# Control both the regression used and the MiceImputer itselfmice_imputer_arguments=dict(n=3,strategy={"salary":"pmm","gender":"bayesian binary logistic","age":"norm"},predictors={"salary":"all","gender":["salary","education","weight"]},imp_kwgs={"pmm":{"fill_value":"random"}},visit="left-to-right")complex_lm=MiLinearRegression(model_lib="sklearn",# use sklearn linear regressionmi_kwgs=mice_imputer_arguments# control the multiple imputer)# fit the model on each multiply imputed datasetcomplex_lm.fit(X_train,y_train)# get summary of fit, which includes pooled parameters under Rubin's rules# also provides diagnostics related to analysis after multiple imputationcomplex_lm.summary()# make predictions on new dataset using pooled parameterspredictions=complex_lm.predict(X_test)Note that we can also pass a pre-specifiedMiceImputer(orMultipleIputer) to either analysis model instead of usingmi_kwgs. The option is ours, and it's a matter of preference. If we pass a pre-specifiedMiceImputer, anything inmi_kwgsis ignored, although themi_kwgsargument is still validated.fromautoimpute.imputationsimportMiceImputerfromautoimpute.analysisimportMiLinearRegression# create a multiple imputer firstcustom_imputer=MiceImputer(n=3,strategy="pmm",return_list=True)# pass the imputer to a linear regression modelcomplex_lm=MiLinearRegression(mi=custom_imputer,model_lib="statsmodels")# proceed the same as the previous examplescomplex_lm.fit(X_train,y_train).predict(X_test)complex_lm.summary()For a deeper understanding of how the package works and its available features, see ourtutorials website.Versions and DependenciesPython 3.8+Dependencies:numpyscipypandasstatsmodelsscikit-learnxgboostpymcseabornmissingnoSeerequirements.txtfor versioning infoA note for Windows Users:Autoimpute has NOT been tested on Windows using the upgrade to pymc. Historically, autoimpute has had trouble on Windows, as demonstrated by bullets below.Autoimpute works on Windows but users may have trouble with pymc for bayesian methods.(See discourse)Users may receive a runtime error‘can’t pickle fortran objects’when sampling using multiple chains.There are a couple of things to do to try to overcome this error:Reinstall theano and pymc. Make sure to delete .theano cache in your home folder.Upgrade joblib in the process, which is reponsible for generating the error (pymc uses joblib under the hood).Setcores=1inpm.sample. This should be a last resort, as it means posterior sampling will use 1 core only. Not using multiprocessing will slow down bayesian imputation methods significantly.Reach out and let us know if you've worked through this issue successfully on Windows and have a better solution!LicenseDistributed under the MIT license. SeeLICENSEfor more information.ContributingGuidelines for contributing to our project. SeeCONTRIBUTINGfor more information.Contributor Code of ConductAdapted from Contributor Covenant, version 1.0.0. SeeCode of Conductfor more information.