Package: dptools
dptools: data preprocessing functions for PythonOverviewThedptoolsPython package provides helper functions to simplify common data processing tasks in a data science pipeline, including feature engineering, data aggregation, working with missing values and more.The package currently encompasses the following functions:Feature engineering:add_date_features(): create date and time-based featuresadd_text_features(): create text-based features (including counts and TF-IDF)aggregate_data(): aggregate data and create features based on aggregated statisticsencode_factors(): perform label or dummy encoding of categorical featuresData processing:split_nested_features(): split features nested in a single columnfill_missings(): replace missings with specific valuescorrect_colnames(): correct column names to be unique and remove foreign symbolsprint_missings(): print information on features with missing valuesprint_factor_levels(): print levels of categorical featuresData cleaning:find_correlated_features(): identify features with a high pairwise correlationfind_constant_features(): identify features with a single unique valueImport and versioning:read_csv_with_json(): read CSV where some columns are in JSON formatsave_csv_version(): save CSV with an automatically assigned version to prevent overwritingInstallationThe latest stable release ofdptoolscan be installed from PyPI:pip install dptoolsYou may also install the development version from Github:pip install git+https://github.com/kozodoi/dptools.gitAfter the installation, you can import the included functions:fromdptoolsimport*ExamplesThis section contains a few examples of using functions fromdptoolsfor different data preprocessing tasks. Please refer to the docstring documentation in the implemented functions for further examples.Creating a toy data setFirst, let us create a toy data frame to demonstrate the package functionality.# import dependenciesimportpandasaspdimportnumpyasnp# create data framedata={'age':[27,np.nan,30,25,np.nan],'height':[170,168,173,177,165],'gender':['female','male',np.nan,'male','female'],'income':['high','medium','low','low','no income']}df=pd.DataFrame(data)ageheightgenderincome27.0170femalehighNaN168malemedium30.0173NaNlow25.0177malelowNaN165femaleno incomeAggregating features# aggregating the datafromdptoolsimportaggregate_datadf_new=aggregate_data(df,group_var='gender',num_stats=['mean','max'],fac_stats='mode')genderage_meanage_maxheight_meanheight_maxincome_modefemale27.027.0167.5170'high'male25.025.0172.5177'low'Creating text-based features# creating text-based featuresfromdptoolsimportadd_text_featuresdf_new=add_text_features(df,text_vars='income')ageheightgenderincome_word_countincome_char_countincome_tfidf_0...income_tfidf_327.0170female141.0...0.0NaN168male160.0...1.030.0173NaN130.0...0.025.0177male130.0...0.0NaN165female290.0...0.0Working with missings# print statistics on missing valuesfromdptoolsimportprint_missingsprint_missings(df)TotalPercentage20.4gender10.2Finding correlated features# displays one correlated feature from each pairfromdptoolsimportfind_correlated_featuresfeats=find_correlated_features(df,cutoff=0.4,method='spearman')featsFound 1 correlated features.['age']Data versioning# first call saves df as 'data_v1.csv'fromdptoolsimportsave_csv_versionsave_csv_version('data.csv',df,index=False)# second call saves df as 'data_v2.csv' as data_v1.csv already existssave_csv_version('data.csv',df,index=False)DependenciesInstallation requires Python 3.7+ and the following packages:numpypandassklearnscipyFeedbackIn case you need help on the included data preprocessing functions or you want to report an issue, please do so at the correspondingGitHub page.