Package: datasurveyor
DatasurveyorAuthor:Nick BukerIntroduction:Datasurveyor is a small collection of tools for exploratory data analysis. It leverages Pandas, but the tools are able to ingest either DataFrames or Series. The output is a tidy DataFrame for easy viewing of results. Currently, datasurveyor focuses on rapidly identifying data quality issues, but the scope will likely expand as the package becomes "battle tested".Table of contents:Installing datasurveyor:Datasurveyor installation instructionsUsing datasurveyor:Datasurveyor use instructionsBinary featuresImporting BinaryFeaturesChecking if all values the sameChecking if values are mostly the sameChecking the rangeCategorical featuresImporting CategoricalFeaturesChecking if values are mostly the sameChecking number of categoriesGeneral featuresImporting GeneralFeaturesChecking for nullsChecking for fuzzy nullsUnique featuresImporting UniqueFeaturesChecking uniquenessContributing and Testing:Contributing to datasurveyorTesting datasurveyorInstalling datasurveyor:Datasurveyor can be install via pip. As always, use of a project-level virtual environment is recommended.Note: Datasurveyor requires Python >= 3.6.$pipinstalldatasurveyorUsing DatasurveyorTo demonstrate the tools available in datasurveyor, let's use a Pandas DataFrame nameddf.idnamestateplatformapp_instlyltyspend01NickWAiosTrue0012GinaORandroidTrue1nan23RobWAiosFalse01034AdamIDwebTrue115045HannaWAiosTrue11256SusanNullandroidFalse0067QuentinWAiosTrue1nan78CaitlynunknownwebTrue0889MattWAwebTrue150910NickWAiosTrue0-10A data dictionary fordfis below.columndtypedescriptionidint64unique customer identifiernameobjectcustomer namestateobjectstate of residenceplatformobjectsystem platformapp_instboolapp installation flaglyltyint64loyalty program flagspendfloat64total customer spendBinary featuresDescriptionThe methods withinBinaryFeaturesare intended for use with binary data (data with two possible values). Datasurveyor expects binary features to be stored as bools or integers (with values of 0 or 1). In the example data,app_instandlyltyare binary features.Importing BinaryFeaturesThe binary feature tools can be imported with the command below.fromdatasurveyorimportBinaryFeaturesasBFChecking if all values the sameThecheck_all_samemethod can be used to check if binary features contain exclusively the same value. This method can be applied to a single binary feature or a collection of binary features.BF.check_all_same(df['app_inst'])all_same0FalseBF.check_all_same(df[['app_inst','lylty']])columnall_same0app_instFalse1lyltyFalseChecking if values are mostly the sameThecheck_mostly_samemethod can be used to check if binary features contain mostly the same value (default threshold 95%). This method can be applied to a single binary feature or a collection of binary features.BF.check_mostly_same(df['app_inst'])mostly_samethreshmean0False0.950.8BF.check_mostly_same(df[['app_inst','lylty']])columnmostly_samethreshmean0app_instFalse0.950.81lyltyFalse0.950.5The user can specify whatever threshold is appropriate for their usecase. Ifthresh=0.7is applied, the method will flag features with at least 70% the same value.BF.check_mostly_same(df['app_inst'],thresh=0.7)mostly_samethreshmean0True0.70.8BF.check_mostly_same(df[['app_inst','lylty']],thresh=0.7)columnmostly_samethreshmean0app_instTrue0.70.81lyltyFalse0.70.5Checking the rangeThecheck_outside_rangemethod can be used to detect features with data outside the expected range of 0 and 1. Note that the outside of range condition is only possible for binary features encoded as integer data type.BF.check_outside_range(df['app_inst'])outside_range0FalseBF.check_outside_range(df[['app_inst','lylty']])columnoutside_range0app_instFalse1lyltyFalseCategorical featuresDescriptionThe methods withinCategoricalFeaturesare intended for use with categorical data (data denoting categories). Datasurveyor expects categorical features to be stored as object (string) or integer type. In the example data,stateandplatformare categorical features.Importing CategoricalFeaturesThe categorical feature tools can be imported with the command below.fromdatasurveyorimportCategoricalFeaturesasCFChecking if values are mostly the sameThecheck_mostly_samemethod can be used to check if categorical features contain mostly the same value (default threshold 95%). This method can be applied to a single categorical feature or a collection of categorical features.CF.check_mostly_same(df['state'])mostly_samethreshmost_commoncountprop0False0.95WA60.6CF.check_mostly_same(df[['state','platform']])columnmostly_samethreshmost_commoncountprop0stateFalse0.95WA60.61platformFalse0.95ios50.5The user can specify whatever threshold is appropriate for their usecase. Ifthresh=0.6is applied, the method will flag features with at least 60% the same value.CF.check_mostly_same(df['state'],thresh=0.6)mostly_samethreshmost_commoncountprop0True0.6WA60.6CF.check_mostly_same(df[['state','platform']],thresh=0.6)columnmostly_samethreshmost_commoncountprop0stateTrue0.6WA60.61platformFalse0.6ios50.5Checking number of categoriesThen_categoriesmethod can be used to count the number of categories. This method can be applied to a single categorical feature or a collection of categorical features.CF.check_n_categories(df['state'])n_categories04CF.check_n_categories(df[['state','platform']])columnn_categories0state41platform3General featuresDescriptionThe methods withinGeneralFeaturesare intended for use with any data. Datasurveyor expects inputs to be of type Pandas Series or DataFrame, but has no type expectations for the data within those structures.Importing GeneralFeaturesThe general feature tools can be imported with the command below.fromdatasurveyorimportGeneralFeaturesasGFChecking for nullsThecheck_nullsmethod can be used to check for nulls. This method can be applied to a single feature or a collection of features.GF.check_nulls(df['spend'])nulls_presentnull_countprop_null0True20.2GF.check_nulls(df)columnnulls_presentnull_countprop_null0idFalse001nameFalse002stateFalse003platformFalse004app_instFalse005lyltyFalse006spendTrue20.2Checking for nullsThecheck_fuzzy_nullsmethod can be used to check for values that commonly denote nulls. This method can be applied to a single feature or a collection of features.GF.check_fuzzy_nulls(df['state'])fuzzy_nulls_presentfuzzy_null_countprop_fuzzy_null0True10.1GF.check_fuzzy_nulls(df)columnfuzzy_nulls_presentfuzzy_null_countprop_fuzzy_null0idFalse001nameFalse002stateTrue10.13platformFalse004app_instFalse005lyltyFalse006spendFalse00The defaults items checked for are: 'null', 'Null', 'NULL', '' (empty string), and ' ' (single space). The user can specify additional items to check for using theadd_fuzzy_nullsargument.GF.check_fuzzy_nulls(df['state'],add_fuzzy_nulls=['unknown'])fuzzy_nulls_presentfuzzy_null_countprop_fuzzy_null0True20.2GF.check_fuzzy_nulls(df,add_fuzzy_nulls=['unknown'])columnfuzzy_nulls_presentfuzzy_null_countprop_fuzzy_null0idFalse001nameFalse002stateTrue20.23platformFalse004app_instFalse005lyltyFalse006spendFalse00Unique featuresDescriptionThe methods withinUniqueFeaturesare intended for use with data where each observation has a unique value. Datasurveyor expects unique features to be stored as datetime, object (string), or integer type. In the example data,idis a unique feature.Importing UniqueFeaturesThe unique feature tools can be imported with the command below.fromdatasurveyorimportUniqueFeaturesasUFChecking uniquenessThecheck_uniquenessmethod can be used to check if potentially unique features contain unique values. This method can be applied to a single unique feature or a collection of unique features.UF.check_uniqueness(sample_df['id'])dupes_presentdupe_countprop_dupe0False00UF.check_uniqueness(df[['id','name']])columndupes_presentdupe_countprop_dupe0idFalse001nameTrue10.1Contributing to datasurveyorIf you are interested in contributing to this project:Fork thedatasurveyor repo.Clone the forked repository to your machine.Create a git branch.Make changes and push them to GitHub.Submit your changes for review by creating a pull request. In order to be approved changes should include:Appropriate updates to theREADME.mdGoogle style docstringsTests providing proper coverage of new codeTestingFor those interested in contributing to datasurveyor forking and editing the project, pytest is the testing framework used. To run the tests, create a virtual environment, install the contents ofdev_requirements.txt, and run the following command from the root directory of the project. The testing scripts can be found in thetests/directory.$pytestTo run tests and view coverage, use the below command:$pytest--cov=datasurveyor