Package: scikit-na
scikit-nascikit-nais a Python package for missing data (NA) analysis. The package includes many functions for statistical analysis, modeling, and data visualization. The latter is done using two packages —matplotlibandAltair.FeaturesInteractive report (based onipywidgets)Descriptive statisticsRegression modelingHypotheses testsData visualizationInstallationPackage can be installed from PyPi:pipinstallscikit-naor from this repo:pipinstallgit+https://github.com/maximtrp/scikit-na.gitExampleWe will use Titanic dataset (from Kaggle) that contains NA values in three columns: Age, Cabin, and Embarked.SummaryPer each columnBy default,summary()function returns the results for each column.importscikit_naasnaimportpandasaspddata=pd.read_csv('titanic_dataset.csv')# Excluding three columns without NA to fit the table herena.summary(data,columns=data.columns.difference(['SibSp','Parch','Ticket']))AgeCabinEmbarkedFareNamePassengerIdPclassSexSurvivedNA count1776872000000NA, % (per column)19.8777.10.22000000NA, % (of all NAs)20.4479.330.23000000NA unique (per column)195292000000NA unique, % (per column)10.7377100000000Rows left after dropna()714204889891891891891891891Rows left after dropna(), %80.1322.999.78100100100100100100NA uniqueis the number of NA values per each column that are unique for it, i.e. do not intersect with NA values in the other columns (or that will remain in dataset if we drop NA values in the other columns).Whole datasetWe can also get a summary of missing data for the whole dataset:na.summary(data,per_column=False)datasetTotal columns12Total rows891Rows with NA708Rows without NA183Total cells10692Cells with NA866Cells with NA, %8.1Cells with non-missing data9826Cells with non-missing data, %91.9CorrelationsTo calculate correlations between columns in terms of missing data, just callcorrelate()function with your DataFrame as the first argument:na.correlate(data,method="spearman").round(3)EmbarkedAgeCabinEmbarked1-0.024-0.087Age-0.02410.144Cabin-0.0870.1441This method can be used to uncover hidden patterns in missing data across many
columns in a dataset. Columns with no missing data are automatically excluded.There is a function to visualize correlations with a heatmap:na.altair\.plot_corr(data,corr_kws={'method':'spearman'}).properties(width=150,height=150)VisualizationHeatmapNow, let's visualize NA values on a heatmap. We will be usingAltair+Vegabackend:na.altair.plot_heatmap(data)Droppables are those values that will be dropped if we simply usepandas.DataFrame.dropna()on thewhole dataset.Stairs plotStairs plot is one more useful visualization of dataset shrinkage on applyingpandas.Series.dropna()method to each column sequentially (sorted by the
number of NA values, by default):na.altair.plot_stairs(data)After dropping all NAs inCabincolumn, we are left with 21 more NAs (inAgeandEmbarkedcolumns). This plot also shows tooltips with exact numbers of NA
values that are dropped per each column.HistogramYou may need to adjust some parameters before a histogram starts looking as you expect:chart=na.altair.plot_hist(data,col='Pclass',col_na='Age')\.properties(width=200,height=200)chart.configure_axisX(labelAngle=0)Regression modelWe can build a logistic regression model withAgeas a dependent variable andFare,Parch,Pclass,SibSp,Survivedas independent variables.
Internally,pandas.Series.isna()method is called onAgecolumn, and the
resulting boolean values are converted to integers (True/Falsebecomes1/0). Finally, fitting a logistic model is done bystatsmodelspackage:# Selecting columns with numeric data# Dropping "PassengerId" columnsubset=data.loc[:,data.dtypes!=object].drop(columns=['PassengerId'])model=na.model(subset,col_na='Age')model.summary()Optimization terminated successfully.
Current function value: 0.467801
Iterations 7
                        Logit Regression Results                           
===============================================================================
Dep. Variable:                    Age   No. Observations:                   891
Model:                          Logit   Df Residuals:                       885
Method:                           MLE   Df Model:                             5
Date:                Sat, 05 Jun 2021   Pseudo R-squ.:                  0.06164
Time:                        17:51:31   Log-Likelihood:                 -416.81
converged:                       True   LL-Null:                        -444.19
Covariance Type:            nonrobust   LLR p-value:                  1.463e-10
===============================================================================
                coef    std err          z      P>|z|      [0.025      0.975]
-------------------------------------------------------------------------------
(intercept)    -2.7294      0.429     -6.369      0.000      -3.569      -1.890
Fare            0.0010      0.003      0.376      0.707      -0.004       0.006
Parch          -0.8874      0.223     -3.984      0.000      -1.324      -0.451
Pclass          0.5953      0.147      4.046      0.000       0.307       0.884
SibSp           0.2548      0.095      2.684      0.007       0.069       0.441
Survived       -0.1026      0.198     -0.519      0.604      -0.490       0.285
===============================================================================Interactive reportUsescikit_na.report()function to show interactive report interface:na.report(data)ContributionAny contribution is highly appreciated: pull requests, suggestions, or bug reports.