Package: datawig
DataWig - Imputation for TablesDataWig learns Machine Learning models to impute missing values in tables.See our user-guide and extended documentationhere.InstallationCPUpip3installdatawigGPUIf you want to run DataWig on a GPU you need to make sure your version of Apache MXNet Incubating contains the GPU bindings.
Depending on your version of CUDA, you can do this by running the following:wgethttps://raw.githubusercontent.com/awslabs/datawig/master/requirements/requirements.gpu-cu${CUDA_VERSION}.txt
pipinstalldatawig--no-deps-rrequirements.gpu-cu${CUDA_VERSION}.txt
rmrequirements.gpu-cu${CUDA_VERSION}.txtwhere${CUDA_VERSION}can be75(7.5),80(8.0),90(9.0), or91(9.1).Running DataWigThe DataWig API expects your data as apandas DataFrame. Here is an example of how the dataframe might look:Product TypeDescriptionSizeColorShoeIdeal for Running12UKBlackSDCardsBest SDCard ever ...8GBBlueDressThisyellowdressM?Quickstart ExampleFor most use cases, theSimpleImputerclass is the best starting point. For convenience there is the functionSimpleImputer.completethat takes a DataFrame and fits an imputation model for each column with missing values, with all other columns as inputs:importdatawig,numpy# generate some data with simple nonlinear dependencydf=datawig.utils.generate_df_numeric()# mask 10% of the valuesdf_with_missing=df.mask(numpy.random.rand(*df.shape)>.9)# impute missing valuesdf_with_missing_imputed=datawig.SimpleImputer.complete(df_with_missing)You can also impute values in specific columns only (calledoutput_columnbelow) using values in other columns (calledinput_columnsbelow). DataWig currently supports imputation of categorical columns and numeric columns.Imputation of categorical columnsimportdatawigdf=datawig.utils.generate_df_string(num_samples=200,data_column_name='sentences',label_column_name='label')df_train,df_test=datawig.utils.random_split(df)#Initialize a SimpleImputer modelimputer=datawig.SimpleImputer(input_columns=['sentences'],# column(s) containing information about the column we want to imputeoutput_column='label',# the column we'd like to impute values foroutput_path='imputer_model'# stores model data and metrics)#Fit an imputer model on the train dataimputer.fit(train_df=df_train)#Impute missing values and return original dataframe with predictionsimputed=imputer.predict(df_test)Imputation of numerical columnsimportdatawigdf=datawig.utils.generate_df_numeric(num_samples=200,data_column_name='x',label_column_name='y')df_train,df_test=datawig.utils.random_split(df)#Initialize a SimpleImputer modelimputer=datawig.SimpleImputer(input_columns=['x'],# column(s) containing information about the column we want to imputeoutput_column='y',# the column we'd like to impute values foroutput_path='imputer_model'# stores model data and metrics)#Fit an imputer model on the train dataimputer.fit(train_df=df_train,num_epochs=50)#Impute missing values and return original dataframe with predictionsimputed=imputer.predict(df_test)In order to have more control over the types of models and preprocessings, theImputerclass allows directly specifying all relevant model features and parameters.For details on usage, refer to the providedexamples.AcknowledgmentsThanks toDavid Greenbergfor the package name.Building documentationgitclonegit@github.com:awslabs/datawig.gitcddatawig/docs
makehtml
open_build/html/index.htmlExecuting TestsClone the repository from git and set up virtualenv in the root dir of the package:python3 -m venv venvInstall the package from local sources:./venv/bin/pip install -e .Run tests:./venv/bin/pip install -r requirements/requirements.dev.txt
./venv/bin/python -m pytestUpdating PyPi distributionBefore updating, increment the version in setup.py.git clone git@github.com:awslabs/datawig.git
cd datawig
# build local distribution for current version
python setup.py sdist
# upload to PyPi
twine upload --skip-existing dist/*