Package: ranch-hand
DCEF - Data Cleaning Exploration FrameworkWe all know how awkward it is to clean data in jupyter notebooks.  Multiple cells of exploratory work, trying different transforms, looking up different transforms, adhoc functions that work in one notebook and have to be either copied/pasta-ed to the next notebook, or rewritten from scratch.  Data Cleaning Explorationg Framework  (DCEF) makes all of that better by providing a visual UI for common cleaning operations AND emitting python code that performs the transformation. Specifically, the DCEF is a tool built to interactively explore, clean, and transform pandas dataframes.InstallationIf using JupyterLab,dcefrequires JupyterLab version 3 or higher.You can installdcefusingpipUsingpip:pipinstalldcefDocumentationTo get started with using DCEF, check out the full documentation:https://dcef.readthedocs.io/Using DCEFin a jupyter lab notebook just add the following to a cellfromdcef.dcef_widgetimportDCEFWidgetDCEFWidget(df=df)#df being the dataframe you want to exploreand you will see the UI for DCEFUsing commandsAt the core DCEF commands operate on columns.  You must first click on a cell (not a header) in the top pane to select a column.Next you must click on a command likedropcol,fillna, orgroupbyto create a new commandAfter creating a new command, you will see that command in the commands list, now you must edit the details of a command.  Select the command by clicking on the bottom cell.At this point you can either delete the command by clicking theXbutton or change command parameters.Writing your own commandsBuiltin commands are found inall_transforms.pySimple exampleHere is a simple example commandclassDropCol(Command):command_default=[s('dropcol'),s('df'),"col"]command_pattern=[None]@staticmethoddeftransform(df,col):df.drop(col,axis=1,inplace=True)returndf@staticmethoddeftransform_to_py(df,col):return"    df.drop('%s', axis=1, inplace=True)"%colcommand_defaultis the base configuration of the command when first added,s('dropcol')is a special notation for the function name.s('df')is a symbol notation for the dataframe argument (see LISP section for details)."col"is a placeholder for the selected column.sincedropcoldoes not take any extra arguments,command_patternis[None]deftransform(df,col):df.drop(col,axis=1,inplace=True)returndfThistransformis the function that manipulates the dataframe.  Fordropcolwe take two arguments, the dataframe, and the column name.deftransform_to_py(df,col):return"    df.drop('%s', axis=1, inplace=True)"%coltransform_to_pyemits equivalent python code for this transform.  Code is indented 4 space for use in a function.Complex exampleclassGroupBy(Transform):command_default=[s("groupby"),s('df'),'col',{}]command_pattern=[[3,'colMap','colEnum',['null','sum','mean','median','count']]]@staticmethoddeftransform(df,col,col_spec):grps=df.groupby(col)df_contents={}fork,vincol_spec.items():ifv=="sum":df_contents[k]=grps[k].apply(lambdax:x.sum())elifv=="mean":df_contents[k]=grps[k].apply(lambdax:x.mean())elifv=="median":df_contents[k]=grps[k].apply(lambdax:x.median())elifv=="count":df_contents[k]=grps[k].apply(lambdax:x.count())returnpd.DataFrame(df_contents)TheGroupBycommand is complex.  it takes a 3rd argument ofcol_spec.col_specis an argument of typecolEnum.  AcolEnumargument tells the UI to display a table with all column names, and a drop down box of enum options.In this case each column can have an operation of eithersum,mean,median, orcountapplied to it.Note also the leading3in thecommand_pattern.  That is telling the UI that these are the specs for the 3rd element of the command.  Eventually commands will be able to have multiple configured arguments.Argument typesArguments can currently be configured asinteger- allowing an integer inputenum- allowing a strict set of options, returned as a string to the transformcolEnum- allowing a strict set of options per column, returned as a dictionary keyed on column with values of enum optionsOrder of Operations for data cleaningThe ideal order of operations is as followsColumn level fixesdrop (remove this column)fillna (fill NaN/None with a value)safe int (convert a colum to integers where possible, and nan everywhere else)OneHotEncoding ( create multiple boolean columns from the possible values of this column )MakeCategorical ( change the values of string to a Categorical Data type)QuantizeDataFrame transformations
these transforms largely keep the shape of the data the sameResampleManyColdDecoding (the opposite of OneHotEncoding, take multiple boolean columns and transform into a single categoricalIndex shift (add a column with the value from previous row's column)Dataframe transformations 2
These result in a single new dataframe with a vastly different shapeStack/Unstack columnsGroupBy (with UI for sellect group by function for each column)DataFrame transformations 2
These transforms emit multiple DataFramesRelational extract (extract one or more columns into a second dataframe that can be joined back to a foreign key column)Split on column (emit separate dataframes for each value of a categorical, no shape editting)DataFrame combinationconcat (concatenate multiple dataframes, with UI affordances to assure a similar shape)join (join two dataframes on a key, with UI affordances)DCEF can only work on a single input dataframe shape at a time.  Any newly created columns are visible on output, but not available for manipulation in the same DCEF Cell.Componentsa rich table widget that is embeddable into applications and in the jupyter notebook.A UI for selecting and trying transforms interactivelyAn output table widget showing the transformed dataframeWhat works now, what's comingExists nowReact frontend appDisplays a datatframeSimple UI for column level functionsShows generated python codeShows transformed data frameDCEF serverServes up dataframes for use by frontendresponds to dcef commandsshows generated python codeDeveloper User experiencedefine DCEF commands in python onloyDCEF IntepreterBased on Peter Norvig's lispy.py, a simple syntax that is easy for the frontend to generate (no parens, just JSON arrays)DCEF core (actual transforms supported)dropcolfillnaone hotsafe intGroupByNext major featuresJupyter Notebook widgetembed the same UI from the frontend into a jupyter notebook shellNo need to fire up a separate server, commands sent via ipywidgets.commsAdd a "send generated python to next cell" functionReact frontend appStylingServer only, some UI for DataFrame selectionPre filtering concept (only operate on first 1000 rows, some sample of all rows)DataFrame joining UISummary statistics tab for incoming dataframeMulti index columnsDateTimeIndex supportDCEF coreMakeCategoricalQuantizeResampleManyColdDecodingIndexShiftComputedStack/UnstackRelationalExtractSplitconcatjoinDevelopment installationFor a development installation:gitclonehttps://github.com/paddymul/dcef.gitcddcef
condainstallipywidgets=8jupyterlab
pipinstall-ve.Enabling development install for Jupyter notebook:Enabling development install for JupyterLab:jupyterlabextensiondevelop.--overwriteNote for developers: the--symlinkargument on Linux or OS X allows one to modify the JavaScript code in-place. This feature is not available with Windows.
`ContributionsWe :heart: contributions.Have you had a good experience with this project? Why not share some love and contribute code, or just let us know about any issues you had with it?We welcome issue reportshere; be sure to choose the proper issue template for your issue, so that we can be sure you're providing the necessary information.Before sending aPull Request, please make sure you read our