Package: tf2-keras-pandas
tf2-keras-pandastl;dr:keras-pandas allows users to rapidly build and iterate on deep learning models. Updated for tensorflow 2.0Getting data formatted and into keras can be tedious, time consuming, and require domain expertise, whether your a
veteran or new to Deep Learning.keras-pandasovercomes these issues by (automatically) providing:Data transformations: A cleaned, transformed and correctly formattedXandy(good for keras, sklearn or any
other ML
platform)Data piping: Correctly formatted keras input, hidden and output layers to quickly start iterating onThese approaches are build on best in world approaches from practitioners, kaggle grand masters, papers, blog posts,
and coffee chats, to simple entry point into the world of deep learning, and a strong foundation for deep learning
experts.For more info, check out the:CodeDocumentationPyPiIssue trackerQuick StartLet's build a model with thelending club data set. This data set is
particularly fun because this data set contains a mix of text, categorical and numerical data types, and features a
lot of null values.pipinstall--upgradetf2-keras-pandasfromtensorflow.kerasimportModelfromkeras_pandasimportlibfromkeras_pandas.AutomaterimportAutomaterfromsklearn.model_selectionimporttrain_test_split# Load dataobservations=lib.load_lending_club()# Train /test splittrain_observations,test_observations=train_test_split(observations)train_observations=train_observations.copy()test_observations=test_observations.copy()# List out variable typesdata_type_dict={'numerical':['loan_amnt','annual_inc','open_acc','dti','delinq_2yrs','inq_last_6mths','mths_since_last_delinq','pub_rec','revol_bal','revol_util','total_acc','pub_rec_bankruptcies'],'categorical':['term','grade','emp_length','home_ownership','loan_status','addr_state','application_type','disbursement_method'],'text':['desc','purpose','title']}output_var='loan_status'# Create and fit Automaterauto=Automater(data_type_dict=data_type_dict,output_var=output_var)auto.fit(train_observations)# Transform datatrain_X,train_y=auto.fit_transform(train_observations)test_X,test_y=auto.transform(test_observations)# Create and fit keras (deep learning) model.x=auto.input_nubx=auto.output_nub(x)model=Model(inputs=auto.input_layers,outputs=x)model.compile(optimizer='adam',loss=auto.suggest_loss())And that's it! In a couple of lines, we've created a model that accepts a few dozen variables, and can create a world
class deep learning modelUsageInstallationYou can installtf2-keras-pandaswithpip:pipinstall-Utf2-keras-pandasCreating an AutomaterTheAutomaterobject is the central object inkeras-pandas. It accepts a dictionary of the format{'datatype':  ['var1', var2']}For example we could create an automater using the built innumerical,categorical, andtextdatatypes, by
calling:# List out variable typesdata_type_dict={'numerical':['loan_amnt','annual_inc','open_acc','dti','delinq_2yrs','inq_last_6mths','mths_since_last_delinq','pub_rec','revol_bal','revol_util','total_acc','pub_rec_bankruptcies'],'categorical':['term','grade','emp_length','home_ownership','loan_status','addr_state','application_type','disbursement_method'],'text':['desc','purpose','title']}output_var='loan_status'# Create and fit Automaterauto=Automater(data_type_dict=data_type_dict,output_var=output_var)As a side note, the response variable must be in one of the variable type lists (e.g.loan_statusis incategorical_vars)One variable typeIf you only have one variable type, only use one variable type!# List out variable typesdata_type_dict={'categorical':['term','grade','emp_length','home_ownership','loan_status','addr_state','application_type','disbursement_method']}output_var='loan_status'# Create and fit Automaterauto=Automater(data_type_dict=data_type_dict,output_var=output_var)Multiple variable typesIf you have multiple variable types, feel free to use all of them! Built in datatypes are listed inAutomater.datatype_handlers# List out variable typesdata_type_dict={'numerical':['loan_amnt','annual_inc','open_acc','dti','delinq_2yrs','inq_last_6mths','mths_since_last_delinq','pub_rec','revol_bal','revol_util','total_acc','pub_rec_bankruptcies'],'categorical':['term','grade','emp_length','home_ownership','loan_status','addr_state','application_type','disbursement_method'],'text':['desc','purpose','title']}output_var='loan_status'# Create and fit Automaterauto=Automater(data_type_dict=data_type_dict,output_var=output_var)Custom datatypesIf there's a specific datatype you'd like to use that's not built in (such as images, videos, or geospatial), you can
include it by usingAutomater'sdatatype_handlersparameter.A template datatype can be found inkeras_pandas/data_types/Abstract.py. Filling out this template will yield a new
datatype handler. If you're happy with your work and want to share your new datatype handler, create a PR (and check
outcontributing.md)Nooutput_varIf your model doesn't need a response var, or your use case doesn't usekeras-pandas's output functionality, you
can skip theoutput_varby setting it to None# List out variable typesdata_type_dict={'categorical':['term','grade','emp_length','home_ownership','loan_status','addr_state','application_type','disbursement_method']}output_var=None# Create and fit Automaterauto=Automater(data_type_dict=data_type_dict,output_var=output_var)Fitting the AutomaterBefore use, theAutomatormust be fit. Thefit()method accepts a pandas DataFrame, which must contain all of the
columns listed during initialization.auto.fit(observations)Transforming dataNow, we can use ourAutomaterto transform the dataset, from a pandas DataFrame to numpy objects properly formatted
for Keras's input and output layers.X,y=auto.transform(observations,df_out=False)This will return two objects:X: An array, containing numpy object for each Keras input. This is generally one Keras input for each user
input variable.y: A numpy object, containing the response variable (if one was provided)Using input / output nubsSetting up correctly formatted, heuristically 'good' input and output layers is oftenTediousTime consumingDifficult for those new to KerasWith this in mind,keras-pandasprovides correctly formatted input and output 'nubs'.The input nub is correctly formatted to accept the output fromauto.transform(). It contains one Keras Input layer
for each generated input, may contain addition layers, and has all input piplines joined with aConcatenatelayer.The output layer is correctly formatted to accept the response variable numpy object.ChangelogPR title (#PR number, or #Issue if no PR)There's nothing here! (yet)DevelopmentUpdated README and setup.py links (No PR)3.1.0Add boolean datatype (#104)Added Contributing.md section for new datatypes (#101)Added datatypes to docs in index.rst (#101)Modified documentation to automatically generate API docs (#101)3.0.1Changing CI to Circleci (#100)Adding datatypes to CONTRIBUTING.md, adding CONTRIBUTING.md to docs (#96)Adding docs badge (#95)Adding support for unusual variable names / format keras names to be valid in name scope (#92)Adding examples (#93)Upgradedrequestslibrary torequests==2.20.1, based on security concern (#94)3.0.0Brand new release, withAddedNewDatatypeinterface, with easier to understand pipelines for each datatypeAll existing datatypes (Numerical,Categorical,Text&TimeSeries) re-implmented in this new formatSupport for custom data types generated by usersDuck-typing helper method (keras_pandas/lib.check_valid_datatype()) to confirm that a datatype has valid
signatureNew testing, streamlined and standardizedSupport for transforming unseen categorical levels, via theUNKtoken (experimental)ModifiedUpdatedAutomaterinterface, which accepts a dictionary of data typesHeavily updated READMEMore consistent logging and data formatting for sample data setsRemovedRemoved examples, will be re-implemented in future releaseAll existing unittestsBulk of new datatypes incontributing.md, will be re-added in future release2.2.0Add timeseries support (#78)Add timeseries examples (#79)2.1.0Boolean support deprecated. Boolean (bool) data type can be treated as a special case of categorical data types2.0.2Remove a lot of the unnecessary dependencies (#75)Update dependencies to contemporary versions (#74)2.0.1Fix issue w/ PyPi conflict2.0.0Adding CI/CD and PyPi links, and updating contact section w/ about the author (#70)Major rewrite / update of examples (#72)Fixes bug in embedding transformer. Embeddings will now be at least length 1.Add functionality to check ifresp_varis in the list of user provided variablesAdded better null filling w/CategoricalImputerAdded filling unseen values w/CategoricalImputerConverted default transformer pipeline to usecopy.deepcopyinstead ofcopy.copy. This was a hotfix for a
previously unknown issue.Standardizing setting logging level, only in test base class and examples (when__main__)1.3.5Adding regression example w/ inverse_transformation (#64)Fixing issue where web socket connections were being opened needlessly (#65)1.3.4AddingManifest.in, with including files references insetup.py(#54)1.3.2Fixed poorly written text embedding index unit test (#52)Added license (#49)EarlierLots of things happened. Break things and move fast