Package: keras-pandas
keras-pandastl;dr:keras-pandas allows users to rapidly build and iterate on deep learning models.Getting data formatted and into keras can be tedious, time consuming, and difficult, whether your a veteran or new to
Keras.keras-pandasovercomes these issues by (automatically) providing:A cleaned, transformed and correctly formattedXandy(good for keras, sklearn or any other ML platform)An 'input nub', without the hassle of worrying about input shapes or data typesAn output layer, correctly formatted for the kind of response variable providedWith these resources, it's possible to rapidly build and iterate on deep learning models, and focus on the parts of
modeling that you enjoy!For more info, check out the:CodeDocumentationIssue trackerAuthor's websitePyPiCI/CDQuick StartLet's build a model with thelending club data set. This data set is
particularly fun because this data set contains a mix of text, categorical and numerical data types, and features a
lot of null values.fromkerasimportModelfromkeras_pandasimportlibfromkeras_pandas.AutomaterimportAutomaterfromsklearn.model_selectionimporttrain_test_split# Load dataobservations=lib.load_lending_club()# Train /test splittrain_observations,test_observations=train_test_split(observations)train_observations=train_observations.copy()test_observations=test_observations.copy()# List out variable typesdata_type_dict={'numerical':['loan_amnt','annual_inc','open_acc','dti','delinq_2yrs','inq_last_6mths','mths_since_last_delinq','pub_rec','revol_bal','revol_util','total_acc','pub_rec_bankruptcies'],'categorical':['term','grade','emp_length','home_ownership','loan_status','addr_state','application_type','disbursement_method'],'text':['desc','purpose','title']}output_var='loan_status'# Create and fit Automaterauto=Automater(data_type_dict=data_type_dict,output_var=output_var)auto.fit(train_observations)# Transform datatrain_X,train_y=auto.fit_transform(train_observations)test_X,test_y=auto.transform(test_observations)# Create and fit keras (deep learning) model.x=auto.input_nubx=auto.output_nub(x)model=Model(inputs=auto.input_layers,outputs=x)model.compile(optimizer='adam',loss=auto.suggest_loss())And that's it! In a couple of lines, we've created a model that accepts a few dozen variables, and can create a world
class deep learning modelUsageInstallationYou can installkeras-pandaswithpip:pipinstall-Ukeras-pandasCreating an AutomaterTheAutomaterobject is the central object inkeras-pandas. It accepts a dictionary of the format{'datatype':  ['var1', var2']}For example we could create an automater using the built innumerical,categorical, andtextdatatypes, by
calling:# List out variable typesdata_type_dict={'numerical':['loan_amnt','annual_inc','open_acc','dti','delinq_2yrs','inq_last_6mths','mths_since_last_delinq','pub_rec','revol_bal','revol_util','total_acc','pub_rec_bankruptcies'],'categorical':['term','grade','emp_length','home_ownership','loan_status','addr_state','application_type','disbursement_method'],'text':['desc','purpose','title']}output_var='loan_status'# Create and fit Automaterauto=Automater(data_type_dict=data_type_dict,output_var=output_var)As a side note, the response variable must be in one of the variable type lists (e.g.loan_statusis incategorical_vars)One variable typeIf you only have one variable type, only use one variable type!# List out variable typesdata_type_dict={'categorical':['term','grade','emp_length','home_ownership','loan_status','addr_state','application_type','disbursement_method']}output_var='loan_status'# Create and fit Automaterauto=Automater(data_type_dict=data_type_dict,output_var=output_var)Multiple variable typesIf you have multiple variable types, feel free to use all of them! Built in datatypes are listed inAutomater.datatype_handlers# List out variable typesdata_type_dict={'numerical':['loan_amnt','annual_inc','open_acc','dti','delinq_2yrs','inq_last_6mths','mths_since_last_delinq','pub_rec','revol_bal','revol_util','total_acc','pub_rec_bankruptcies'],'categorical':['term','grade','emp_length','home_ownership','loan_status','addr_state','application_type','disbursement_method'],'text':['desc','purpose','title']}output_var='loan_status'# Create and fit Automaterauto=Automater(data_type_dict=data_type_dict,output_var=output_var)Custom datatypesIf there's a specific datatype you'd like to use that's not built in (such as images, videos, or geospatial), you can
include it by usingAutomater'sdatatype_handlersparameter.A template datatype can be found inkeras_pandas/data_types/Abstract.py. Filling out this template will yield a new
datatype handler. If you're happy with your work and want to share your new datatype handler, create a PR (and check
outcontributing.md)Nooutput_varIf your model doesn't need a response var, or your use case doesn't usekeras-pandas's output functionality, you
can skip theoutput_varby setting it to None# List out variable typesdata_type_dict={'categorical':['term','grade','emp_length','home_ownership','loan_status','addr_state','application_type','disbursement_method']}output_var=None# Create and fit Automaterauto=Automater(data_type_dict=data_type_dict,output_var=output_var)Fitting the AutomaterBefore use, theAutomatormust be fit. Thefit()method accepts a pandas DataFrame, which must contain all of the
columns listed during initialization.auto.fit(observations)Transforming dataNow, we can use ourAutomaterto transform the dataset, from a pandas DataFrame to numpy objects properly formatted
for Keras's input and output layers.X,y=auto.transform(observations,df_out=False)This will return two objects:X: An array, containing numpy object for each Keras input. This is generally one Keras input for each user
input variable.y: A numpy object, containing the response variable (if one was provided)Using input / output nubsSetting up correctly formatted, heuristically 'good' input and output layers is oftenTediousTime consumingDifficult for those new to KerasWith this in mind,keras-pandasprovides correctly formatted input and output 'nubs'.The input nub is correctly formatted to accept the output fromauto.transform(). It contains one Keras Input layer
for each generated input, may contain addition layers, and has all input piplines joined with aConcatenatelayer.The output layer is correctly formatted to accept the response variable numpy object.ContactHey, I'm Brendan Herger, avaiable athttps://www.hergertarian.com/. Please feel free
to reach out to me at13herger <at> gmail <dot> comI enjoy bridging the gap between data science and engineering, to build and
deploy data products. I'm not currently pursuing contract work.I've enjoyed building a unique combination of machine learning, deep learning, and software engineering skills. In my
previous work at Capital One and startups, I've has built authorization fraud, insider threat, and legal discovery
automation platforms. In each of these cases I've lead a team of data scientists and data engineers to enable and
elevate our client's business workflow (and capture some amazing data).When I'm not knee deep in a code base, I can be found traveling, sharing my collection of Japanese teas, and playing
board games with my partner in Seattle.ChangelogPR title (#PR number, or #Issue if no PR)There's nothing here! (yet)DevelopmentThere's nothing here! (yet)3.1.0Add boolean datatype (#104)Added Contributing.md section for new datatypes (#101)Added datatypes to docs in index.rst (#101)Modified documentation to automatically generate API docs (#101)3.0.1Changing CI to Circleci (#100)Adding datatypes to CONTRIBUTING.md, adding CONTRIBUTING.md to docs (#96)Adding docs badge (#95)Adding support for unusual variable names / format keras names to be valid in name scope (#92)Adding examples (#93)Upgradedrequestslibrary torequests==2.20.1, based on security concern (#94)3.0.0Brand new release, withAddedNewDatatypeinterface, with easier to understand pipelines for each datatypeAll existing datatypes (Numerical,Categorical,Text&TimeSeries) re-implmented in this new formatSupport for custom data types generated by usersDuck-typing helper method (keras_pandas/lib.check_valid_datatype()) to confirm that a datatype has valid
signatureNew testing, streamlined and standardizedSupport for transforming unseen categorical levels, via theUNKtoken (experimental)ModifiedUpdatedAutomaterinterface, which accepts a dictionary of data typesHeavily updated READMEMore consistent logging and data formatting for sample data setsRemovedRemoved examples, will be re-implemented in future releaseAll existing unittestsBulk of new datatypes incontributing.md, will be re-added in future release2.2.0Add timeseries support (#78)Add timeseries examples (#79)2.1.0Boolean support deprecated. Boolean (bool) data type can be treated as a special case of categorical data types2.0.2Remove a lot of the unnecessary dependencies (#75)Update dependencies to contemporary versions (#74)2.0.1Fix issue w/ PyPi conflict2.0.0Adding CI/CD and PyPi links, and updating contact section w/ about the author (#70)Major rewrite / update of examples (#72)Fixes bug in embedding transformer. Embeddings will now be at least length 1.Add functionality to check ifresp_varis in the list of user provided variablesAdded better null filling w/CategoricalImputerAdded filling unseen values w/CategoricalImputerConverted default transformer pipeline to usecopy.deepcopyinstead ofcopy.copy. This was a hotfix for a
previously unknown issue.Standardizing setting logging level, only in test base class and examples (when__main__)1.3.5Adding regression example w/ inverse_transformation (#64)Fixing issue where web socket connections were being opened needlessly (#65)1.3.4AddingManifest.in, with including files references insetup.py(#54)1.3.2Fixed poorly written text embedding index unit test (#52)Added license (#49)EarlierLots of things happened. Break things and move fast