Package: bulldog
BulldogThe guardian dog that prevents you from writing poor code when doing data analysis in Python.InstallationSimply run:pip install bulldogPhilosophyBulldog is a library for writing better code in your analysis that largely borrows from the state management libraries for application development (Redux,Flux,Vuex,Katana..).Bulldog models are composed of five main building blocks:data, which is our model initial datadata_modifiers, which are special function whose main task is to modify the model's databusiness_logic, which are function whose main task is to execute the business logic and invokedata_modifiersanalyses, which subscribe to change on the model'sdatahistory, which is a backlog of all the operations that have occurred and the corresponding state of the modeldataThe philosophy behind bulldog is toseparatedata transformations, business logic and analyses, in order to make
clarity, testing and debugging easier to achieve.Working with bulldogTo create a bulldog model simply run:frombulldog.modelimportModelimportpandasaspdmodel=Model(data={'df':pd.DataFrame(pd.np.ones((100,100))),'other_data':[1,2,3]})All the data is stored in ourmodel.dataand it's not directly modifiable. In fact, whenever we accessmodel.datawe are actually accessing a copy of the original model data.In order to alter/modify our data we need to create some special functions calleddata_modifiers.@model.data_modifierdefdata_step(data,factor):df=data['df']df*=factorreturndata# this will modify the dataAs we can see data modifiers are just simple, pure functions that take our model data as input and perform some kind of alteration on it
and return the altered data. The signature of a business model isfunction(data, *args, **kwargs)and it needs to be
decorated with the@model.data_modifierdecorator (wheremodelis your instance of Bulldog'sModel).If we want to execute adata_modifier, rather than calling it directly we need to ask the model to commit it:model.commit('data_step',factor=9)# 'data_step' is the name of our `data_modifier`Note that any other way of calling the function will result in an error. E.g.:data_step(data=model.data,factor=9)# wrong; this will throw an errorGreat! but what if we need to run some business logic to conditionally modify our dataset?
Maybe we need to download some data and based on that perform some actions that will eventually
lead us to modify our data. In this case we should use abusiness_logicfunction.@model.business_logic@model.checkpointdefaction1(data,commit):data['df']/=8000# this has no effect whatsoever on our data, remember? We are modifying a copyifmax(data['df'])<0.38:commit("data_step",9)# but this will actually modify our dataAs we can seebusiness_logicare function with the signaturefunction(data, commit, *args, **kwargs)which take as input the data
and have the possibility of committingdata_modifierfunctions to our original modelYou might have noted the additional@model.checkpointdecorator (which can also be applied todata_modifiers). It will basically tell our model to store the current state data after computing
this function (and store it inmodel.history), allowing us to restore it or inspect it at a later stage, which is very convenient for debugging.Similarly todata_modifiers, alsobusiness_logiccannot be execute directly, and have to be dispatched through the model in this way:model.dispatch('action1')Now, you might wonder how to run analyses on the model's data. That's fairly simple!@model.analysis@model.parallelizabledefanalysis(data,history):df=data['df']time.sleep(3)print('fast 1',list(history.keys())[-1].name,pd.np.mean(df.values))Analyses are functions with signaturefunction(data, history)that are run automatically every time a checkpoint step of our model is executed.
Optionally analyses can be run in parallel (if you use the@model.parallelizabledecorator, as above). This is particularly convenient
in case we are computing a large number of metrics and want to leverage our CPU as much as possible.
Note that only analyses can be parallelized in Bulldog.Custom checkpoints managementOut of the box, Bulldog doesn't implement any custom diffing logic for the modeldata(since it's a generic dictionary which could contain anything),
but you can provide your own functions to checkpoint & restore your data. For example you might want to write/read:from a databasefrom a pickled file on diskfrom h5dfdiffs from custom diffing tools (or generic ones likecsv-diff)If you want to provide some custom save/load logic to handle checkpoint save & restore, pass these two functions to the Model initializer:on_checkpoint_save(data, version_key, history): this function is responsible for saving thedata(or a diff of it which you can compute by comparing it with your modelhistory, holding every other checkpoint data)on_checkpoint_restore(version_key, history): this function is responsible for restoring data from a previous checkpointFor example if you want to read from disk pickled objects you might do:defon_checkpoint_save(data,key,history):file_name='data_{}.pkl'.format(key.step)pickle.dump(data,open('data_{}.pkl'.format(key.step),'wb'))returnfile_name# only the file name will be saved in memorydefon_checkpoint_restore(key,history):file_name=history[key]returnpickle.load(open(file_name,'rb'))# store this in model.datamodel=Model(data={'df':pd.DataFrame(pd.np.ones((100,100))),},on_checkpoint_save=on_checkpoint_save,on_checkpoint_restore=on_checkpoint_restore)Advanced usageBulldog has a few nice features for people that use interactive editors (likeipythonorjupyter notebook).You can prevent the samebusiness_logicfrom running multiple times by settingunique_bl_step=TrueinModel. This will prevent your state from being modified multiple times if you re-run cells in a notebook.You can restore the version model data at a previous checkpoint by running eitherrollback(n_steps)orrevert_version(Version). This is useful both for reproducibility/debugging and for jupyter users who don't want to re-run a whole lengthy analysis after a wrong alteration of the model data.Testing:still to be developed. Ideally bulldog will allow you to test every single component in a much easier way and possibly also with mocked data.