Package: batchflow
BatchFlowBatchFlowhelps you conveniently work with random or sequential batches of your data
and define data processing and machine learning workflows even for datasets that do not fit into memory.For more details seethe documentation and tutorials.Main features:flexible batch generatondeterministic and stochastic pipelinesdatasets and pipelines joins and mergesdata processing actionsflexible model configurationwithin batch parallelismbatch prefetchingready to use ML models and proven NN architecturesconvenient layers and helper functions to build custom modelsa powerful research engine with parallel model training and extended experiment logging.Basic usagemy_workflow=my_dataset.pipeline().load('/some/path').do_something().do_something_else().some_additional_action().save('/to/other/path')The trick here is that all the processing actions are lazy. They are not executed until their results are needed, e.g. when you request a preprocessed batch:my_workflow.run(BATCH_SIZE,shuffle=True,n_epochs=5)orforbatchinmy_workflow.gen_batch(BATCH_SIZE,shuffle=True,n_epochs=5):# only now the actions are fired and data is being changed with the workflow defined earlier# actions are executed one by one and here you get a fully processed batchorNUM_ITERS=1000foriinrange(NUM_ITERS):processed_batch=my_workflow.next_batch(BATCH_SIZE,shuffle=True,n_epochs=None)# only now the actions are fired and data is changed with the workflow defined earlier# actions are executed one by one and here you get a fully processed batchTrain a neural networkBatchFlowincludes ready-to-use proven architectures like VGG, Inception, ResNet and many others.
To apply them to your data just choose a model, specify the inputs (like the number of classes or images shape)
and calltrain_model. Of course, you can also choose a loss function, an optimizer and many other parameters, if you want.frombatchflow.models.torchimportResNet34my_workflow=my_dataset.pipeline().init_model('model',ResNet34,config={'loss':'ce','classes':10}).load('/some/path').some_transform().another_transform().train_model('ResNet34',inputs=B.images,targets=B.labels).run(BATCH_SIZE,shuffle=True)For more advanced cases and detailed API seethe documentation.InstallationBatchFlowmodule is in the beta stage. Your suggestions and improvements are very welcome.BatchFlowsupports python 3.6 or higher.Stable python packageWithpoetrypoetry add batchflowWith old-fashionedpippip3 install batchflowDevelopment versionWithpoetrypoetry add --editable git+https://github.com/analysiscenter/batchflowWith old-fashionedpippip install --editable git+https://github.com/analysiscenter/batchflowExtrasSomebatchflowfunctions and classed require additional dependencies.
In order to use that functionality you might need to installbatchflowwith extras (e.g.batchflow[nn]):image - working with image datasets and plottingnn - for neural networks (includes torch, torchvision, ...)datasets - loading standard datasets (MNIST, CIFAR, ...)profile - performance profilingjupyter - utility functions for notebooksresearch - multiprocess researchtelegram - for monitoring pipelines via a telegram botdev - batchflow development (pylint, pytest, ...)You can install several extras at once, likebatchflow[image,nn,research].Projects based on BatchFlowSeismiQB- ML for seismic interpretationSeismicPro- ML for seismic processingPetroFlow- ML for well interpretationPyDEns- DL Solver for ODE and PDERadIO- ML for CT imagingCardIO- ML for heart signalsCiting BatchFlowPlease cite BatchFlow in your publications if it helps your research.Roman Khudorozhkov et al. BatchFlow library for fast ML workflows. 2017. doi:10.5281/zenodo.1041203@misc{roman_kh_2017_1041203,
  author       = {Khudorozhkov, Roman and others},
  title        = {BatchFlow library for fast ML workflows},
  year         = 2017,
  doi          = {10.5281/zenodo.1041203},
  url          = {https://doi.org/10.5281/zenodo.1041203}
}