Package: quick-batch
quick_batchquick_batchis an ultra-simple command-line tool for large batch python-driven processing and transformation.  It was designed to be fast to deploy, transparent, and portable.  This allows you to scale anyprocessorfunction that needs to be run over a large set of input data, enabling batch/parallel processing of the input with minimal setup and teardown.quick_batchGetting startedUsageScalingInstallationTheprocessor.pyfileWhy use quick_batchGetting startedAll you need to scale batch transformations withquick_batchis atransformation function(s) in aprocessor.pyfileDockerfilecontaining a container build appropriate to y our processoran optionalrequirements.txtfile containing required python modulesDocument paths to these objects as well as other parameters in aconfig.yamlconfig file of the form below.Underprocessoryou can either define adockerfile_pathto your Dockerfile or animage_nameto a pre-built image to be pulled.data:input_path:/path/to/your/input/dataoutput_path:/path/to/your/output/datalog_path:/path/to/your/log/filequeue:feed_rate:<int - number of examples processed per processor instance>order_files:<boolean - whether or not to order input files by size>processor:dockerfile_path:/path/to/your/Dockerfile ORimage_name:<image_name_to_pull>requirements_path:/path/to/your/requirements.txtprocessor_path:/path/to/your/processor/processor.pynum_processors:<int - instances of processor to run in parallel>quick_batchwill point yourprocessor.pyat theinput_pathdefined in thisconfig.yamland process the files listed in it in parallel at a scale given by your choice ofnum_processors.Output will be written to theoutput_pathspecified in the configuration file.You can see theexamplesdirectory for examples of valid configs, processors, requirements, and dockerfiles.UsageTo start processing with yourconfig.yamlusequick_batch'sconfigcommand at the terminal by typingquick_batchconfig/path/to/your/config.yamlThis will start the build and deploy process for processing your data as defined in yourconfig.yaml.ScalingUse thescalecommoand to manually scale the number of processors / containers running your processquick_batchscale<num_processors>Here<num_processors>is an integer >= 1.   For example, to scale to 3 parallel processors / containers:quick_batch scale 3InstallationTo install quick_batch, simply usepip:pipinstallquick-batchTheprocessor.pyfileCreate aprocessor.pyfile with the following basic pattern:import...defprocessor(todos):forfile_nameintodos.file_paths_to_process:# processing codeThetodosobject will carry infeed_ratenumber of file names to process in.file_paths_to_process.Note: the function nameprocessoris mandatory.Why use quick_batchquick_batch aims to bedead simple to use:versus standard cloud service batch transformation services that require significant configuration / service understandingultra fast setup:versus setup of heavier orchestration tools likeairflowormlflow, which may be a hinderance due to time / familiarity / organisational constraints100% portable:- use quick_batch on any machine, anywhereprocessor-invariant:quick_batch works with arbitrary processes, not just machine learning or deep learning tasks.transparent and open source:quick_batch uses Docker under the hood and only abstracts away the not-so-fun stuff - including instantiation, scaling, and teardown.  you can still monitor your processing using familiar Docker command-line arguments (likedocker service ls,docker service logs, etc.).