Package: batch-processing
Batch ProcessingA Python library to make batchable Python scripts.
Main features are:Collects and filters any number of input files from the disk.Parallelisation using multiprocessing pool.OS-independent paths (thanks to theUnified Pathlibrary).Saves at each run the parameters of the run in a timestamped JSON file.Can use serialized input parameters in addition of the command line arguments.Can replicate arbitrary input folder structures in the output folder.Can handle multiple input files for one output.InstallationInstall from pypi using pip:pip install batch-processingBatch scriptsHere is a minimal example of a Batch script:importargparseimportsysfrompathlibimportPathfrombatch_processingimportBatchdefparse_command_line(batch:Batch):parser=argparse.ArgumentParser(description='Batch example',formatter_class=argparse.ArgumentDefaultsHelpFormatter)# Setup custom script arguments herereturnbatch.parse_args(parser)defparallel_process(input:Path,output:Path,args:argparse.Namespace,additional_argument:float):# Load an input file, apply some processing, and save to output file herereturnresult# optionally return some resultdefmain(argv):# Instantiate batchbatch=Batch(argv)batch.set_io_description(input_help='input files',output_help='output directory')# Parse argumentsargs=parse_command_line(batch)# Optionally do some pre-processing here beforeadditional_argument=5.0# Start processing and retrieve the results at the end if anydata=batch.run(parallel_process,additional_argument,output_ext=Batch.USE_INPUT_EXT)# Optionally do some post processing hereif__name__=="__main__":main(sys.argv[1:])UsageArguments are defined using the standardargparsePython library.
As soon as the Batch library is used, predefined arguments are automatically added to the argparse parser.The complete list of accepted arguments can be obtained by running a script using the Batch library with the-hor--helpflag.InputsAny number of inputs can be given using the-iflag. Simple wildcard*and recursive wildcard**are accepted.-i input1 input2 input3
-i "folder1/**/*.png" "folder2/**/*.png"TagsThe inputs can additionaly be categorized using tags. Tags are defined by appending a word that begins with + in the input list.-i +tag1 "folder1/**/*.png" +tag2 "folder2/**/*.png"FilteringThe--onlyflag allows to select the inputs that match the given patterns, whereas--excludeallows to reject them.-i **/*.png --exclude "regex1" "regex2" --only "regex1" "regex2"GroupingInputs can be grouped using the--group-inputsargument.
The inputs that should be grouped together must share a common key, that is computed from the regex provided in the argument.In case of grouped inputs, the first argument of the batch processing function becomes a list of Path instead of a single Path:defparallel_process(inputs:list[Path],output:Path,args):# Load the input files, apply some processing, and save to output file hereFor example, if we have the following input structure:folder1/input1input2folder2/input3input4And we want to group the inputs by folder, we can use the regex--group-inputs "(.+)/".
Theparallel_processfunction will be called two times withinputs = ['folder1/input1', 'folder1/input2']andinputs = ['folder2/input3', 'folder2/input4'].OuputsOutput can be given using the-oflag. Depending on the batch operation mode, the output can either be a folder or a file.The additional--prefixand--suffixflags allow a append a prefix and/or a suffix to the output filename.Root directoryInputs can be given relative to a root directory. That is to say, it is possible to call:--root-dir D:/Images -i "folder1/*.jpg" "folder2/*.jpg"Instead of:-i "D:/Images/folder1/*.jpg" "D:/Images/folder2/*.jpg"The main usage is that it allows to use the special directive:dirin the arguments.:dirrepresents the folder hierarchy between the root path and the considered file.For example:-o output_dir/:dirwill replicate the input folder hierarchy in the output folder (we will get outputs forfolder1inputs inoutput_dir/folder1, and outputs forfolder2inputs inoutput_dir/folder2).-o output_dir --prefix :dirwill flatten the hierarchy in output_dir, but will prefix the outputs by the folder (for example output corresponding tofolder1/img1.jpgwill beoutput_dir/folder1_img1.jpg).MultiprocessingWhen multiprocessing is used, the number of parallel processes can be chosen with the-jflag.
By default, it is fixed to the number of CPU cores + 2, capped to 12.LoggingThe batch library automatically initializes theloggingstandard Python library.
The--log-levelflag allows to display messages up to the given level (by defaultinfo), while the--log-fileflag allows
to redirect the log output to a file.--log-level debug --log-file logs.txtSerializationAll arguments (batch arguments and custom script arguments) can be serialized in a JSON file.
The JSON keys correspond to thedestvalues of the argparse argument definitions.{"args":{"alpha":0.5,"beta":0.7}}Serialized argument files can be given on the command line with the-aargument.-i inputs -o output -a args.jsonPriority rulesIf multiple JSON files are provided, they will be merged together along with the command line arguments.
When an argument appears multiple times, the priority (higher to lower) is:The command line.The serialized files from right to left.-a first.json second.json --beta 0.9In this example, thebetavalue fromsecond.jsonwill override thebetavalue fromfirst.json.
Finally, the--beta 0.9argument will override thebetavalues from bothfirst.jsonandsecond.json.RecursionA JSON argument file can itself refer to another JSON.{"args":{"alpha":0.5,"arg_files":["data.json"]}}In this case,"alpha": 0.5will override thealphavalue fromdata.json.ArchivingEach time a Batch script is ran, an unique JSON file is created in the output folder, tracking the current values of the arguments.
This is especially useful for:Debugging: we can see the actual parameters as parsed by the Batch library.Reproducing the results: we can use back this file as input (with-a) in order to launch again the script with the same arguments as archived.Archiving: it documents how the outputs have been generated, for future reference.