Package: album-distributed
Album plugin for distributed callsThis is an early version of enhancing album with calls for batch and distributed processing.Installationinstall AlbumActivate the album environment:conda activate albumInstall this plugin:pip install https://gitlab.com/album-app/plugins/album-distributed/-/archive/main/album-distributed-main.zipUsageFist, install a solution - replacesolution.pywith the path to your solution / solution folder or with thegroup:name:versioncoordinates of your solution.album install solution.pyNow you can use the plugin:album run-distributed solution.pyThe plugin does two things:It figures out if the input arguments match multiple tasks - in this case, it generates the different task arguments.It runs all matching tasks, the mode for running these tasks can be chosen.Since the matching part can be tricky, please use the--dry-runargument to first print a list of matched tasks:album run-distributed solution.py --dry-runOn Windows, replace the slashes with backslashes in the examples on this page.Please let us know if you run into issues.Matching input argumentsTo generate multiple tasks, patterns in file name arguments can be used to match multiple files.Using patterns in a single argumentYou should be able to use allglobfeatures when using it in a single argument. Here are some examples:In the following scenariossolution.pyhas an argument calledinput_data.Match all.tiffiles in the current folder:album run-distributed solution.py --input_data *.tifMatch all.tiffiles in a specific folder where the file name starts withinput:album run-distributed solution.py --input_data /data/input*.tifMatch all.tiffiles recursively, starting from the current folder:album run-distributed solution.py --input_data **/*.tifUsing patterns in multiple argumentsWhen using patterns in multiple arguments, this plugin will try to figure out the corresponding argument values based on which of the patterns match with existing files.
This is likely to fail in a bunch of situations - please use the--dry-runargument to test if the matched tasks correspond with your expectation.In the following scenariossolution.pyhas two arguments calledinput_dataandoutput_data.Use all.tiffiles in the current folder and append_outto the file name for the output argument.album run-distributed solution.py --input_data *.tif --output_data *_out.tifDo the same thing recursively:album run-distributed solution.py --input_data **/*.tif --output_data **/*_out.tifLet the output argument values live in a different folder:album run-distributed solution.py --input_data *.tif --output_data output/*.tifSince Album does not yet distinguish between input and output arguments, be aware that if theoutput_dataargument in these scenarios matches existing files, the plugin will also try to generate correspondinginput_filevalues. We will work on improving this.ModesYou can set the mode by using the--modeargument:album run-distributed solution.py --mode queueBy default, the plugin will use thebasicmode.BasicIn this mode, all tasks will be performed one after each other. The console output of each task will be printed.QueueIn this mode, a set of thread workers will be created to process tasks in parallel. The console output of each task will not be printed.
You can control how many threads should be created with the--threadsargument:album run-distributed solution.py --mode queue --threads 16