Joe Wexler: fMRI Pipelines on HPC with DataLad and ReproMan
Channel: DataLad
Date: 2024-04-09

In this lightning talk, I will share my experience using DataLad, git-annex and ReproMan to run software pipelines on hundreds of fMRI datasets on an HPC cluster. Potential topics may include: (a) The use of ReproMan to avoid the difficulties of using datalad containers-run in parallel on an HPC. (b) How to use DataLad on a scratch filesystem that periodically purges files to save space. (c) A simple algorithm I implemented in ReproMan to prevent excess runtime due to outliers in parallel jobs. (d) The pros and cons of the YODA-BIDS layout for neuroimaging data. I hope my talk will prompt discussion with those hoping to learn more from my experience as well as those who have found alternative solutions to similar challenges.

Part of Distribits 2024: https://distribits.live