Project: datamanagement
Issue DM-3144: Audit and improve warm-start configuration options
Many of our command-line tasks - particularly the high-level MPI-based drivers we're moving over from the HSC side - typically reuse intermediate data products they find on disk rather than regenerate them by default, and have a suite of configuration options to control this behavior. While this aids in faster reprocessing of aborted or failed jobs, it can produce results users would consider surprising ("I re-ran with a new version of the pipeline and nothing changed"), and (IMO) should not be the default behavior for any task. We also need to guarantee that any warm-start reprocessing always produces the exact same results as a single consistent run. I do not believe this is currently the case for some of the options in ProcessCcdTask. Finally, we should put these configuration options somewhere other than the main task config, because they control how the processing is done, not what the results are, and hence should not be checked against existing config files in an output data repo before running. This will probably require a new mechanism in pipe_base; perhaps a second config class associated with each Task, containing only options that affect the "how" of processing without affecting the "what". The first step of implementing this issue should be an RFC.
Story points: 8