Project: datamanagement
Issue DM-4397: Scale CommandLineTask multiprocessing timeout with workload
The default timeout value for aborting a multiprocessing run in CommandLineTask is too short. Currently if no time length is supplied by the user, the default value gets set to 9999s. However if a processing task is quite large it is possible for the processing pool to take much longer to arrive at the result. Currently if the processing pool does not complete it's run within that time limit, python multiprocessing will throw a timeout error. The timeout value should be scaled such that the supplied value is assumed to be the timeout length for one processing task, and should be scaled by the number of tasks divided by the number of cpus available. The command line task documentation should be updated to reflect this change.
Story points: 2