Project: datamanagement
Issue DM-4342: Disentangle log messages from different processes
It's difficult to disentangle interleaved log messages when running with multiprocessing. Two possible ways to do this would be: 1. Have each process write to a different log file. Maybe we could allow the user to specify log filenames including {{%(pid)s}} and {{%(hostname)s}}. More useful would be to allow the full range of keys from a {{dataId}}, but that might require some changes in how processing runs. 2. Prepend each line of the log with context information ({{dataId}}, pid, hostname), allowing the user to use {{grep}} to do the disentangling. To avoid overwhelming the log with the context information, a hash of the context information could be used instead, with the lookups published in the log before the processing starts. h5. Brief summary on the changes: - Chain logs of sub-tasks to their parent task logs - Add {{PrependedFormatter}} as the new default formatter for log files (specified through {{--logdest}} command line argument for {{cmdLineTask}}). The standard output to the terminal remains the same as before. - Any string can be used to label a pex_logging Log, and with {{PrependedFormatter}} this label prepends each log message. For {{cmdLineTask}}, the dataId is prepended. - The HSC's commit on prepending the timestamps is also ported to {{PrependedFormatter}}. - An example log message: {code:java} 2016-03-08T22:29:56.889933: [{'filter': 'r', 'tract': 0, 'patch': '2,3'}]: mergeCoaddDetections: Culled 1731 of 7751 peaks {code}
Story points: 12