I think as a starting point it would be better to just log at application boundaries rather than job boundaries. I.e. just have a single log for the entire spark application. This class makes the assumption that jobs are (mostly) sequential, but in fact stages and tasks from different jobs can be interleaved. So rolling the log based on onJobStart/onJobEnd is not correct.