Kill job runner at least 500 ms after we get a 'stop' signal, if not earlier.