Create task config dir on kill, if necessary, so task state can be correctly tracked