Remove janitor from initialization after it has already been stopped -- since ScheduledExectuor does not support being restarted.
(This avoids test to spit stack traces)