Only hold on to scheduler references if we're running a debugger.