Remove owning raw pointers from Scheduler