Don't hold a reference to the loader.

We only need the loader to schedule tasks, no longer.