Refactor how listeners are detached, which allowed the shutdown code to
be greatly cleaned up.