Move on handler functionality to its own module
Also support on_exit on setup_all