Restructure teardown handling, setting it in the handler is useless