Be smarter about how controllers are added/removed.