This should depend on the state, not the mode: setting mode to REMOVE does not mean it should not be included in stability calculations; in fact that's a critically important case.  Only when transitioning to REMOVED should the controller be removed (and then, it should always be removed).