Correctly handle the transition out of alt-hold if were controlling