Don't remove handlers if none are present.