Don't crash the user manager if remove is clicked when no user is selected