only try to uninvite already invited clients.