Remove already invited members from invite list