Handling the case where someone declines

And also if someone leaves the invitee lounge.