Add more business logic to un-invite member

If the last member delete itself, the group will be delete
automatically.