Handle edge case of empty group in take_group_member

This shouldn't happen, but seems worth protecting against since it
doesn't add much complexity.