Fix leave group scenario
Refactor user_join_group_spec