fix logic error when group add or leave

Signed-off-by: xiehuc <xiehuc@gmail.com>