Get the UI work properly for add/remove Group members

[Skip ci]