Add support for removing users, and putting users in a set of groups