Remove players and groups from access list