Actually add users to be ignored to ignoreUsers