Only group users by modes that have a prefix symbol