Fix user count when removing server users