Only log lost followers when there are losses.