Try not to refontify the buffer if it hasn't been modified since we last fontified it.