Do not empty all fallback fields in sorting mode 4 (see #6498)