Fielddata: Moved the growing logic to IntArrayRef