Fielddata: Fix the ordinals impl for sparse fields.

Caused by #6908