Don't eagerly pull ordinals if doc values instance extends RandomAccessOrds