Build an ordinal cache in front of the actual parent ids BytesRefHash