Should it rather use the terms enum of the sorted set values? (probably using a FilterTermsEnum since ordinals start at 0 in Lucene and 1 in Elasticsearch)