I think it should rather use a BytesRefHash along with an array of integers in order to save the allocation of one HashedBytesRef for each term.