I plan to do it as part of another issue since this has a larger scope, but yes, I would like to remove this rehashing and to use a better hashing function for BytesRef. I'm thinking of Murmur 3 right now, but I'm open to other ideas.