Don't cache BytesRef in ThreadLocal