It doesn't look like seeking with the state there is any faster then without it.  If I use a TermEnum that just caches the frequency then that caching will be less intrusive, I think, but still pretty quick.