elasticsearch/elasticsearch — pull request #8689 by antonha (2014-11-27 17:18:23+00:00)
Title: Use RoaringDocIdSet instead of FixedBitset for caching.

The default DocIdBitSet used for caching is currently the FixedBitSet implementation. For very sparse or dense bitsets, there are better implementations.

Lucene currently uses WAH8DocIdSet. https://github.com/elasticsearch/elasticsearch/pull/7577 suggested that ES should also switch to that implementation. The PR was rejected, in favour of getting the RoaringDocIdSet from Lucene 5.

For ES users with large indices, it would be beneficial if the filter cache did not use so much memory. This PR uses the RoaringDocIdSet from Lucene trunk (with minor modifications) instead of the FixedBitSet implementation. For the modifications, see inline comments.

I've ran the test suite locally a few times, and it works. I did see some issues, but fixed them.