elasticsearch/elasticsearch — pull request #2485 by bleskes (2012-12-14 20:27:12+00:00)
Title: 0.20

Here is proposed to solution to the problem described in #2468 - the large of amount of memory consumed by the ordinal arrays of the field cache in highly uneven distribution.

I have introduced a new dedicated storage class ( src/main/java/org/elasticsearch/index/field/data/MultiValueOrdinalArray.java ) . The storage class' space complexity is linear in both the number of segment documents as the number of ordinals needed.

Performance is identical to the current implementation if documents have no value or only a single one. For documents with two or more ordinals there is a slight per doc overhead due to some integer arithmetics. I suspect this is more then compensated by a more efficient traversing of memory, but this has to be measured (the class walks memory linearly within a single array rather jump through multiple arrays).

I've changed all the Field Data classes to use the new container. To keep impact on the code minimal, I have not changed the initial loading of the ordinals but rather compile the ordinal arrays into the new data store. This means that loading the Field cache initially has the same memory requirement as before.

Another caveat to note: at the moment you can store up to ~2^30 ordinals in it. For me this is not a problem, but it is a limitation the current implementation doesn't have (with enough memory).

My hope is that you can see this as a drop in replacement for the next 0.20.x release as I believe it is equivalent to the current implementation but has a much better memory signature.

To help integrating it, I merged it already with the laters 0.20 branch (I also have a 0.19 variant, which we currently run in production) . All unit tests pass running this new code and we run it in production.

I know the team is busy with Lucene 4 but I hope you can take a couple of hours for this. I'd love to able to go back to vanilla ES rather than have the overhead of running a custom build.

Let me know if I can help in any way,
Boaz