elasticsearch/elasticsearch — Issue #1683: Question/comment about multi-value field data construction
Author: Alex-Ikanow
Created: 2012-02-08 16:04:47+00:00
State: open

Feel free just to delete this if I'm missing something obvious, but I'm curious about the design decisions in the construction of the "ordinals" array for multi-valued field types.

You create an ArrayList of size , each entry of the ArrayList being int[maxDocs] (containing the term id)

Presumably the way you use the ordinals array is to loop over the terms for a given document id, ie

ArrayList.get(i)[docId] for an iterating index 'i'

This seems sub-optimal for 2 reasons:
- As discussed in https://groups.google.com/group/elasticsearch/browse_thread/thread/31d87c84dd387367#, the memory usage for document sets that can have a large number of values for a field can get massive
- (Less importantly) You're having to reload your L2 cache with each "inner loop" iteration of 'i'

If you switched round the order, you'd have more pain initializing the field data, but after that it would be much more compact in memory and faster (maybe only marginally?) ... which seems like the right trade-off, since you're constructing the field data once (particularly with the default resident cache) but using it many times.

Or alternatively (eg if creating maxDocs * small objects is too painful for whatever reason), take the top X% of documents (in terms of array length) after you're done, treat them differently (ie indexed the other way round, in a separate structure), and then delete as many ordinals entries as you can. (etc etc)

I'm sure many other es users are going to have decent-sized arrays of fields in their documents, what with the trendiness of social networks (friend lists), geo etc etc.