elasticsearch/elasticsearch — pull request #5706 by bleskes (2014-04-07 11:52:33+00:00)
Title: Added a AppendingDeltaPackedLongBuffer-based storage format to single value field data

The AppendingDeltaPackedLongBuffer uses delta compression in paged fashion. For data which is roughly monotonic this results in a reduced memory signature.

By default we use the storage format expected to use the least memory. You can force a choice using a new field data setting `memory_storage_hint` which can be set to `ORDINALS`, `PACKED` or `PAGED`

Running some benchmarks on simulated time based data shows 25-30% reduction in memory usage with a very small performance overhead (current implementations uses PACKED as a memory format with 0.5 acceptable overhead ratio):

```

------------------ SUMMARY -------------------------------
docs: 5000000
match percentage: 0.1
memory format hint: PACKED
acceptable_overhead_ratio: 0.5
field data: 19mb
name took millis
hist_l 16.9s 33
------------------ SUMMARY -------------------------------

------------------ SUMMARY -------------------------------
docs: 5000000
match percentage: 0.1
memory format hint: PAGED
acceptable_overhead_ratio: 0.5
field data: 14.6mb
name took millis
hist_l 18.2s 36
------------------ SUMMARY -------------------------------

------------------ SUMMARY -------------------------------
docs: 5000000
match percentage: 0.1
memory format hint: PACKED
acceptable_overhead_ratio: 0.0
field data: 16mb
name took millis
hist_l 17.4s 34
------------------ SUMMARY -------------------------------

------------------ SUMMARY -------------------------------
docs: 5000000
match percentage: 0.1
memory format hint: PAGED
acceptable_overhead_ratio: 0.0
field data: 10.8mb
name took millis
hist_l 21s 42
------------------ SUMMARY -------------------------------

```