elasticsearch/elasticsearch — pull request #6354 by mikemccand (2014-05-30 13:36:53+00:00)
Title: Add low level Lucene tool to see which types/fields are using the most space in the index

At the Hackfest after Berlin Buzzwords we created a simple diagnostic
tool to see low-level index statistics (bytes used, total postings
ints) per-type and per-field.

It's just a standalone command-line (static main) tool for now, and it
takes some time to run (it visits all postings, docs only). It
produces output like this:

Reader maxDoc=537472 delDocs=0 (0%)

stored fields bytes by _type & field:
foo: 100.00%, 6665.5 MB
_source
98.62%, 6573.2 MB
_uid
1.38%, 92.3 MB

postings terms bytes:
foo: 100.00%, 142.7 MB
_all
38.58%, 55.1 MB
message
38.31%, 54.7 MB
_uid
9.34%, 13.3 MB
request
7.31%, 10.4 MB
...

postings total ints:
foo: 100.00%, 446.1 M
_all
norms,DOCS_AND_FREQS_AND_POSITIONS
46.01%, 205.2 M
message
norms,DOCS_AND_FREQS_AND_POSITIONS
38.55%, 172.0 M
agent
norms,DOCS_AND_FREQS_AND_POSITIONS
4.07%, 18.1 M
...

It works by first visiting all _uid terms to make a mapping from docID
-> _type, and then visits stored fields & postings to gather up stats
by type and field.

It's just a start, but it should be useful when you want some
low-level details about which types are contributing to which field's
index bytes usage, how fields were indexed, etc.