elasticsearch/elasticsearch — Issue #1449: AbstractCompoundWordTokenFilterFactory should store the dictionary as a CharArraySet instance, not a Set
Author: nkvoll
Created: 2011-11-10 11:45:24+00:00
State: open

AbstractCompoundWordTokenFilterFactory stores the word list as Set, but in Lucene, CompoundWordTokenFilterBase checks whether the dictionary is a CharArraySet, and if it's not, creates a new CharArraySet instance and populates it with the words from the input collection. This causes each TokenFilter to store a duplicate version of the dictionary in-memory.

The net effect is that ES ends up using a lot more memory than needed when using decompounding.

For a reproducible example, see https://gist.github.com/93e84d4fc412ae16d899, which fails if ES_MAX_MEM=1g, but works when ES_MAX_MEM=2g (on linux).