Issue 1:
NGram Tokenizers are a very useful filter to match parts of compound words. Compound words can be commonly found p.e. in the German language, but occur in English as well (p.e. "thunderstorm").

https://www.elastic.co/guide/en/elasticsearch/guide/current/ngrams-compound-words.html

NGram (also called n-gram) Tokenizers split up words into token sequences. Each token has a specified minimum gram length (min_gram) and maximum gram length (max_gram). Tokens consisting of three letters p.e. are called trigrams.

Using trigrams as our starting point, the word "peanut" for example would be divided into the three-letter-tokens "pea", "ean", "anu" and "nut".

Searching for "peanut" would therefore also (partially) match words like "peace", "Sean", "Anubis" and "nutrition". They all contain one of these tokens, but might not be words (or the document if they are all in one) we want to see as our results.

Statistically speaking we would then be suffering type I errors (false positives).

https://en.wikipedia.org/wiki/Type_I_and_type_II_errors

Obviously we might then try longer min_gram lengths, p.e. four or five letters long.

With five-grams, "peanut" would be split up in "peanu" and "eanut", giving us far more exact results.

One might come to the conclusion, the longer the n-gram, the better.

Though this is be true to some extent and for some purposes, we will run into another problem.

Elasticsearch's NGram Tokenizers ignore search terms with a length smaller than the specified min_gram!

That means the actual search term will often be significantly shorter and more vague than intended. If there will be a search term left at all.

We might p.e. search for "step by step", trying to find backpacks of this company. Here we have multiple query strings, combined by the AND operator.

If we use our five-gram Tokenizer every word will be ignored. We will have no results resulting in type II errors (false negatives).

Bigrams would respect every word, but lead to a lot of type I errors (see above the "peanut" example).

Trigrams would only exclude the word "by" and respect the rest while being more accurate. But trigrams match the unwanted "Steckdosenleiste" (= multi-outlet power strip). "Ste", "tep", "Ste", "tep" on the one hand and "ste", "tec", "eck", "ckd", "kdo", "dos", "ose", "sen", "enl", "nle", "lei", "eis", "ist", "ste" on the other means 50% of the search terms are matched!

So we would might opt for a four-gram Tokenizer, bringing us good results this time.

But then searching for "Jay Smith" would also give us somebody called "John Smith" with the same exact match percentage (type I error).

Even if we wouldn't care for words smaller than four letters, sure enough one can come up with an example where five-grams would be needed.

Searching for "Recht" (= law), a four-gram Tokenizer will match "Echt + rechnung" (= calculation/bill + real):

"Recht" splits up into "rech", "echt" while "Echt + rechnung" splits up into "echt" and "rech", "echn", "chnu", "hnun", "nung", presenting a perfect match. This might come as a surprise, as those words are in two differents fields (the fields "name" and "description"), making a good example of how unintended and undesired a match can be.

Respecting the token order ("Rech" before "echt") of our search term would help here, but is not supported. Same goes for the "step by step" example.

If it were possible for the Elasticsearch NGram Tokenizer to fall back to smaller sized n-grams, using five-grams for "Smith" and trigrams for "Jay" (instead of ignoring "Jay completely), our problems described above would be gone. We would have relative n-gram precision according to the length of our search terms. But as it stands, we are forced to choose between lots of type I or lots of type II errors.

To sum the examples we have the following situation:

A gram size smaller than five leads to problems with our search term "Recht" (type I errors).

A gram size larger than three ignores "jay" in "jay smith" (type I error).

A gram size larger than four ignores "step by step" (type II error).

This is why a combination of three to five grams (p.e. min_gram=3 and max_gram=5) also fails.

Feel free to play around with the examples in my gist: https://www.found.no/play/gist/6d27486e51bdda88dd84

Edit the min_gram and max_gram sizes to your liking in the "Analysis" section (center left), rerun the queries ("Run" in the top right) and see the altered "Results" section (bottom right) with our Searches # 1, # 2 and # 3.

Searching for "jay smith" corresponds to Search # 1, searching for "step by step" to Search # 2 and finally searching for "recht" to Search # 3. To try out other search terms, edit the queries in the "Searches" section (top right).

Or edit the "Documents" section (top left), to experiment with different words to be searched for.

To summarize:

If we want to match search terms to compound words ("thunderstorm"), NGram Tokenizers are a fine tool. They split up words into tokens of a predefined size.

But Elasticsearch's fixed size NGram Tokenizers force us to choose between either lots of type I or type II errors. What works well for short words like "Jay" won't work for longer words like "Recht" (German for law) and vice versa.

Desired would be a flexible NGram Tokenizer. This means being able to switch from longer NGram Tokenizers (p.e. five-grams) to shorter (p.e. trigrams), as the search terms become shorter. Another, maybe more complicated way could be to make the NGram Tokenizer respect the order of the tokens.

Issue 2:
It would be great if the nGram tokenizer came with an option to force creation of a whole-word token for cases when the word length falls outside the min-gram/max-gram range. Currently, we have to resort to a dual-mapping approach to make sure these tokens get created, which limits how we can construct per-field queries.